hadoop集群运行python mapreduce程序错误及其解决办法

自己用python写了个mapreduce程序，结果运行时候一直报错；
运行命令：

点击(此处)折叠或打开

1. hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar -mapper /Application/lianshuchengjin/MR/map.py -reducer /Application/lianshuchengjin/MR/reduce.py -input /fc/src/test/* -output /fc/src/result

第一次报错为：
ava.lang.RuntimeException: Error in configuring object        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:386)        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)        at org.apache.hadoop.mapred.Child$4.run(Child.java:266)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:396)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)        at org.apache.hadoop.mapred.Child.main(Child.java:260)Caused by: java.lang.reflect.InvocationTargetException        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav

百度了很多没有解决，最后修改了Map.py和reduce.py程序权限，用户属性等，chown -R hadoop:hadoop map.py reduce.py 包括要执行的日志文件/fc/src/test/*的权限也修改为hadoop:hadoop ，运行程序，又报错：
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

最终解决办法是运行命令的问题，改为下面的命令运行就对了；

点击(此处)折叠或打开

1. hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u6.jar -mapper "python map.py" -reducer "python reduce.py" -input /fc/src/test/*.gz -output /fc/src/result -file /Application/lianshuchengjin/MR/map.py -file /Application/lianshuchengjin/MR/reduce.py