hadoop stream perl

依照我原来的
http://www.blogjava.net/Skynet/archive/2009/09/25/296420.html
以及官方文档
http://hadoop.apache.org/common/docs/r0.15.2/streaming.html


我现在有个 CPU 密集型的 运算 ，比如时间戳 转换
# 原始文件
> [email protected]    1019    20110622230010

# 先把 文件切分成 我们需要的 map 运算数量 (根据行数)
$ split -l 500000 userid_appid_time.pplog.day.data

$vim  run.pm

1. #!/usr/bin/perl -an
   
2. chomp;
   
3. @F=split "\t";
   
4. $F[2]=~s/(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/\1-\2-\3 \4:\5:\6/g;
   
5. $h{$F[2]}=`date -d \"$F[2]\" +%s` if not exists $h{$F[2]};
   
6. print "$F[0]\t$F[1]\t$h{$F[2]}

$ 运行
hadoop jar ./contrib/streaming/hadoop-streaming-0.20.2-cdh3u1.jar  \
      -input hdfs:///tmp/pplog/userid_appid_time.pplog.day/x*  \
      -mapper   run.pm  \
      -file /opt/sohudba/20111230/uniqname_pool/run.pm  \
      -output hdfs:///tmp/lky/streamingx3

$ 输出结果：


$ 帮助

1. hadoop jar ./contrib/streaming/hadoop-streaming-0.20.2-cdh3u1.jar -info
2. 12/01/09 16:38:00 ERROR streaming.StreamJob: Missing required options: input, output
3. Usage: $HADOOP_HOME/bin/hadoop jar \
4. $HADOOP_HOME/hadoop-streaming.jar [options]
5. Options:
6. -input DFS input file(s) for the Map step
7. -output DFS output directory for the Reduce step
8. -mapper | The streaming command to run
9. -combiner | The streaming command to run
10. -reducer | The streaming command to run
11. -file File/dir to be shipped in the Job jar file
12. -inputformat TextInputFormat(default)|SequenceFileAsTextInputFormat|
13. -outputformat TextOutputFormat(default)|
14. -partitioner
15. -numReduceTasks Optional.
16. -inputreader Optional.
17. -cmdenv = Optional. Pass env.var to streaming commands
18. -mapdebug Optional. To run this script when a map task fails
19. -reducedebug Optional. To run this script when a reduce task fails
20. -io Optional.
21. -verbose
22. 
    
23. Generic options supported are
24. -conf specify an application configuration file
25. -D use value for given property
26. -fs | specify a namenode
27. -jt | specify a job tracker
28. -files specify comma separated files to be copied to the map reduce cluster
29. -libjars specify comma separated jar files to include in the classpath.
30. -archives specify comma separated archives to be unarchived on the compute machines.
31. 
    
32. The general command line syntax is
33. bin/hadoop command [genericOptions] [commandOptions]
34. 
    
35. 
    
36. In -input: globbing on is supported and can have multiple -input
37. Default Map input format: a line is a record in UTF-8
38. the key part ends at first TAB, the rest of the line is the value
39. Custom input format: -inputformat package.MyInputFormat
40. Map output format, reduce input/output format:
41. Format defined by what the mapper command outputs. Line-oriented
42. 
    
43. The files named in the -file argument[s] end up in the
44. working directory when the mapper and reducer are run.
45. The location of this working directory is unspecified.
46. 
    
47. To set the number of reduce tasks (num. of output files):
48. -D mapred.reduce.tasks=10
49. To skip the sort/combine/shuffle/sort/reduce step:
50. Use -numReduceTasks 0
51. A Task's Map output then becomes a 'side-effect output' rather than a reduce input
52. This speeds up processing, This also feels more like "in-place" processing
53. because the input filename and the map input order are preserved
54. This equivalent -reducer NONE
55. 
    
56. To speed up the last maps:
57. -D mapred.map.tasks.speculative.execution=true
58. To speed up the last reduces:
59. -D mapred.reduce.tasks.speculative.execution=true
60. To name the job (appears in the JobTracker Web UI):
61. -D mapred.job.name='My Job'
62. To change the local temp directory:
63. -D dfs.data.dir=/tmp/dfs
64. -D stream.tmpdir=/tmp/streaming
65. Additional local temp directories with -cluster local:
66. -D mapred.local.dir=/tmp/local
67. -D mapred.system.dir=/tmp/system
68. -D mapred.temp.dir=/tmp/temp
69. To treat tasks with non-zero exit status as SUCCEDED:
70. -D stream.non.zero.exit.is.failure=false
71. Use a custom hadoopStreaming build along a standard hadoop install:
72. $HADOOP_HOME/bin/hadoop jar /path/my-hadoop-streaming.jar [...]\
73. [...] -D stream.shipped.hadoopstreaming=/path/my-hadoop-streaming.jar
74. For more details about jobconf parameters see:
75. http://wiki.apache.org/hadoop/JobConfFile
76. To set an environement variable in a streaming command:
77. -cmdenv EXAMPLE_DIR=/home/example/dictionaries/
78. 
    
79. Shortcut:
80. setenv HSTREAMING "$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar"
81. 
    
82. Example: $HSTREAMING -mapper "/usr/local/bin/perl5 filter.pl"
83. -file /local/filter.pl -input "/logs/0604*/*" [...]
84. Ships a script, invokes the non-shipped perl interpreter
85. Shipped files go to the working directory so filter.pl is found by perl
86. Input files are all the daily logs for days in month 2006-04
87. 
    
88. Streaming Command Failed!