问题描述
我正在尝试在我的 Amazon Elastic MapReduce 作业中启用错误输入跳过.我正在遵循此处描述的绝妙食谱:
I am trying to enable bad input skipping on my Amazon Elastic MapReduce jobs. I am following the wonderful recipe described here:
http://devblog.factual.com/practical-hadoop-streaming-dealing-with-brittle-code
上面的链接说我需要以某种方式在 EMR 作业上设置以下配置参数:
The link above says that I need to somehow set the following configuration parameters on an EMR job:
mapred.skip.mode.enabled=true
mapred.skip.map.max.skip.records=1
mapred.skip.attempts.to.start.skipping=2
mapred.map.tasks=1000
mapred.map.max.attempts=10
如何使用 Boto 在 JobFlow 上设置这些(和其他)mapred.XXX 参数?
How do I set these (and other) mapred.XXX parameters on a JobFlow using Boto?
推荐答案
经过数小时的挣扎、阅读代码和实验,答案如下:
After many hours of struggling, reading code, and experimentation, here is the answer:
您需要添加一个新的 BootstrapAction,如下所示:
You need to add a new BootstrapAction, like so:
params = ['-s','mapred.skip.mode.enabled=true',
'-s', 'mapred.skip.map.max.skip.records=1',
'-s', 'mapred.skip.attempts.to.start.skipping=2',
'-s', 'mapred.map.max.attempts=5',
'-s', 'mapred.task.timeout=100000']
config_bootstrapper = BootstrapAction('Enable skip mode', 's3://elasticmapreduce/bootstrap-actions/configure-hadoop', params)
conn = EmrConnection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
step = StreamingStep(name='My Step', ...)
conn.run_jobflow(..., bootstrap_actions=[config_bootstrapper], steps=[step], ...)
当然,如果您有多个引导操作,您应该将其添加到 bootstrap_actions 数组中.
Of course, if you have more than one bootstrap action, you should just add it to the bootstrap_actions array.
这篇关于使用 boto 设置 hadoop 参数?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!