本文介绍了在CDH4.0.1 VM上使用MRV1运行wordcount示例的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我从



我发现以下列出的服务在系统启动后运行。


  • MRV1服务



hadoop-0.20-mapreduce-jobtracker

hadoop-0.20-mapreduce-tasktracker


  • MRV2服务



hadoop-yarn-nodemanager

hadoop-yarn-resourcemanager

hadoop -mapreduce-historyserver


  • HDFS服务



hadoop-hdfs-namenode

hadoop-hdfs-datanode



单词计数示例运行正常并按预期生成输出

  / usr / bin / hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount输入输出

然而,上面的运行使用MRv2 - YARN框架



我的目标是使用MRv1运行。正如Cloudera文档中的建议,我停止了MRV2服务,并编辑了/etc/hadoop/conf/mapred-site.xml

 < property> 
< name> mapreduce.framework.name< / name>
<值>纱线< /值>

改为classic(也试过local p>

 <属性> 
< name> mapreduce.framework.name< / name>
<值>传统< /值>

我期望它使用MRV1(jobtracker和tasktracker)运行。但是,我看到以下错误:

  12/10/10 21:48:39信息mapreduce.Cluster:无法使用org.apache.hadoop.mapred.LocalClientProtocolProvider由于错误:LocalJobRunner的mapreduce.jobtracker.address配置值无效:172.30.5.21:8021
12/10/10 21:48:39错误安全性。 UserGroupInformation:PriviledgedActionException as:cloudera(auth:SIMPLE)cause:java.io.IOException:无法初始化群集。请检查您的配置mapreduce.framework.name和相应的服务器地址。
java.io.IOException:无法初始化群集。请检查您的配置mapreduce.framework.name和相应的服务器地址。
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster。< init>(Cluster.java:83)
......

有人可能会提示什么可能是错的。为什么错误指向无效配置?

解决方案

我认为你的集群仍然指向MRv2配置目录,而不是MRv1。 p>

更新/安装 hadoop-conf 集群中每个节点的指向MRv1配置目录的备选优先级。 p>

然后重新启动您的所有服务。



例如:

  $ sudo update-alternatives --install / etc / hadoop / conf hadoop-conf /etc/hadoop/conf.mrv1 50 
$ sudo update-alternatives --set hadoop- conf /etc/hadoop/conf.mrv1


I downloaded the VM from https://downloads.cloudera.com/demo_vm/vmware/cloudera-demo-vm-cdh4.0.0-vmware.tar.gz

I found that below listed services are running after the system boots.

  • MRV1 Services

hadoop-0.20-mapreduce-jobtracker
hadoop-0.20-mapreduce-tasktracker

  • MRV2 services

hadoop-yarn-nodemanager
hadoop-yarn-resourcemanager
hadoop-mapreduce-historyserver

  • HDFS Services

hadoop-hdfs-namenode
hadoop-hdfs-datanode

The word count example runs fine and generates the output as expected

/usr/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount input output

However, the above runs using the MRv2 - YARN framework

My goal is to run using MRv1. As suggested on the Cloudera documentation, I stop the MRV2 services, and edited /etc/hadoop/conf/mapred-site.xml

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property

to "classic" (also tried "local")

  <property>
    <name>mapreduce.framework.name</name>
    <value>classic</value>
  </property

I expected it to run using MRV1 (jobtracker and tasktracker). However, I see the following error:

12/10/10 21:48:39 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "172.30.5.21:8021"
12/10/10 21:48:39 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
        ......

Can someone suggest what could be wrong. Why is the error pointing to invalid configuration?

解决方案

I think your cluster still points to MRv2 configuration directory instead on MRv1.

Update/Install hadoop-conf alternative in each node in the cluster pointing to MRv1 configuration directory with high priority.

Then restart all your services.

Eg:

$ sudo update-alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.mrv1 50
$ sudo update-alternatives --set hadoop-conf /etc/hadoop/conf.mrv1

这篇关于在CDH4.0.1 VM上使用MRV1运行wordcount示例的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-20 15:02