xml - 如何将多个节点添加到Hadoop 2.2.0 Yarn？

我有一个进行中的Hadoop集群配置，其中包括四台计算机，但是在MapReduce WebUI中，我只看到一个 Activity 节点，而不是四个 Activity 节点，即触发MapReduce的olympus名称节点。

我知道类似的questions，但是这样做可能还是由于版本差异而无济于事？

我的启动似乎工作正常:

deploy@olympus:~$ start-dfs.sh && start-yarn.sh
Starting namenodes on [olympus]
olympus: starting namenode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-namenode-olympus.out
zeus: starting datanode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-datanode-zeus.out
hera: starting datanode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-datanode-hera.out
olympus: starting datanode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-datanode-olympus.out
poseidon: starting datanode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-datanode-poseidon.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/dev/hadoop/2.2.0/logs/hadoop-deploy-secondarynamenode-olympus.out
starting yarn daemons
starting resourcemanager, logging to /opt/dev/hadoop/2.2.0/logs/yarn-deploy-resourcemanager-olympus.out
poseidon: starting nodemanager, logging to /opt/dev/hadoop/2.2.0/logs/yarn-deploy-nodemanager-poseidon.out
olympus: starting nodemanager, logging to /opt/dev/hadoop/2.2.0/logs/yarn-deploy-nodemanager-olympus.out
zeus: starting nodemanager, logging to /opt/dev/hadoop/2.2.0/logs/yarn-deploy-nodemanager-zeus.out
hera: starting nodemanager, logging to /opt/dev/hadoop/2.2.0/logs/yarn-deploy-nodemanager-hera.out

我的yarn-site.xml在所有四台机器上都看起来像这样:

<configuration>
  <property>
    <name>yarn.resourcemanager.resourcetracker.address</name>
    <value>olympus:8025</value>
    <description>Enter your ResourceManager hostname.</description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>olympus:8030</value>
    <description>Enter your ResourceManager hostname.</description>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>olympus:8050</value>
    <description>Enter your ResourceManager hostname.</description>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>olympus:8041</value>
    <description>Enter your ResourceManager hostname.</description>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <final>true</final>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    <final>true</final>
  </property>
</configuration>

我的mapred-site.xml在所有四台机器上都看起来像这样:

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>olympus:8021</value>
    <final>true</final>
  </property>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <final>true</final>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>olympus:10020</value>
    <description>Enter your JobHistoryServer hostname.</description>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>olympus:19888</value>
    <description>Enter your JobHistoryServer hostname.</description>
  </property>
</configuration>

我的core-site.xml在所有四台机器上都看起来像这样:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://olympus:8020/</value>
    <final>true</final>
  </property>
</configuration>

我的hdfs-site在所有四台机器上都看起来像这样:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/deploy/hdfs/name</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.datanode.name.dir</name>
    <value>file:/home/deploy/hdfs/data</value>
    <final>true</final>
  </property>
</configuration>

最后，我的slaves文件在所有四台机器上都看起来像这样:

 olympus
 zeus
 hera
 poseidon

UDPATE:

我在日志文件yarn-deploy-nodemanager-zeus.log中看到以下错误消息:

 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8031

该0.0.0.0:8031在哪里定义为yarn或hadoop配置的一部分？它不应该尝试连接到olympus:8031或olympus:8032吗？

最佳答案

为了使NodeManagers上的zeus，hera和poseidon能够注册到ResourceManager守护程序，然后才能出现在 Activity 节点列表中，必须可以访问8032上的olympus端口。

通常，默认的默认操作系统会打开防火墙，这可以在start-dfs.sh && start-yarn.sh之后自动阻止这种情况的发生。您可以尝试禁用操作系统的防火墙(如果已启用)，然后重试相同的启动，一切应该正常进行。可以使用service iptables stop关闭大多数Linux OS的防火墙，但是您可以检查并使用适用于您的OS的防火墙。

如果防火墙关闭，那么您将需要登录zeus，hera或poseidon之一，并检查日志文件/opt/dev/hadoop/2.2.0/logs/yarn-deploy-nodemanager-*.log和/opt/dev/hadoop/2.2.0/logs/hadoop-deploy-datanode.log是否存在错误(连接性排序等)，然后根据提供的内容进行进一步的故障排除。它。

关于xml - 如何将多个节点添加到Hadoop 2.2.0 Yarn？，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/21555434/