hadoop - 在Hadoop中，Namenode如何获取机架详细信息和属于机架的数据节点？

假设在Hadoop集群中，我们有2个机架rck1和rck2。每个机架有5个节点。 Namenode如何得知节点1属于机架1，节点3属于机架2。

最佳答案

您必须配置系统以指定如何确定机架信息。例如，this Cloudera link告诉您如何在Cloudera Manager中为主机配置机架。

另外，this Apache link解释了如何通过配置文件在java类的外部脚本中指定此信息。

尽管可以使用更深的层次结构，但是拓扑通常采用/ myrack / myhost的形式。他们在python中有以下示例，该示例为每个机架假定一个/ 24子网，因此提取IP地址的前三个字节用作机架号-如果可以相应地设置节点IP地址，则可以采用类似的方法，或编写您自己的脚本以根据IP地址或每个节点上的其他可用信息确定机架(即使在主机名与机架之间的简单硬编码映射在您的示例中也可以用相对较少的节点进行工作)。

#!/usr/bin/python
# this script makes assumptions about the physical environment.
#  1) each rack is its own layer 3 network with a /24 subnet, which
# could be typical where each rack has its own
#     switch with uplinks to a central core router.
#
#             +-----------+
#             |core router|
#             +-----------+
#            /             \
#   +-----------+        +-----------+
#   |rack switch|        |rack switch|
#   +-----------+        +-----------+
#   | data node |        | data node |
#   +-----------+        +-----------+
#   | data node |        | data node |
#   +-----------+        +-----------+
#
# 2) topology script gets list of IP's as input, calculates network address, and prints '/network_address/ip'.

import netaddr
import sys
sys.argv.pop(0)                                                  # discard name of topology script from argv list as we just want IP addresses

netmask = '255.255.255.0'                                        # set netmask to what's being used in your environment.  The example uses a /24

for ip in sys.argv:                                              # loop over list of datanode IP's
address = '{0}/{1}'.format(ip, netmask)                      # format address string so it looks like 'ip/netmask' to make netaddr work
try:
   network_address = netaddr.IPNetwork(address).network     # calculate and print network address
   print "/{0}".format(network_address)
except:
   print "/rack-unknown"                                    # print catch-all value if unable to calculate network address