Spark docker安装在azure vm(centos 7.2)中，我想从本地计算机(Windows)访问hdfs。
我在Windows中运行curl -i -v -L http://52.234.XXX.XXX:50070/webhdfs/v1/user/helloworld.txt?op=OPEN，异常(exception)是
$ curl -i -v -L http://52.234.XXX.XXX:50070/webhdfs/v1/user/helloworld.txt?op=OP EN
* timeout on name lookup is not supported
* Trying 52.234.XXX.XXX...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 52.234.XXX.XXX (52.234.XXX.XXX) port 50070 (#0)
> GET /webhdfs/v1/user/helloworld.txt?op=OPEN HTTP/1.1
> Host: 52.234.XXX.XXX:50070
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 307 TEMPORARY_REDIRECT
< Cache-Control: no-cache
< Expires: Fri, 16 Mar 2018 02:16:37 GMT
< Date: Fri, 16 Mar 2018 02:16:37 GMT
< Pragma: no-cache
< Expires: Fri, 16 Mar 2018 02:16:37 GMT
< Date: Fri, 16 Mar 2018 02:16:37 GMT
< Pragma: no-cache
< Location: http://sandbox:50075/webhdfs/v1/user/helloworld.txt?op=OPEN&namenode rpcaddress=sandbox:9000&offset=0
< Content-Type: application/octet-stream
< Content-Length: 0
< Server: Jetty(6.1.26)
<
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host 52.234.227.186 left intact
* Issue another request to this URL: 'http://sandbox:50075/webhdfs/v1/user/hello world.txt?op=OPEN&namenoderpcaddress=sandbox:9000&offset=0'
* timeout on name lookup is not supported
* Trying 10.122.118.83...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0HT TP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Fri, 16 Mar 2018 02:16:37 GMT
Date: Fri, 16 Mar 2018 02:16:37 GMT
Pragma: no-cache
Expires: Fri, 16 Mar 2018 02:16:37 GMT
Date: Fri, 16 Mar 2018 02:16:37 GMT
Pragma: no-cache
Location: http://sandbox:50075/webhdfs/v1/user/helloworld.txt?op=OPEN&namenoderp caddress=sandbox:9000&offset=0
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26)
* connect to 10.122.118.83 port 50075 failed: Timed out
* Failed to connect to sandbox port 50075: Timed out
* Closing connection 1
curl: (7) Failed to connect to sandbox port 50075: Timed out
centos的公共(public)IP地址是:52.234.XXX.XXX。
是由未知ip '10 .122.118.83'引起的吗？是datanode的IP地址吗？我已经在azure vm网络设置中打开了这些端口。
我以
docker run -it -p 8088:8088 -p 8042:8042 -p 9000:9000 -p 8087:8087 -p 50070:50070 -p 50010:50010 -p 50075:50075 -p 50475:50475 --name sparkdocker -h sandbox --network=host sequenceiq/spark:1.6.0 bashhadoop的fs.defaultFS是'hdfs:// sandbox:9000'在同一个资源组中的centos和其他Azure机器访问hdfs(上传，下载，读取文件)没有问题。
Spark docker ifconfig:
docker0 Link encap:Ethernet HWaddr 02:42:D9:2A:5D:BB
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3889 (3.7 KiB) TX bytes:6674 (6.5 KiB)
eth0 Link encap:Ethernet HWaddr 00:0D:3A:14:B5:C1
inet addr:10.0.0.7 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:60543 errors:0 dropped:0 overruns:0 frame:0
TX packets:68081 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22930277 (21.8 MiB) TX bytes:11271703 (10.7 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:14779 errors:0 dropped:0 overruns:0 frame:0
TX packets:14779 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:4032619 (3.8 MiB) TX bytes:4032619 (3.8 MiB)
centos vm ifconfig:
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:d9:2a:5d:bb txqueuelen 0 (Ethernet)
RX packets 53 bytes 3889 (3.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 57 bytes 6674 (6.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.7 netmask 255.255.255.0 broadcast 10.0.0.255
ether 00:0d:3a:14:b5:c1 txqueuelen 1000 (Ethernet)
RX packets 60750 bytes 23017881 (21.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 68320 bytes 11310643 (10.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 14857 bytes 4042781 (3.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 14857 bytes 4042781 (3.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
最佳答案
如果要将远程主机名公开给外部网络，则不能使用sandbox的本地IP将其作为10.0.0.7。由于数据节点和名称节点之间的各种网络调用返回到远程网络中的外部客户端，因此在整个请求中，它都需要一个外部可解析的IP或DNS记录。
通过查看端口8088上的群集，对YARN服务也是如此
我相信这是core-site.xml中的设置，这需要像hdfs://external.namenode.fqdn:port这样的东西
fs.default.name
并且在hdfs-site.xml中将两者都设置为true-因为在云环境中，主机名通常是静态的，而IP可以更改。另外，在Azure网络中，节点知道如何通信，但是在群集之外，内部DNS名称无法解析
dfs.client.use.datanode.hostname
dfs.datanode.use.datanode.hostname
如果您在Azure中运行，我可能建议您仅使用HD见解，而不是使用单个datanode沙箱
无论如何，您都不需要远程Spark实例。您可以在本地开发。将该Spark应用程序部署到远程YARN(或Spark Standalone)集群。您也不需要HDFS ... Spark可以从Azure blob存储读取，并且可以在独立的调度程序中运行
另一个建议:永远不要打开不安全Hadoop集群的所有端口，并将公共(public)IP发布到Web。请在一端使用SSH转发以安全地连接到Azure网络
关于azure - 从本地计算机访问以在Azure虚拟机中触发Docker，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/49312266/