根据alexce的说明here安装了docker,scrapsjs和splash。
然后,运行docker run -p 8050:8050 scrapinghub/splash
,我得到以下输出:
2016-05-08 17:17:45+0000 [-] Log opened.
2016-05-08 17:17:45.978866 [-] Splash version: 2.1
2016-05-08 17:17:45.979553 [-] Qt 5.5.1, PyQt 5.5.1, WebKit 538.1, sip 4.17, Twisted 16.1.1, Lua 5.2
2016-05-08 17:17:45.980138 [-] Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4]
2016-05-08 17:17:45.980401 [-] Open files limit: 1048576
2016-05-08 17:17:45.981020 [-] Can't bump open files limit
2016-05-08 17:17:46.086232 [-] Xvfb is started: ['Xvfb', ':1', '-screen', '0', '1024x768x24']
2016-05-08 17:17:46.161902 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2016-05-08 17:17:46.260357 [-] verbosity=1
2016-05-08 17:17:46.260607 [-] slots=50
2016-05-08 17:17:46.261170 [-] argument_cache_max_entries=500
2016-05-08 17:17:46.262476 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2016-05-08 17:17:46.264565 [-] Site starting on 8050
2016-05-08 17:17:46.265203 [-] Starting factory <twisted.web.server.Site object at 0x7f270ec81e10>
它卡在那里。我尝试根据指示here进行故障排除(重新安装docker,验证vm是否正在运行,重新生成证书,设置env),但仍然没有执行任何操作。
My settings file和Spider
当在
meta=
start_requests`中运行不带yield scrapy.Request' call within
参数的蜘蛛时,蜘蛛运行良好(除了未捕获动态字段),因此我不确定这是否与docker或splash有关。提前致谢。
最佳答案
当在Docker上运行Splash时,控制台首先确实会“卡在那里”。 Splash正在等待通过其端点之一的请求。
$ sudo docker run -p 8050:8050 scrapinghub/splash
2016-05-09 10:21:42+0000 [-] Log opened.
2016-05-09 10:21:42.773541 [-] Splash version: 2.1
2016-05-09 10:21:42.774298 [-] Qt 5.5.1, PyQt 5.5.1, WebKit 538.1, sip 4.17, Twisted 16.1.1, Lua 5.2
2016-05-09 10:21:42.774453 [-] Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4]
2016-05-09 10:21:42.774632 [-] Open files limit: 1048576
2016-05-09 10:21:42.774842 [-] Can't bump open files limit
2016-05-09 10:21:42.879868 [-] Xvfb is started: ['Xvfb', ':1', '-screen', '0', '1024x768x24']
2016-05-09 10:21:43.072351 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2016-05-09 10:21:43.214478 [-] verbosity=1
2016-05-09 10:21:43.214617 [-] slots=50
2016-05-09 10:21:43.214703 [-] argument_cache_max_entries=500
2016-05-09 10:21:43.215195 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2016-05-09 10:21:43.217494 [-] Site starting on 8050
2016-05-09 10:21:43.217635 [-] Starting factory <twisted.web.server.Site object at 0x7f529d0fee48>
要测试Splash是否运行正常,请在http://localhost:8050/上尝试其Web UI
您应该看到类似以下内容:
然后,您可以尝试输入一些URL,然后单击“渲染我!”。
关于python - Scrapy Splash服务器未通过docker run启动,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/37102952/