本文介绍了杂乱的外壳不起作用的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我对scrapy还是陌生的,所以我想尝试scrapy shell进行调试和学习,但是奇怪的是shell命令根本不起作用.
I am new to scrapy, then I want to try scrapy shell to debug and learn, but it's strange the shell command doesn't work at all.
- 似乎该网站已成功爬网,但没有输出更多内容.程序正在等待执行,似乎已死,我必须使用ctrl-c结束它.
您能帮忙找出问题所在吗?
can you help to figure out what's wrong?
我正在使用Anaconda + scrapy 1.0.3
I'm using Anaconda + scrapy 1.0.3
$ ping 135.251.157.2
Pinging 135.251.157.2 with 32 bytes of data:
Reply from 135.251.157.2: bytes=32 time=13ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Ping statistics for 135.251.157.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 13ms, Maximum = 14ms, Average = 13ms
$ scrapy shell "http://135.251.157.2/"
2016-01-28 21:35:18 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-28 21:35:18 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-28 21:35:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-28 21:35:18 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-28 21:35:19 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-28 21:35:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-28 21:35:19 [scrapy] INFO: Enabled item pipelines:
2016-01-28 21:35:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-28 21:35:19 [scrapy] INFO: Spider opened
2016-01-28 21:35:24 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
2016-01-28 21:35:24 [root] DEBUG: Using default logger
2016-01-28 21:35:24 [root] DEBUG: Using default logger
ctrl-c
$
推荐答案
我想关闭此线程,因为我发现根本原因与不同的终端有关.当我使用Git Bash时,它不起作用,但是如果我使用Anaconda Prompt,它会很好地工作.
I'd like to close this thread, as I find out the root cause is related with different terminal. When I use Git Bash, it doesn't work, but if I use Anaconda Prompt, it works quite well.
[Anaconda2] D:\SVN\tools\Spider\demo>scrapy shell "http://135.251.157.2/"
2016-01-29 13:40:33 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-29 13:40:33 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-29 13:40:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-29 13:40:33 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-29 13:40:33 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTrans
ferMiddleware, DownloaderStats
2016-01-29 13:40:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-29 13:40:33 [scrapy] INFO: Enabled item pipelines:
2016-01-29 13:40:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-29 13:40:33 [scrapy] INFO: Spider opened
2016-01-29 13:40:41 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
[s] Available Scrapy objects:
[s] crawler <scrapy.crawler.Crawler object at 0x0136B290>
[s] item {}
[s] request <GET http://135.251.157.2/>
[s] response <200 http://135.251.157.2/>
[s] settings <scrapy.settings.Settings object at 0x034204B0>
[s] spider <DefaultSpider 'default' at 0x3e3c6d0>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
2016-01-29 13:40:41 [root] DEBUG: Using default logger
2016-01-29 13:40:41 [root] DEBUG: Using default logger
In [1]:
这篇关于杂乱的外壳不起作用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!