如何添加第三方 Scrapy 中间件

问题描述

我正在使用scrapy 1.1.我想添加 scrapy-fake-user-agentScrapy 中间件，可以无缝和随机地旋转用户代理.用户代理字符串由 fake-useragent 模块提供.

I'm working with scrapy 1.1 . I want to add scrapy-fake-user-agent Scrapy middleware that would rotate user agents seamlessly and randomly. User Agent strings are supplied by the fake-useragent module.

按照网站上的指示，我有:

following the directions from the site, I have:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

但是当我运行它时，我得到:

However when I run it I get:

ImportError: No module named scrapy_fake_useragent.middleware

我做错了什么?

我在 virtualenv 中打开了一个 python shell:

edit: I opened a python shell in the virtualenv:

In[4]: import scrapy.downloadermiddlewares as x
In[5]: x.useragent.UserAgentMiddleware
Out[5]: scrapy.downloadermiddlewares.useragent.UserAgentMiddleware

In[7]: import scrapy_fake_useragent as x
In[8]: dir(x)
Out[8]: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

那么scrapy.downloadermiddlewares.useragent.UserAgentMiddleware"似乎可用但scrapy_fake_useragent.middleware.RandomUserAgentMiddleware"不可用?

So 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware 'seems to be available but not 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware' ?

推荐答案

我也遇到了同样的问题.后来我注意到我不在virutalenv里面.这些软件包安装在 virutalenv 中.当移动到 virtualenv 时，它很好.

I too had the same problem. Later I noticed that I am not inside the virutalenv. Those packages were installed in virutalenv. When moved into virtualenv, it is fine.

这篇关于如何添加第三方 Scrapy 中间件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！