Redis与Scrapy

Redis与Scrapy


Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. ——Redis Home Page


1. 安装scrapy-redis模块

  • pip install scrapy-redis
  • easy_install scrapy-redis

2. 安装和运行Redis

3. Scrapy配置Redis

  • settings.py配置Redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST =True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIS_URL = None
REDIS_HOST = '127.0.0.1' #REDIS服务器的ip
REDIS_PORT = 6379 #REDIS服务器端口
  • Spider调用Redis
from scrapy_redis.spiders import RedisSpider
class xxx(RedisSpider):
..................
redis_key = 'username:password'
..................
04-25 07:09