Redis与Scrapy
Redis与Scrapy
Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. ——Redis Home Page
1. 安装scrapy-redis模块
- pip install scrapy-redis
- easy_install scrapy-redis
2. 安装和运行Redis
- http://redis.io/download
- 运行Redis:
redis-server redis.conf
- 清空缓存:
redis-cli flushdb
3. Scrapy配置Redis
- settings.py配置Redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST =True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIS_URL = None
REDIS_HOST = '127.0.0.1' #REDIS服务器的ip
REDIS_PORT = 6379 #REDIS服务器端口
- Spider调用Redis
from scrapy_redis.spiders import RedisSpider
class xxx(RedisSpider):
..................
redis_key = 'username:password'
..................