问题描述
看下面的代码:
import requests
import grequests
import requests_cache
requests_cache.install_cache('bla')
urls = [
'http://www.heroku.com',
'http://python-tablib.org',
'http://httpbin.org',
'http://python-requests.org',
'http://kennethreitz.com'
]
rs = (grequests.get(u) for u in urls)
results = grequests.map(rs)
我希望执行完此操作后,我会在当前目录中找到 bla.sqlite
文件并执行
I would expect that after executing this I will find bla.sqlite
file in the current directory and executing
results = grequests.map(rs)
将更快,因为数据将从sqlite缓存中获取。不幸的是,这不是真的,根本没有创建文件,也没有加速。当我使用grequests insetead的请求时,一切正常。所以问题是标题说:是否可以使grequests和requests_cache一起工作?,如果可以,怎么办?
will be MUCH faster because data will be taken from sqlite cache. Unfortunately this is not true, file wasn't created at all and there is no speedup. When I use requests insetead of grequests everything works fine. So the question is a title says: Is it possible to make grequests and requests_cache work together? and if yes, how?
推荐答案
requests_cache.install_cache()
函数修补了 requests.Session
,但您已经导入了 grequests
,它用于:
The requests_cache.install_cache()
function patches requests.Session
, but you already imported grequests
, which used:
from requests import Session
因此, grequests
从未使用修补的会话对象。
As such, grequests
never uses the patched session object.
在安装缓存后将导入移动到 :
Move the import to after you installed the cache:
import requests_cache
requests_cache.install_cache('bla')
import grequests
或者,创建一个,并将其作为传递给
参数: grequests.get()
(及相关)方法会话
Alternatively, create a CachedSession
object and pass that in to the grequests.get()
(and related) methods as the session
parameter:
import grequests
import requests_cache
session = requests_cache.CachedSession('bla')
urls = [
'http://www.heroku.com',
'http://python-tablib.org',
'http://httpbin.org',
'http://python-requests.org',
'http://kennethreitz.com'
]
rs = (grequests.get(u, session=session) for u in urls)
results = grequests.map(rs)
请注意,缓存存储后端可能无法安全处理并发访问。例如, sqlite
后端使用线程锁,这很可能会发生冲突。
Take into account that the cache storage backend may not be able to handle the corcurrent access safely. The sqlite
backend uses a thread lock, for example, which might well clash.
这篇关于是否可以使grequests和request_cache一起工作?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!