urllib2和cookielib线程安全

问题描述

据我所知cookielib不是线程安全的；但随后该帖子又说是5岁，所以可能是错误的.

As far as I've been able to tell cookielib isnt thread safe; but then again the post stating so is five years old, so it might be wrong.

尽管如此，我一直在想-如果我生成这样的类:

Nevertheless, I've been wondering - If I spawn a class like this:

class Acc:
    jar = cookielib.CookieJar()
    cookie = urllib2.HTTPCookieProcessor(jar)       
    opener = urllib2.build_opener(cookie)

    headers = {}
    def __init__ (self,login,password):
        self.user = login
        self.password = password

    def login(self):
        return False # Some magic, irrelevant

    def fetch(self,url):
        req = urllib2.Request(url,None,self.headers)
        res = self.opener.open(req)
        return res.read()

对于每个工作线程，它都能工作吗? (或者有更好的方法吗?)每个线程都会使用自己的帐户；因此，工人不会共享他们的cookie的事实不是问题.

for each worker thread, would it work? (or is there a better approach?) Each thread would use it's own account; so the fact that workers wouldn't share their cookies is not a problem.

推荐答案

您要使用 pycurl ( libcurl 的python接口).它是线程安全的，支持cookie，https等.该接口有点奇怪，但是需要一点时间来习惯.

You want to use pycurl (the python interface to libcurl). It's thread-safe, supports cookies, https, etc.. The interface is a bit strange, but it just takes a bit of getting used to.

我只使用了带有HTTPBasicAuth + SSL的pycurl，但确实找到了使用pycurl和cookie的示例此处.我相信您需要将pycurl.COOKIEFILE(第74行)和pycurl.COOKIEJAR(第82行)更新为具有唯一的名称(也许是从id(self.crl)键入).

I've only used pycurl w/ HTTPBasicAuth + SSL, but I did find an example using pycurl and cookies here. I believe you'll need to update the pycurl.COOKIEFILE (line 74) and pycurl.COOKIEJAR (line 82) to have some unique name (maybe keying off of id(self.crl)).

我记得，您需要为每个请求创建一个新的pycurl.Curl()，以维护线程安全.

As I remember, you'll need to create a new pycurl.Curl() for each request to maintain thread safety.

这篇关于urllib2和cookielib线程安全的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！