Python urllib2:无法分配请求的地址

问题描述

我使用带有代理的 urllib2 发送数千个请求.我在执行时收到许多以下错误:

I am sending thousands of requests using urllib2 with proxies. I have received many of the following error on execution:

urlopen error [Errno 99] Cannot assign requested address

我在此处读到这可能是由于套接字已被绑定.是这样吗?有关如何解决此问题的任何建议?

I read here that it may be due to a socket already being bonded. Is that the case? Any suggestions on how to fix this?

推荐答案

这是我之前准备的类似问题的答案......重用套接字时套接字使用错误

Here is an answer to a similar looking question that I prepared earlier.... much earlier...Socket in use error when reusing sockets

错误是不同的，但潜在的问题可能是相同的:您正在消耗所有可用端口并尝试在 TIME_WAIT 状态结束之前重用它们.

The error is different, but the underlying problem is probably the same: you are consuming all available ports and trying to reuse them before the TIME_WAIT state has ended.

如果它在您的应用程序的能力/规范范围内，一个明显的策略是控制连接速率以避免这种情况.

If it is within the capability/spec for your application, one obvious strategy is to control the rate of connections to avoid this situation.

或者，您可以使用 httplib 模块.httplib.HTTPConnection() 允许您指定一个 source_address 元组，您可以使用它指定建立连接的端口，例如这将从 localhost:9999 连接到 localhost:1234:

Alternatively, you could use the httplib module. httplib.HTTPConnection() lets you specify a source_address tuple with which you can specify the port from which to make the connection, e.g. this will connect to localhost:1234 from localhost:9999:

import httplib
conn = httplib.HTTPConnection('localhost:1234', source_address=('localhost',9999))
conn.request('GET', '/index.html')

然后是管理源端口分配的问题，如我之前的回答中所述.如果您使用的是 Windows，则可以使用此方法绕过默认端口范围 1024-5000.

Then it is a matter of managing the source port assignment as described in my earlier answer. If you are on Windows you can use this method to get around the default range of ports 1024-5000.

(当然)您可以建立多少个连接是有上限的，什么样的应用程序需要快速连续建立数千个连接是有问题的.

There is (of course), an upper limit to how many connections you are going to be able to make and it is questionable what sort of an application would require making thousands of connections in rapid succession.

这篇关于Python urllib2:无法分配请求的地址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！