我正在尝试检索一个网页,该网页需要从代理后面访问,并且还需要http身份验证:
$ wget -d --user=atwood --ask-password http://example.com/admin/admin.php
这很好,我将粘贴http头(下面是请求和响应)。
使用
python-requests
检索同一页将返回404错误:下面是python代码,前面是用户为Inactivist发布的非常好的方法:
url = 'http://example.com/admin/admin.php'
proxy_config = {
'http': '1.2.3.4',
'https': '1.2.3.4',
'ftp': '1.2.3.4'
}
head = {
'User-Agent': 'Wget/1.13.4 (linux-gnu)',
'Connection': 'Close',
'Proxy-Connection': 'Keep-Alive'
}
response = requests.get(url, auth=('atwood', 'hunter2'), proxies=proxy_config, headers=head)
print("Status code: %s" % (response.status_code, ))
print("URL: %s" % (response.url, ))
print(pformat(response.text))
下面是
requests
http头(请求和响应),它们实际上正确地返回了请求的页面:$ export http_proxy=http://1.2.3.4:3128
$ wget -d --user=atwood --ask-password http://example.com/admin/admin.php
Setting --user (user) to atwood
Setting --ask-password (askpassword) to 1
Password for user `atwood':
DEBUG output created by Wget 1.13.4 on linux-gnu.
URI encoding = `UTF-8'
URI encoding = `UTF-8'
--2014-01-07 11:15:59-- http://example.com/admin/admin.php
Host `example.com' has not issued a general basic challenge.
Connecting to 1.2.3.4:3128... connected.
Created socket 3.
Releasing 0x000000000159bf20 (new refcount 0).
Deleting unused 0x000000000159bf20.
---request begin---
GET http://example.com/admin/admin.php HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: example.com
Connection: Close
Proxy-Connection: Keep-Alive
---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 401 Unauthorized
Date: Tue, 07 Jan 2014 09:16:00 GMT
Server: Apache/2.2.21 (Linux/SUSE)
X-Powered-By: PHP/5.3.8
WWW-Authenticate: Basic realm="CONTACT-ADMIN"
Content-Length: 43
Content-Type: text/html
X-Cache: MISS from proxyServer
X-Cache-Lookup: MISS from proxyServer:3128
Via: 1.0 proxyServer (squid/3.1.19)
Connection: keep-alive
---response end---
401 Unauthorized
Registered socket 3 for persistent reuse.
Skipping 43 bytes of body: [Login incorrect, please try again: |||BAD|
] done.
Inserted `example.com' into basic_authed_hosts
Reusing existing connection to 1.2.3.4:3128.
Reusing fd 3.
---request begin---
GET http://example.com/admin/admin.php HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: example.com
Connection: Close
Proxy-Connection: Keep-Alive
Authorization: Basic NjY2Njp0cmlwczEyMw==
---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Date: Tue, 07 Jan 2014 09:16:00 GMT
Server: Apache/2.2.21 (Linux/SUSE)
X-Powered-By: PHP/5.3.8
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset=utf-8
X-Cache: MISS from proxyServer
X-Cache-Lookup: MISS from proxyServer:3128
Via: 1.0 proxyServer (squid/3.1.19)
Connection: close
---response end---
200 OK
URI content encoding = `utf-8'
Length: unspecified [text/html]
Saving to: `admin.php'
[ <=> ] 14,096 --.-K/s in 0.1s
2014-01-07 11:16:00 (92.8 KB/s) - `admin.php' saved [14096]
你可能会注意到我已经匿名了我正在获取的url。实际上,我已经三次检查了返回404的url是否与
wget
中的url相同。 最佳答案
看起来python中的代理端口与wget中使用的不同(我猜是3128与默认的8080)。