带有多线程的python-requests

问题描述

我正在创建一个HTTP客户端，该客户端每秒可以生成数百个连接，并在每个连接上最多发送10个请求.我正在使用线程，因此可以实现并发.这是我的代码:

I am working on creating a HTTP client which can generate hundreds of connections each second and send up to 10 requests on each of those connections. I am using threading so concurrency can be achieved.Here is my code:

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://20.20.1.2/tempurl.html')
            if response1.status_code == 200:
                client_notify('r')
        except(exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout) as Err:
            client_notify('F')
            break
        requestCounter += 1

def main():
    for q in range(connectionPerSec):
        s1 = requests.session()
        t1 = threading.Thread(target=generate_req, args=(s1,))
        t1.start()

问题:

1. 在requestRate = 1的情况下，扩展速度没有超过200个/秒.我在同一台客户端计算机上针对服务器运行了其他可用的HTTP客户端，测试运行正常并且可以扩展.

1. It is not scaling above 200 connections/sec with requestRate = 1. I ran other available HTTP clients on the same client machine and against the server, test runs fine and it is able to scale.

当requestRate = 10时，连接/秒降低到30.原因:无法每秒创建目标数量的线程.

When requestRate = 10, connections/sec drops to 30.Reason: Not able to create targeted number of threads every second.

对于问题2，客户端计算机无法创建足够的请求会话并启动新线程.一旦requestRate设置为大于1，事情就会开始崩溃.我怀疑这与请求使用的HTTP连接池有关.

For issue #2, client machine is not able to create enough request sessions and start new threads. As soon as requestRate is set to more than 1, things start to fall apart.I am suspecting it has something to do with HTTP connection pooling which requests uses.

请在这里提出我在做什么错

Please suggest what am I doing wrong here.

推荐答案

我无法使事情崩溃，但是以下代码具有一些新功能:

I wasn't able to get things to fall apart, however the following code has some new features:

1)扩展日志记录，包括特定的每线程信息

1) extended logging, including specific per-thread information

2)最后所有join()线程，以确保父进程不会使它们挂起

2) all threads join()ed at the end to make sure the parent process doesntt leave them hanging

3)多线程print倾向于交织消息，这可能很麻烦.此版本使用yield，因此以后的版本可以接受消息并清晰打印.

3) multithreaded print tends to interleave the messages, which can be unwieldy. This version uses yield so a future version can accept the messages and print them clearly.

import exceptions, requests, threading, time

requestRate = 1
connectionPerSec = 2


def client_notify(msg):
    return time.time(), threading.current_thread().name, msg

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://127.0.0.1/')
            if response1.status_code == 200:
                print client_notify('r')
        except (exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout):
            print client_notify('F')
            break
        requestCounter += 1

def main():
    for cnum in range(connectionPerSec):
        s1 = requests.session()
        th = threading.Thread(
            target=generate_req, args=(s1,),
            name='thread-{:03d}'.format(cnum),
        )
        th.start()

    for th in threading.enumerate():
        if th != threading.current_thread():
            th.join()


if __name__=='__main__':
    main()

输出

(1407275951.954147, 'thread-000', 'r')
(1407275951.95479, 'thread-001', 'r')

这篇关于带有多线程的python-requests的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！