Python并发编程-concurrent包
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.concurrent.futures包概述
3.2版本引入的模块。 异步并行任务编程模块，提供一个高级的异步可执行的便利接口。 提供了2个池执行器:
ThreadPoolExecutor 异步调用的线程池的Executor
ProcessPoolExecutor 异步调用的进程池的Executor
二.ThreadPoolExecutor对象
1>.首先需要定义一个池的执行器对象，Executor类子类对象。
ThreadPoolExecutor(max_workers=1)
池中至多创建max_workers个线程的池来同时异步执行，返回Executor实例
submit(fn, *args, **kwargs)
提交执行的函数及其参数，返回Future类的实例
shutdown(wait=True)
清理池
2>.Future类
done()
如果调用被成功的取消或者执行完成，返回True
cancelled()
如果调用被成功的取消，返回True
running()
如果正在运行且不能被取消，返回True
cancel()
尝试取消调用。如果已经执行且不能取消返回False，否则返回True
result(timeout=None)
取返回的结果，timeout为None，一直等待返回;timeout设置到期，抛出"concurrent.futures.TimeoutError"异常
exception(timeout=None)
取返回的异常，timeout为None，一直等待返回;timeout设置到期，抛出"concurrent.futures.TimeoutError"异常
3>.ThreadPoolExecutor案例实战
#！/usr/bin/env python
#_*_conding:utf-8_*_
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie import threading
from concurrent import futures
import logging
import time # 输出格式定义
FORMAT = '%(asctime)-15s\t [%(processName)s:%(threadName)s, %(process)d:%(thread)8d] %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT) def worker(n):
logging.info('begin to work-{}'.format(n))
time.sleep(5)
logging.info('finished {}'.format(n)) """
线程池并不着急创建线程，用的时候创建，创建后不清除，复用
线程池一旦创建了线程，就不需要频繁清除
"""
executer = futures.ThreadPoolExecutor(max_workers=3) #创建线程池，池容量为3 fs = []
for i in range(3):
future = executer.submit(worker, i)
fs.append(future) for i in range(3, 6):
future = executer.submit(worker, i)
fs.append(future) while True:
time.sleep(2)
logging.info(threading.enumerate()) flag = True
for f in fs: # 判断是否还有未完成的任务
logging.info(f.done())
flag = flag and f.done()
# if not flag: # 注释了这个if，输出的日志看的清楚些
# break
print("-" * 30) if flag:
executer.shutdown()
logging.info(threading.enumerate())
break
2019-12-04 19:11:33,108 [MainProcess:ThreadPoolExecutor-0_0, 16684:123145346408448] begin to work-0
2019-12-04 19:11:33,108 [MainProcess:ThreadPoolExecutor-0_1, 16684:123145351663616] begin to work-1
2019-12-04 19:11:33,109 [MainProcess:ThreadPoolExecutor-0_2, 16684:123145356918784] begin to work-2
------------------------------
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(ThreadPoolExecutor-0_0, started daemon 123145346408448)>, <Thread(ThreadPoolExecutor-0_1, started daemon 123145351663616)>, <Thread(ThreadPoolExecutor-0_2, started daemon 123145356918784)>]
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:35,113 [MainProcess:MainThread, 16684:140736847635392] False
------------------------------
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(ThreadPoolExecutor-0_0, started daemon 123145346408448)>, <Thread(ThreadPoolExecutor-0_1, started daemon 123145351663616)>, <Thread(ThreadPoolExecutor-0_2, started daemon 123145356918784)>]
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:37,114 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:38,111 [MainProcess:ThreadPoolExecutor-0_0, 16684:123145346408448] finished 0
2019-12-04 19:11:38,111 [MainProcess:ThreadPoolExecutor-0_0, 16684:123145346408448] begin to work-3
2019-12-04 19:11:38,111 [MainProcess:ThreadPoolExecutor-0_1, 16684:123145351663616] finished 1
2019-12-04 19:11:38,112 [MainProcess:ThreadPoolExecutor-0_1, 16684:123145351663616] begin to work-4
2019-12-04 19:11:38,112 [MainProcess:ThreadPoolExecutor-0_2, 16684:123145356918784] finished 2
2019-12-04 19:11:38,112 [MainProcess:ThreadPoolExecutor-0_2, 16684:123145356918784] begin to work-5
------------------------------
2019-12-04 19:11:39,115 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(ThreadPoolExecutor-0_0, started daemon 123145346408448)>, <Thread(ThreadPoolExecutor-0_1, started daemon 123145351663616)>, <Thread(ThreadPoolExecutor-0_2, started daemon 123145356918784)>]
2019-12-04 19:11:39,115 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:39,115 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:39,115 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:39,115 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:39,116 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:39,116 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(ThreadPoolExecutor-0_0, started daemon 123145346408448)>, <Thread(ThreadPoolExecutor-0_1, started daemon 123145351663616)>, <Thread(ThreadPoolExecutor-0_2, started daemon 123145356918784)>]
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] True
------------------------------
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:41,120 [MainProcess:MainThread, 16684:140736847635392] False
2019-12-04 19:11:43,116 [MainProcess:ThreadPoolExecutor-0_2, 16684:123145356918784] finished 5
2019-12-04 19:11:43,116 [MainProcess:ThreadPoolExecutor-0_0, 16684:123145346408448] finished 3
2019-12-04 19:11:43,116 [MainProcess:ThreadPoolExecutor-0_1, 16684:123145351663616] finished 4
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(ThreadPoolExecutor-0_0, started daemon 123145346408448)>, <Thread(ThreadPoolExecutor-0_1, started daemon 123145351663616)>, <Thread(ThreadPoolExecutor-0_2, started daemon 123145356918784)>]
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,122 [MainProcess:MainThread, 16684:140736847635392] True
2019-12-04 19:11:43,123 [MainProcess:MainThread, 16684:140736847635392] [<_MainThread(MainThread, started 140736847635392)>]
------------------------------
以上代码执行结果戳这里
三.ProcessPoolExecutor对象
#！/usr/bin/env python
#_*_conding:utf-8_*_
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie import threading
from concurrent import futures
import logging
import time
# 输出格式定义
FORMAT = '%(asctime)-15s\t [%(processName)s:%(threadName)s, %(process)d:%(thread)8d] %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT) def worker(n):
logging.info('begin to work-{}'.format(n))
time.sleep(5)
logging.info('finished {}'.format(n)) if __name__ == '__main__':
"""
方法一样。就是使用多进程完成。
"""
executer = futures.ProcessPoolExecutor(max_workers=3) # 创建进程池，池容量为3 fs = []
for i in range(3):
future = executer.submit(worker, i)
fs.append(future) for i in range(3, 6):
future = executer.submit(worker, i)
fs.append(future) while True:
time.sleep(2)
logging.info(threading.enumerate()) flag = True
for f in fs: # 判断是否还有未完成的任务
logging.info(f.done())
flag = flag and f.done()
#if not flag: # 注释了这个if，输出的日志看的清楚些
# break
print('-' * 30) if flag:
executer.shutdown() # 清理池。除非不用了，否则不用频繁清理池
logging.info(threading.enumerate()) # 多进程时看主线程已没有必要了
break
2019-12-04 19:17:23,219 [Process-1:MainThread, 16714:140736847635392] begin to work-0
2019-12-04 19:17:23,219 [Process-2:MainThread, 16715:140736847635392] begin to work-1
2019-12-04 19:17:23,219 [Process-3:MainThread, 16716:140736847635392] begin to work-2
------------------------------
2019-12-04 19:17:25,222 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145351159808)>, <Thread(QueueFeederThread, started daemon 123145356414976)>]
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:25,223 [MainProcess:MainThread, 16713:140736847635392] False
------------------------------
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145351159808)>, <Thread(QueueFeederThread, started daemon 123145356414976)>]
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:27,227 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:28,220 [Process-2:MainThread, 16715:140736847635392] finished 1
2019-12-04 19:17:28,220 [Process-1:MainThread, 16714:140736847635392] finished 0
2019-12-04 19:17:28,221 [Process-3:MainThread, 16716:140736847635392] finished 2
2019-12-04 19:17:28,221 [Process-2:MainThread, 16715:140736847635392] begin to work-3
2019-12-04 19:17:28,222 [Process-1:MainThread, 16714:140736847635392] begin to work-4
2019-12-04 19:17:28,222 [Process-3:MainThread, 16716:140736847635392] begin to work-5
2019-12-04 19:17:29,232 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145351159808)>, <Thread(QueueFeederThread, started daemon 123145356414976)>]
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:29,233 [MainProcess:MainThread, 16713:140736847635392] False
------------------------------
2019-12-04 19:17:31,237 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145351159808)>, <Thread(QueueFeederThread, started daemon 123145356414976)>]
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] False
2019-12-04 19:17:31,238 [MainProcess:MainThread, 16713:140736847635392] False
------------------------------
2019-12-04 19:17:33,223 [Process-2:MainThread, 16715:140736847635392] finished 3
2019-12-04 19:17:33,223 [Process-1:MainThread, 16714:140736847635392] finished 4
2019-12-04 19:17:33,224 [Process-3:MainThread, 16716:140736847635392] finished 5
------------------------------
2019-12-04 19:17:33,240 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145351159808)>, <Thread(QueueFeederThread, started daemon 123145356414976)>]
2019-12-04 19:17:33,240 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,240 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,240 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,240 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,241 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,241 [MainProcess:MainThread, 16713:140736847635392] True
2019-12-04 19:17:33,245 [MainProcess:MainThread, 16713:140736847635392] [<_MainThread(MainThread, started 140736847635392)>]
以上代码执行结果戳这里
四.支持上下文管理
#！/usr/bin/env python
#_*_conding:utf-8_*_
#@author :yinzhengjie
#blog:http://www.cnblogs.com/yinzhengjie """
concurrent.futures.ProcessPoolExecutor继承自concurrent.futures._base.Executor，而父类有 __enter__ 、__exit__方法，支持上下文管理。可以使用with语句。 __exit__方法本质还是调用的shutdown(wait=True)，就是一直阻塞到所有运行的任务完成 使用方法
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(pow, 323, 1235)
print(future.result())
""" import threading
from concurrent import futures
import logging
import time # 输出格式定义
FORMAT = '%(asctime)-15s\t [%(processName)s:%(threadName)s, %(process)d:%(thread)8d] %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT) def worker(n):
logging.info('begin to work-{}'.format(n))
time.sleep(5)
logging.info('finished {}'.format(n)) if __name__ == '__main__':
executer = futures.ProcessPoolExecutor(max_workers=3) # 创建进程池，池容量为3 with executer: #使用上下文改造进程池的例子，增加返回计算的结果
fs = []
for i in range(3):
future = executer.submit(worker, i)
fs.append(future) for i in range(3, 6):
future = executer.submit(worker, i)
fs.append(future) while True:
time.sleep(2)
logging.info(threading.enumerate())
flag = True
for f in fs: # 判断是否还有未完成的任务
logging.info(f.done())
flag = flag and f.done()
#if not flag: # 注释了这个if，输出的日志看的清楚些
# break print('-' * 30)
if flag:
break
#executer.shutdown() # 上下文清理了资源
logging.info('===end====')
logging.info(threading.enumerate()) # 多进程时看主线程已没有必要了
2019-12-04 19:25:33,520 [Process-1:MainThread, 16781:140736847635392] begin to work-0
2019-12-04 19:25:33,520 [Process-2:MainThread, 16782:140736847635392] begin to work-1
2019-12-04 19:25:33,521 [Process-3:MainThread, 16783:140736847635392] begin to work-2
2019-12-04 19:25:35,522 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145557798912)>, <Thread(QueueFeederThread, started daemon 123145563054080)>]
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:35,523 [MainProcess:MainThread, 16780:140736847635392] False
------------------------------
------------------------------
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145557798912)>, <Thread(QueueFeederThread, started daemon 123145563054080)>]
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:37,527 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:38,521 [Process-1:MainThread, 16781:140736847635392] finished 0
2019-12-04 19:25:38,522 [Process-1:MainThread, 16781:140736847635392] begin to work-3
2019-12-04 19:25:38,522 [Process-2:MainThread, 16782:140736847635392] finished 1
2019-12-04 19:25:38,522 [Process-3:MainThread, 16783:140736847635392] finished 2
2019-12-04 19:25:38,523 [Process-2:MainThread, 16782:140736847635392] begin to work-4
2019-12-04 19:25:38,524 [Process-3:MainThread, 16783:140736847635392] begin to work-5
------------------------------
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145557798912)>, <Thread(QueueFeederThread, started daemon 123145563054080)>]
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:39,532 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:39,533 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145557798912)>, <Thread(QueueFeederThread, started daemon 123145563054080)>]
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] True
------------------------------
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:41,536 [MainProcess:MainThread, 16780:140736847635392] False
2019-12-04 19:25:43,522 [Process-1:MainThread, 16781:140736847635392] finished 3
2019-12-04 19:25:43,524 [Process-2:MainThread, 16782:140736847635392] finished 4
2019-12-04 19:25:43,524 [Process-3:MainThread, 16783:140736847635392] finished 5
2019-12-04 19:25:43,537 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>, <Thread(Thread-1, started daemon 123145557798912)>, <Thread(QueueFeederThread, started daemon 123145563054080)>]
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
2019-12-04 19:25:43,538 [MainProcess:MainThread, 16780:140736847635392] True
------------------------------
2019-12-04 19:25:43,541 [MainProcess:MainThread, 16780:140736847635392] ===end====
2019-12-04 19:25:43,542 [MainProcess:MainThread, 16780:140736847635392] [<_MainThread(MainThread, started 140736847635392)>]
以上代码执行结果戳这里
五.总结
该库统一了线程池、进程池调用，简化了编程。 是Python简单的思想哲学的体现。
唯一的缺点:无法设置线程名称。但这都不值一提。