问题描述
如何使用线程和子流程模块生成并行bash流程?当我启动线程ala时，这里的第一个答案是:如何在Python中使用线程?进程按顺序运行，而不是并行运行.
How does one use the threading and subprocess modules to spawn parallel bash processes? When I start threads ala the first answer here: How to use threading in Python?, the bash processes run sequentially instead of in parallel.
推荐答案
您不需要线程来并行运行子流程:
You don't need threads to run subprocesses in parallel:
from subprocess import Popen
commands = [
'date; ls -l; sleep 1; date',
'date; sleep 5; date',
'date; df -h; sleep 3; date',
'date; hostname; sleep 2; date',
'date; uname -a; date',
]
# run in parallel
processes = [Popen(cmd, shell=True) for cmd in commands]
# do other things here..
# wait for completion
for p in processes: p.wait()
要限制并发命令的数量，可以使用使用线程并提供与multiprocessing.dummy.Pool -pool-of-workers"rel =" noreferrer> multiprocessing.Pool 使用进程:
To limit number of concurrent commands you could use multiprocessing.dummy.Pool that uses threads and provides the same interface as multiprocessing.Pool that uses processes:
from functools import partial
from multiprocessing.dummy import Pool
from subprocess import call
pool = Pool(2) # two concurrent commands at a time
for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)):
if returncode != 0:
print("%d command failed: %d" % (i, returncode))
此答案演示了限制并发子进程数量的各种技术:它显示了multiprocessing.Pool，concurrent.futures，threading +基于队列的解决方案.
This answer demonstrates various techniques to limit number of concurrent subprocesses: it shows multiprocessing.Pool, concurrent.futures, threading + Queue -based solutions.
您可以在不使用线程/进程池的情况下限制并发子进程的数量:
You could limit the number of concurrent child processes without using a thread/process pool:
from subprocess import Popen
from itertools import islice
max_workers = 2 # no more than 2 concurrent processes
processes = (Popen(cmd, shell=True) for cmd in commands)
running_processes = list(islice(processes, max_workers)) # start new processes
while running_processes:
for i, process in enumerate(running_processes):
if process.poll() is not None: # the process has finished
running_processes[i] = next(processes, None) # start new process
if running_processes[i] is None: # no new processes
del running_processes[i]
break
在Unix上，您可以避免繁忙循环，并在阻止>，以等待所有子进程退出.
On Unix, you could avoid the busy loop and block on os.waitpid(-1, 0), to wait for any child process to exit.
这篇关于Python线程化多个bash子进程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！