我正在Linux上运行一个应用程序foo。在Bash脚本/终端提示符下,我的应用程序使用以下命令运行多线程:
$ foo -config x.ini -threads 4 < inputfile
系统监视器和最高报告foo的平均CPU负载约为380%(四核计算机)。我使用以下命令在Python 2.6x中重新创建了此功能:
proc = subprocess.Popen("foo -config x.ini -threads 4", \
shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
mylist = ['this','is','my','test','app','.']
for line in mylist:
txterr = ''
proc.stdin.write(line.strip()+'\n')
while not proc.poll() and not txterr.count('Finished'):
txterr += subproc.stderr.readline()
print proc.stdout.readline().strip(),
Foo运行速度较慢,并且top报告CPU负载为100%。 Foo在shell = False的情况下也可以正常运行,但是仍然很慢:
proc = subprocess.Popen("foo -config x.ini -threads 4".split(), \
shell=False, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
有没有办法让Python子进程连续填充所有线程?
最佳答案
如果您的python脚本不能足够快地提供foo
进程,那么您可以将读取stdout,stderr卸载到线程中:
from Queue import Empty, Queue
from subprocess import PIPE, Popen
from threading import Thread
def start_thread(target, *args):
t = Thread(target=target, args=args)
t.daemon = True
t.start()
return t
def signal_completion(queue, stderr):
for line in iter(stderr.readline, ''):
if 'Finished' in line:
queue.put(1) # signal completion
stderr.close()
def print_stdout(q, stdout):
"""Print stdout upon receiving a signal."""
text = []
for line in iter(stdout.readline, ''):
if not q.empty():
try: q.get_nowait()
except Empty:
text.append(line) # queue is empty
else: # received completion signal
print ''.join(text),
text = []
q.task_done()
else: # buffer stdout until the task is finished
text.append(line)
stdout.close()
if text: print ''.join(text), # print the rest unconditionally
queue = Queue()
proc = Popen("foo -config x.ini -threads 4".split(), bufsize=1,
stdin=PIPE, stdout=PIPE, stderr=PIPE)
threads = [start_thread(print_stdout, queue, proc.stdout)]
threads += [start_thread(signal_completion, queue, proc.stderr)]
mylist = ['this','is','my','test','app','.']
for line in mylist:
proc.stdin.write(line.strip()+'\n')
proc.stdin.close()
proc.wait()
for t in threads: t.join() # wait for stdout
关于python - Python如何连续填充子进程的多个线程?,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/4802119/