我正在Linux上运行一个应用程序foo。在Bash脚本/终端提示符下,我的应用程序使用以下命令运行多线程:

$ foo -config x.ini -threads 4 < inputfile

系统监视器和最高报告foo的平均CPU负载约为380%(四核计算机)。我使用以下命令在Python 2.6x中重新创建了此功能:
proc = subprocess.Popen("foo -config x.ini -threads 4", \
        shell=True, stdin=subprocess.PIPE, \
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
mylist = ['this','is','my','test','app','.']
for line in mylist:
    txterr = ''
    proc.stdin.write(line.strip()+'\n')
    while not proc.poll() and not txterr.count('Finished'):
        txterr += subproc.stderr.readline()
    print proc.stdout.readline().strip(),

Foo运行速度较慢,并且top报告CPU负载为100%。 Foo在shell = False的情况下也可以正常运行,但是仍然很慢:
proc = subprocess.Popen("foo -config x.ini -threads 4".split(), \
        shell=False, stdin=subprocess.PIPE, \
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

有没有办法让Python子进程连续填充所有线程?

最佳答案

如果您的python脚本不能足够快地提供foo进程,那么您可以将读取stdout,stderr卸载到线程中:

from Queue import Empty, Queue
from subprocess import PIPE, Popen
from threading import Thread

def start_thread(target, *args):
    t = Thread(target=target, args=args)
    t.daemon = True
    t.start()
    return t

def signal_completion(queue, stderr):
    for line in iter(stderr.readline, ''):
        if 'Finished' in line:
           queue.put(1) # signal completion
    stderr.close()

def print_stdout(q, stdout):
    """Print stdout upon receiving a signal."""
    text = []
    for line in iter(stdout.readline, ''):
        if not q.empty():
           try: q.get_nowait()
           except Empty:
               text.append(line) # queue is empty
           else: # received completion signal
               print ''.join(text),
               text = []
               q.task_done()
        else: # buffer stdout until the task is finished
            text.append(line)
    stdout.close()
    if text: print ''.join(text), # print the rest unconditionally

queue = Queue()
proc = Popen("foo -config x.ini -threads 4".split(), bufsize=1,
             stdin=PIPE, stdout=PIPE, stderr=PIPE)
threads =  [start_thread(print_stdout, queue, proc.stdout)]
threads += [start_thread(signal_completion, queue, proc.stderr)]

mylist = ['this','is','my','test','app','.']
for line in mylist:
    proc.stdin.write(line.strip()+'\n')
proc.stdin.close()
proc.wait()
for t in threads: t.join() # wait for stdout

关于python - Python如何连续填充子进程的多个线程?,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/4802119/

10-16 11:35
查看更多