本文介绍了使用Python以真实顺序打印Parellel函数输出的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

希望为Python并行化脚本按顺序打印所有内容.注意c3在b2之前打印-乱序.有什么办法让下面的功能具有等待功能?如果重新运行,有时对于较短的批次,打印顺序是正确的.但是,正在寻找可重现的解决方案.

from joblib import Parallel, delayed, parallel_backend
import multiprocessing

testFrame = [['a',1], ['b', 2], ['c', 3]]

def testPrint(letr, numbr):
  print(letr + str(numbr))
  return letr + str(numbr)

with parallel_backend('multiprocessing'):
  num_cores = multiprocessing.cpu_count()
  results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0],
        numbr = testFrame[i][1]) for i in range(len(testFrame)))

print('##########')
for test in results:
  print(test)

输出:

b2
c3
a1
##########
a1
b2
c3

Seeking:
a1
b2
c3
##########
a1
b2
c3

如果您要对带有序列参数的任务/函数进行并行化(?),并且希望对结果进行重新排序以匹配原始序列的顺序,则可以将序列信息传递给任务/函数,该任务/函数将由参数返回.任务,可用于重建原始订单.

如果原始函数如下所示:

def f(arg):
    l,n = arg
    #do stuff
    time.sleep(random.uniform(.1,10.))
    result = f'{l}{n}'
    return result

将函数重构为接受序列信息,并将其与返回值一起传递.

def f(arg):
    indx, (l,n) = arg
    time.sleep(random.uniform(.1,10.))
    result = (indx,f'{l}{n}')
    return result

enumerate可用于将序列信息添加到数据序列中:

originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)

现在,参数的格式为(index,originalarg) ... (0, ('a',0'), (1, ('b',1)).

多进程的返回值看起来像这样(如果收集在列表中)-

[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]

很容易对每个结果的第一项key=lambda item: item[0]进行排序,并通过对results = [item[1] for item in results]进行排序后挑选出第二项来获得您真正想要的值.

Looking to print everything in order, for a Python parallelized script. Note the c3 is printed prior to the b2 -- out of order. Any way to make the below function with a wait feature? If you rerun, sometimes the print order is correct for shorter batches. However, looking for a reproducible solution to this issue.

from joblib import Parallel, delayed, parallel_backend
import multiprocessing

testFrame = [['a',1], ['b', 2], ['c', 3]]

def testPrint(letr, numbr):
  print(letr + str(numbr))
  return letr + str(numbr)

with parallel_backend('multiprocessing'):
  num_cores = multiprocessing.cpu_count()
  results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0],
        numbr = testFrame[i][1]) for i in range(len(testFrame)))

print('##########')
for test in results:
  print(test)

Output:

b2
c3
a1
##########
a1
b2
c3

Seeking:
a1
b2
c3
##########
a1
b2
c3
解决方案

Once you launch tasks in separate processes you no longer control the order of execution so you cannot expect the actions of those tasks to execute in any predictable order - especially if the tasks can take varying lengths of time.

If you are parallelizing(?) a task/function with a sequence of arguments and you want to reorder the results to match the order of the original sequence you can pass sequence information to the task/function that will be returned by the task and can be used to reconstruct the original order.

If the original function looks like this:

def f(arg):
    l,n = arg
    #do stuff
    time.sleep(random.uniform(.1,10.))
    result = f'{l}{n}'
    return result

Refactor the function to accept the sequence information and pass it through with the return value.

def f(arg):
    indx, (l,n) = arg
    time.sleep(random.uniform(.1,10.))
    result = (indx,f'{l}{n}')
    return result

enumerate could be used to add the sequence information to the sequence of data:

originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)

Now the arguments have the form (index,originalarg) ... (0, ('a',0'), (1, ('b',1)).

And the returned values from the multi-processes look like this (if collected in a list) -

[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]

Which is easily sorted on the first item of each result, key=lambda item: item[0], and the values you really want obtained by picking out the second items after sorting results = [item[1] for item in results].

这篇关于使用Python以真实顺序打印Parellel函数输出的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

08-04 02:26
查看更多