并行执行类方法

问题描述

我需要并行执行同一类的许多实例的方法.为此，我尝试使用 Process.start() 和 < multiprocessing中的c1> 命令模块.

I need to execute in parallel a method of many instances of the same class. For doing this I'm trying to use the Process.start() and the Process.join() commands from the multiprocessing module.

例如一个班级:

class test:
     def __init__(self):
     ...
     ...
     def method(self):
     ...
     ...

其中，method修改某些类变量.如果我创建了该类的两个实例:

where method modifies some of the class variables. If I make two instances of the class:

t1=test()
t2=test()

并执行:

from multiprocessing import Process
pr1=Process(target=t1.method, args=(,))
pr2=Process(target=t2.method, args=(,))
pr1.start()
pr2.start()
pr1.join()
pr2.join()

该类实例的变量不会更新(整个代码太长了，无法粘贴到这里，但这是个主意).

the variables of the instances of the class are not updated (the whole code is too long to be pasted here but this is the idea).

有什么办法可以做到这一点?谢谢

Is there any way to achieve this?Thank you

推荐答案

在子进程中调用obj.method时，子进程将在obj中获取每个实例变量的单独副本.因此，您在子代中对它们所做的更改不会反映在父代中.您需要通过 multiprocessing.Queue ，以使更改在父级生效:

When you call obj.method in a child process, the child process is getting its own separate copy of each instance variable in obj. So, the changes you make to them in the child will not be reflected in the parent. You'll need to explicitly pass the changed values back to the parent via a multiprocessing.Queue in order to make the changes take effect the parent:

from multiprocessing import Process, Queue
q1 = Queue()
q2 = Queue()
pr1 = Process(target=t1.method, args=(q1,))
pr2 = Process(target=t2.method, args=(q2,))
pr1.start()
pr2.start()
out1 = q1.get()
out2 = q2.get()
t1.blah = out1
t2.blah = out2
pr1.join()
pr2.join()

其他选项将使您需要更改的实例变量 multiprocessing.Value 实例，或 multiprocessing.Manager Proxy实例.这样，您将在子级中进行的更改会自动反映在父级中.但这是以增加使用父代变量的开销为代价的.

Other options would be to make the instance variables you need to change multiprocessing.Value instances, or multiprocessing.Manager Proxy instances. That way, the changes you make in the children would be reflected in the parent automatically. But that comes at the cost of adding overhead to using the variables in the parent.

这是使用multiprocessing.Manager的示例.这不起作用:

Here's an example using multiprocessing.Manager. This doesn't work:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.some_list = []  # Normal list

    def method(self):
        self.some_list.append(123)  # This change gets lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

输出:

[]
[]

这有效:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.manager = multiprocessing.Manager()
       self.some_list = self.manager.list()  # Shared Proxy to a list

    def method(self):
        self.some_list.append(123) # This change won't be lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

输出:

[123]
[123]

请记住，multiprocessing.Manager启动一个子进程来管理您创建的所有共享实例，并且每次您访问Proxy实例之一时，实际上是在对Manager过程.

Just keep in mind that a multiprocessing.Manager starts a child process to manage all the shared instances you create, and that every time you access one of the Proxy instances, you're actually making an IPC call to the Manager process.

这篇关于并行执行类方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！