如何在多进程中杀死所有Pool工人?

问题描述

我想停止来自单个工作程序的所有线程.

I want to stop all threads from a single worker.

我有一个线程池，其中有10个工人:

I have a thread pool with 10 workers:

def myfunction(i):
    print(i)
    if (i == 20):
        sys.exit()

p = multiprocessing.Pool(10, init_worker)

for i in range(100):
    p.apply_async(myfunction, (i,))

在完成所有100次迭代之前，我的程序没有停止，其他进程继续运行.我想完全从调用sys.exit()的线程内部停止池.当前编写的方式只会停止调用sys.exit()的工作程序.

My program does not stop and the other processes continue working until all 100 iterations are complete. I want to stop the pool entirely from inside the thread that calls sys.exit(). The way it is currently written will only stop the worker that calls sys.exit().

推荐答案

这不符合您的预期目的，因为在工作进程中调用sys.exit()只会终止工作进程.它对父流程或其他工作程序没有影响，因为它们是独立的流程，并且提高SystemExit仅影响当前流程.您需要向父进程发送一个信号，告诉它应该关闭.一种针对您的用例的方法是使用 Event 在 multiprocessing.Manager 服务器:

This isn't working the way you're intending because calling sys.exit() in a worker process will only terminate the worker. It has no effect on the parent process or the other workers, because they're separate processes and raising SystemExit only affects the current process. You need to send a signal back the parent process to tell it that it should shut down. One way to do this for your use-case would be to use an Event created in a multiprocessing.Manager server:

import multiprocessing

def myfunction(i, event):
    if not event.is_set():
        print i
    if i == 20:
        event.set()

if __name__ == "__main__":
    p= multiprocessing.Pool(10)
    m = multiprocessing.Manager()
    event = m.Event()
    for i in range(100):
        p.apply_async(myfunction , (i, event))
    p.close()

    event.wait()  # We'll block here until a worker calls `event.set()`
    p.terminate() # Terminate all processes in the Pool

输出:

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

正如卢克的回答所指出的，这里有一个竞赛:无法保证所有工作程序都会按顺序运行，例如，myfunction(20, ..)可能会在myfuntion(19, ..)之前运行. 20之后的其他工作程序也有可能在主进程可以对所设置的事件采取行动之前运行.我在打印i之前添加了if not event.is_set():调用来减小了竞争窗口的大小，但是它仍然存在.

As pointed out in Luke's answer, there is a race here: There's no guarantee that all the workers will run in order, so it's possible that myfunction(20, ..) will run prior to myfuntion(19, ..), for example. It's also possible that other workers after 20 will run before the main process can act on the event being set. I reduced the size of the race window by adding the if not event.is_set(): call prior to printing i, but it still exists.

这篇关于如何在多进程中杀死所有Pool工人?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！