我遇到了一些与Windows上的模块属性更新没有传播到Windows上的子进程有关的问题。
以下代码段说明了问题:
import functools
import multiprocessing
import os
from contextlib import contextmanager
_DOMAIN_RANGE_SCALE = 'reference'
def get_domain_range_scale():
return _DOMAIN_RANGE_SCALE
def set_domain_range_scale(scale='Reference'):
global _DOMAIN_RANGE_SCALE
scale = str(scale).lower()
_DOMAIN_RANGE_SCALE = scale
class domain_range_scale(object):
def __init__(self, scale):
self._scale = scale
self._previous_scale = get_domain_range_scale()
def __enter__(self):
set_domain_range_scale(self._scale)
return self
def __exit__(self, *args):
set_domain_range_scale(self._previous_scale)
def __call__(self, function):
@functools.wraps(function)
def wrapper(*args, **kwargs):
with self:
return function(*args, **kwargs)
return wrapper
@contextmanager
def multiprocessing_pool(*args, **kwargs):
pool = multiprocessing.Pool(*args, **kwargs)
yield pool
pool.terminate()
def test_domain_range_scale(*args):
print('Domain Range Scale Inner: {0}, PID: {1}'.format(
get_domain_range_scale(), os.getpid()))
if __name__ == '__main__':
for scale in ('reference', '1', '100'):
with domain_range_scale(scale):
print('*' * 79)
print('Domain Range Scale Outer: {0}, PID: {1}'.format(
get_domain_range_scale(), os.getpid()))
with multiprocessing_pool(processes=4) as pool:
pool.map(test_domain_range_scale, range(10))
Linux/macOS上的输出
*******************************************************************************
Domain Range Scale Outer: reference, PID: 93989
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93992
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93990
Domain Range Scale Inner: reference, PID: 93993
Domain Range Scale Inner: reference, PID: 93991
Domain Range Scale Inner: reference, PID: 93992
*******************************************************************************
Domain Range Scale Outer: 1, PID: 93989
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93995
Domain Range Scale Inner: 1, PID: 93996
Domain Range Scale Inner: 1, PID: 93994
Domain Range Scale Inner: 1, PID: 93997
Domain Range Scale Inner: 1, PID: 93995
*******************************************************************************
Domain Range Scale Outer: 100, PID: 93989
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Domain Range Scale Inner: 100, PID: 94000
Domain Range Scale Inner: 100, PID: 94001
Domain Range Scale Inner: 100, PID: 93998
Domain Range Scale Inner: 100, PID: 93999
Windows上的输出
*******************************************************************************
Domain Range Scale Outer: reference, PID: 6524
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 2124
Domain Range Scale Inner: reference, PID: 5476
Domain Range Scale Inner: reference, PID: 4872
Domain Range Scale Inner: reference, PID: 1932
*******************************************************************************
Domain Range Scale Outer: 1, PID: 6524
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 2716
Domain Range Scale Inner: reference, PID: 1012
Domain Range Scale Inner: reference, PID: 1852
Domain Range Scale Inner: reference, PID: 6544
*******************************************************************************
Domain Range Scale Outer: 100, PID: 6524
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 7456
Domain Range Scale Inner: reference, PID: 5944
最佳答案
问题在于Windows不支持将“fork”作为新进程的启动方法(仅支持“spawn”)。全局变量不是用“spawn”继承的。
当您在_DOMAIN_RANGE_SCALE = 'reference'
下面放置print语句时,您将看到Windows上的子进程将再次运行该脚本
直到导入所需的函数。
在进程启动后,必须使用Pool的if __name__ == '__main__':
-参数显式注册全局变量。
...
def init_global(scale):
global _DOMAIN_RANGE_SCALE
_DOMAIN_RANGE_SCALE = scale
if __name__ == '__main__':
...
with multiprocessing.Pool(processes=4,
initializer=init_global,
initargs=(scale,)) as pool:
pool.map(test_domain_range_scale, range(10))
...
关于python - 模块属性更新不会传播到Windows上的子进程,我们在Stack Overflow上找到一个类似的问题:https://stackoverflow.com/questions/55742917/