问题描述
我正在尝试根据我的要求执行并行处理,对于 4k-5k 元素,代码似乎按预期工作.但是一旦要处理的元素开始增加,代码就会处理一些列表,然后没有抛出任何错误,程序就会突然停止运行.
I am trying to perform parallel processing for my requirements, and the code seems to be working as expected for 4k-5k elements in parallel. But as soon as the elements to be processed start increasing, the code processes a few listings and then without throwing any error, the program stops running abruptly.
我检查过,程序没有挂起,RAM 可用(我有 16 Gb RAM),CPU 利用率甚至不到 30%.似乎无法弄清楚发生了什么.我有 100 万个元素要处理.
I checked and the program is not hung, the RAM is available (I have a 16 Gb RAM) and CPU Utilization is not even 30%. Can't seem to figure out what is happening. I have 1 million elements to be processed.
def get_items_to_download():
#iterator to fetch all items that are to be downloaded
yield download_item
def start_download_process():
multiproc_pool = multiprocessing.Pool(processes=10)
for download_item in get_items_to_download():
multiproc_pool.apply_async(start_processing, args = (download_item, ), callback = results_callback)
multiproc_pool.close()
multiproc_pool.join()
def start_processing(download_item):
try:
# Code to download item from web API
# Code to perform some processing on the data
# Code to update data into database
return True
except Exception as e:
return False
def results_callback(result):
print(result)
if __name__ == "__main__":
start_download_process()
更新 -
发现错误 - BrokenPipeError: [Errno 32] Broken pipe
Found the error- BrokenPipeError: [Errno 32] Broken pipe
跟踪 -
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/usr/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
推荐答案
我在 Linux 上使用 Python 3.8
也有同样的体验.我使用 Python 3.7
设置了一个新环境,multiprocessing.Pool()
现在可以正常工作了.
I had the same experience with Python 3.8
on Linux. I set up a new environment with Python 3.7
and multiprocessing.Pool()
works now without any issue.
这篇关于Python 多处理池突然停止的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!