python - python多处理中的调试错误

我使用Pool模块的multiprocessing函数，以便在不同的数据上并行运行相同的代码。
结果发现，在某些数据上，我的代码引发了异常，但没有给出发生这种情况的确切行：

Traceback (most recent call last):
  File "my_wrapper_script.py", line 366, in <module>
    main()
  File "my_wrapper_script.py", line 343, in main
    results = pool.map(process_function, folders)
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
    raise self._value
KeyError: 'some_key'

我知道multiprocessing.log_to_stderr()，但当出现并发问题时它似乎很有用，这不是我的情况。
有什么想法吗？

最佳答案

如果您使用的是新的Python版本，那么您将看到实际的异常在该异常之前被打印出来。例如，下面是一个失败的示例：

import multiprocessing

def inner():
    raise Exception("FAIL")

def f():
    print("HI")
    inner()

p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()

使用Python3.4运行此程序时出现以下异常：

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "test.py", line 9, in f
    inner()
  File "test.py", line 4, in inner
    raise Exception("FAIL")
Exception: FAIL
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    p.apply(f)
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
Exception: FAIL

如果使用较新版本不是一个选项，那么最简单的方法是将工作函数包装在try/except块中，该块将在重新引发异常之前打印该异常：

import multiprocessing
import traceback

def inner():
    raise Exception("FAIL")

def f():
    try:
        print("HI")
        inner()
    except Exception:
        print("Exception in worker:")
        traceback.print_exc()
        raise

p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()

输出：

HI
Exception in worker:
Traceback (most recent call last):
  File "test.py", line 11, in f
    inner()
  File "test.py", line 5, in inner
    raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    p.apply(f)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
Exception: FAIL

关于python - python多处理中的调试错误，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/29080650/