我在ClientConnectionError
生成的multiprocessing.Queue
中放置了一个asyncio
异常。我这样做是为了将在asyncio land中生成的异常传递回另一个线程/进程中的客户端。
我的假设是,此异常发生在从队列中读取异常的反序列化过程中。如果不这样做的话,很难达到目的。
Traceback (most recent call last):
File "model_neural_simplified.py", line 318, in <module>
main(**arg_parser())
File "model_neural_simplified.py", line 314, in main
globals()[command](**kwargs)
File "model_neural_simplified.py", line 304, in predict
next_neural_data, next_sample = reader.get_next_result()
File "/project_neural_mouse/src/asyncs3/s3reader.py", line 174, in get_next_result
result = future.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "model_neural_simplified.py", line 245, in read_sample
f_bytes = s3f.read(read_size)
File "/project_neural_mouse/src/asyncs3/s3reader.py", line 374, in read
size, b = self._issue_request(S3Reader.READ, (self.url, size, self.position))
File "/project_neural_mouse/src/asyncs3/s3reader.py", line 389, in _issue_request
response = self.communication_channels[uuid].get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/usr/local/lib/python3.6/dist-packages/aiohttp/client_exceptions.py", line 133, in __init__
super().__init__(os_error.errno, os_error.strerror)
AttributeError: 'str' object has no attribute 'errno'
我想问这个问题有点难,但是有人知道这个问题吗?
python 3.6.8,aiohttp.\uuu版本=3.6.0
更新:
我成功地再现了这个问题(塞缪尔在评论中提到了改进最小可重复性测试用例的功劳,后来xtreak在bugs.python.org上提到了进一步将其提取到pickle-only测试用例中的功劳):
import pickle
ose = OSError(1, 'unittest')
class SubOSError(OSError):
def __init__(self, foo, os_error):
super().__init__(os_error.errno, os_error.strerror)
cce = SubOSError(1, ose)
cce_pickled = pickle.dumps(cce)
pickle.loads(cce_pickled)
./python.exe ../backups/bpo38254.py
Traceback (most recent call last):
File "/Users/karthikeyansingaravelan/stuff/python/cpython/../backups/bpo38254.py", line 12, in <module>
pickle.loads(cce_pickled)
File "/Users/karthikeyansingaravelan/stuff/python/cpython/../backups/bpo38254.py", line 8, in __init__
super().__init__(os_error.errno, os_error.strerror)
AttributeError: 'str' object has no attribute 'errno'
参考文献:
https://github.com/aio-libs/aiohttp/issues/4077
https://bugs.python.org/issue38254
最佳答案
OSError
具有a custom __reduce__
implementation;不幸的是,对于与预期参数不匹配的子类,它不是子类友好的。通过手动调用__reduce__
可以看到酸洗的中间状态:
>>> SubOSError.__reduce__(cce)
(modulename.SubOSError, (1, 'unittest'))
tuple
的第一个元素是可调用的,第二个元素是要传递的参数的tuple
。所以当它试图重新创建您的类时,它会:modulename.SubOSError(1, 'unittest')
已经丢失了您最初创建时使用的
OSError
的信息。如果必须接受与
OSError.__reduce__
/OSError.__init__
期望不匹配的参数,则需要编写自己的__reduce__
重写以确保pickle了正确的信息。一个简单的版本可能是:class SubOSError(OSError):
def __init__(self, foo, os_error):
self.foo = foo # Must preserve information for pickling later
super().__init__(os_error.errno, os_error.strerror)
def __reduce__(self):
# Pickle as type plus tuple of args expected by type
return type(self), (self.foo, OSError(*self.args))
采用这种设计,
SubOSError.__reduce__(cce)
现在将返回:(modulename.SubOSError, (1, PermissionError(1, 'unittest')))
其中
tuple
的第二个元素是重新创建实例所需的正确参数(预期会从OSError
更改为PermissionError
;OSError
实际上会根据errno
返回自己的子类)。