pickle:它如何腌制一个函数?

问题描述

在昨天发布的帖子中，我无意中发现更改了一个函数的__qualname__对 pickle 有意想不到的影响.通过运行更多的测试，我发现在pickle一个函数时，pickle并没有像我想象的那样工作，而改变函数的__qualname__对如何产生真正的影响pickle 行为.

下面的代码片段是我运行的测试，

进口泡菜从 sys 导入模块# 一个简单的pickle函数def haha​​ha(): 返回 1打印('哈哈哈'，哈哈哈，'\n')# 改变函数的__qualname__ 哈哈哈hahaha.__qualname__ = 'sdfsdf'print('set haha​​ha __qualname__ to sdfsdf',hahaha,'\n')#复制一份哈哈哈setattr(modules['__main__'],'abcabc',hahaha)print('创建只是哈哈哈的abcabc',abcabc,'\n')尝试:泡菜.转储(哈哈哈)除了作为 e 的例外:打印('泡菜哈哈哈')打印(e，'\n')尝试:pickle.dumps(abcabc)除了作为 e 的例外:print('pickle abcabc，一份哈哈哈')打印(e，'\n')尝试:pickle.dumps(sdfsdf)除了作为 e 的例外:打印('泡菜sdfsdf')打印(e)

通过运行代码片段可以看到，hahaha 和 abcabc 都不能被pickle，因为异常:

无法在 0x7fda36dc5f28 处腌制 :__main__ 上的属性查找 sdfsdf 失败.

我真的被这个异常搞糊涂了，

1. pickle 在pickle 函数时寻找什么?虽然hahaha的__qualname__被改成了'sdfsdf'，但是hahaha及其拷贝的abcabc是仍然可以在会话中调用(因为它们在 dir(sys.modules['__main__']) 中)，那么为什么 pickle 不能pickle它们?

2. 改变函数的 __qualname__ 的真正效果是什么?我知道通过将 hahaha 的 __qualname__ 更改为 'sdfsdf' 不会使 sdfsdf 可调用，因为它不会出现在 dir(sys.modules['__main__']).但是，正如您通过运行片段所看到的，在将 hahaha 的 __qualname__ 更改为 'sdfsdf' 后，对象 hahaha 及其copy abcabc 已更改为类似 的内容.sys.modules['__main__'] 和 中的对象有什么区别?

解决方案

函数对象的酸洗在 pickle.py 中的save_global 方法:

首先通过__qualname__获取函数名:

name = getattr(obj, '__qualname__', None)

之后，在检索模块后，重新导入:

__import__(module_name, level=0)模块 = sys.modules[module_name]

这个新导入的 module 然后用于将函数作为属性查找:

obj2, parent = _getattribute(module, name)

obj2 现在将是该函数的新副本，但由于 sdfsdf 在此模块中不存在，因此此处酸洗失败.

---

你可以做到这一点，但你必须保持一致:

>>>导入系统>>>进口泡菜>>>def haha​​ha(): 返回 1>>>hahaha.__qualname__ = "sdfsdf";>>>setattr(sys.modules[__main__"], sdfsdf", 哈哈哈)>>>泡菜.转储(哈哈哈)b'\x80\x04\x95\x17\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x06sdfsdf\x94\x93\x94.'

In a post I posted yesterday, I accidentally found changing the __qualname__ of a function has an unexpected effect on pickle. By running more tests, I found that when pickling a function, pickle does not work in the way I thought, and changing the __qualname__ of the function has a real effect on how pickle behaves.

The snippets below are tests I ran,

import pickle
from sys import modules

# a simple function to pickle
def hahaha(): return 1

print('hahaha',hahaha,'\n')

# change the __qualname__ of function hahaha
hahaha.__qualname__ = 'sdfsdf'
print('set hahaha __qualname__ to sdfsdf',hahaha,'\n')

# make a copy of hahaha
setattr(modules['__main__'],'abcabc',hahaha)
print('create abcabc which is just hahaha',abcabc,'\n')

try:
    pickle.dumps(hahaha)
except Exception as e:
    print('pickle hahaha')
    print(e,'\n')

try:
    pickle.dumps(abcabc)
except Exception as e:
    print('pickle abcabc, a copy of hahaha')
    print(e,'\n')

try:
    pickle.dumps(sdfsdf)
except Exception as e:
    print('pickle sdfsdf')
    print(e)

As you can see by running the snippets, both hahaha and abcabc cannot be pickled because of the exception:

Can't pickle <function sdfsdf at 0x7fda36dc5f28>: attribute lookup sdfsdf on __main__ failed.

I'm really confused by this exception,

1. What does pickle look for when it pickles a function? Although the __qualname__ of hahaha was changed to 'sdfsdf', the function hahaha as well as its copy abcabc is still callable in the session (as they are in dir(sys.modules['__main__'])), then why pickle cannot pickle them?

2. What is the real effect of changing the __qualname__ of a function? I understand by changing the __qualname__ of hahaha to 'sdfsdf' won't make sdfsdf callable, as it won't show up in dir(sys.modules['__main__']). However, as you can see by running the snippets, after changing the __qualname__ of hahaha to 'sdfsdf', the object hahaha as well as its copy abcabc has changed to something like <function sdfsdf at 'some_address'>. What is the difference between the objects in sys.modules['__main__'] and <function sdfsdf at 'some_address'>?

解决方案

Pickling of function objects is defined in the save_global method in pickle.py:

First, the name of the function is retrieved via __qualname__:

name = getattr(obj, '__qualname__', None)

Afterwards, after retrieving the module, it is reimported:

__import__(module_name, level=0)
module = sys.modules[module_name]

This freshly imported module is then used to look up the function as an attribute:

obj2, parent = _getattribute(module, name)

obj2 would now be a new copy of the function, but since sdfsdf doesn't exist in this module, pickling fails here.

---

You can make this work, but you have to be consistent:

>>> import sys
>>> import pickle
>>> def hahaha(): return 1
>>> hahaha.__qualname__ = "sdfsdf"
>>> setattr(sys.modules["__main__"], "sdfsdf", hahaha)
>>> pickle.dumps(hahaha)
b'\x80\x04\x95\x17\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x06sdfsdf\x94\x93\x94.'

这篇关于pickle:它如何腌制一个函数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！