本文介绍了pickle:它如何腌制一个函数?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

在昨天发布的帖子中,我无意中发现更改了一个函数的__qualname__pickle 有意想不到的影响.通过运行更多的测试,我发现在pickle一个函数时,pickle并没有像我想象的那样工作,而改变函数的__qualname__对如何产生真正的影响pickle 行为.

下面的代码片段是我运行的测试,

进口泡菜从 sys 导入模块# 一个简单的pickle函数def haha​​ha(): 返回 1打印('哈哈哈',哈哈哈,'\n')# 改变函数的__qualname__ 哈哈哈hahaha.__qualname__ = 'sdfsdf'print('set haha​​ha __qualname__ to sdfsdf',hahaha,'\n')#复制一份哈哈哈setattr(modules['__main__'],'abcabc',hahaha)print('创建只是哈哈哈的abcabc',abcabc,'\n')尝试:泡菜.转储(哈哈哈)除了作为 e 的例外:打印('泡菜哈哈哈')打印(e,'\n')尝试:pickle.dumps(abcabc)除了作为 e 的例外:print('pickle abcabc,一份哈哈哈')打印(e,'\n')尝试:pickle.dumps(sdfsdf)除了作为 e 的例外:打印('泡菜sdfsdf')打印(e)

通过运行代码片段可以看到,hahahaabcabc 都不能被pickle,因为异常:

无法在 0x7fda36dc5f28 处腌制 :__main__ 上的属性查找 sdfsdf 失败.

我真的被这个异常搞糊涂了,

  1. pickle 在pickle 函数时寻找什么?虽然hahaha__qualname__被改成了'sdfsdf',但是hahaha及其拷贝的abcabc是仍然可以在会话中调用(因为它们在 dir(sys.modules['__main__']) 中),那么为什么 pickle 不能pickle它们?

  2. 改变函数的 __qualname__ 的真正效果是什么?我知道通过将 hahaha__qualname__ 更改为 'sdfsdf' 不会使 sdfsdf 可调用,因为它不会出现在 dir(sys.modules['__main__']).但是,正如您通过运行片段所看到的,在将 hahaha__qualname__ 更改为 'sdfsdf' 后,对象 hahaha 及其copy abcabc 已更改为类似 的内容.sys.modules['__main__'] 中的对象有什么区别?

解决方案

函数对象的酸洗在 pickle.py 中的save_global 方法:

首先通过__qualname__获取函数名:

name = getattr(obj, '__qualname__', None)

之后,在检索模块后,重新导入:

__import__(module_name, level=0)模块 = sys.modules[module_name]

这个新导入的 module 然后用于将函数作为属性查找:

obj2, parent = _getattribute(module, name)

obj2 现在将是该函数的新副本,但由于 sdfsdf 在此模块中不存在,因此此处酸洗失败.


你可以做到这一点,但你必须保持一致:

>>>导入系统>>>进口泡菜>>>def haha​​ha(): 返回 1>>>hahaha.__qualname__ = "sdfsdf";>>>setattr(sys.modules[__main__"], sdfsdf", 哈哈哈)>>>泡菜.转储(哈哈哈)b'\x80\x04\x95\x17\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x06sdfsdf\x94\x93\x94.'

In a post I posted yesterday, I accidentally found changing the __qualname__ of a function has an unexpected effect on pickle. By running more tests, I found that when pickling a function, pickle does not work in the way I thought, and changing the __qualname__ of the function has a real effect on how pickle behaves.

The snippets below are tests I ran,

import pickle
from sys import modules

# a simple function to pickle
def hahaha(): return 1

print('hahaha',hahaha,'\n')

# change the __qualname__ of function hahaha
hahaha.__qualname__ = 'sdfsdf'
print('set hahaha __qualname__ to sdfsdf',hahaha,'\n')

# make a copy of hahaha
setattr(modules['__main__'],'abcabc',hahaha)
print('create abcabc which is just hahaha',abcabc,'\n')

try:
    pickle.dumps(hahaha)
except Exception as e:
    print('pickle hahaha')
    print(e,'\n')

try:
    pickle.dumps(abcabc)
except Exception as e:
    print('pickle abcabc, a copy of hahaha')
    print(e,'\n')

try:
    pickle.dumps(sdfsdf)
except Exception as e:
    print('pickle sdfsdf')
    print(e)

As you can see by running the snippets, both hahaha and abcabc cannot be pickled because of the exception:

Can't pickle <function sdfsdf at 0x7fda36dc5f28>: attribute lookup sdfsdf on __main__ failed.

I'm really confused by this exception,

  1. What does pickle look for when it pickles a function? Although the __qualname__ of hahaha was changed to 'sdfsdf', the function hahaha as well as its copy abcabc is still callable in the session (as they are in dir(sys.modules['__main__'])), then why pickle cannot pickle them?

  2. What is the real effect of changing the __qualname__ of a function? I understand by changing the __qualname__ of hahaha to 'sdfsdf' won't make sdfsdf callable, as it won't show up in dir(sys.modules['__main__']). However, as you can see by running the snippets, after changing the __qualname__ of hahaha to 'sdfsdf', the object hahaha as well as its copy abcabc has changed to something like <function sdfsdf at 'some_address'>. What is the difference between the objects in sys.modules['__main__'] and <function sdfsdf at 'some_address'>?

解决方案

Pickling of function objects is defined in the save_global method in pickle.py:

First, the name of the function is retrieved via __qualname__:

name = getattr(obj, '__qualname__', None)

Afterwards, after retrieving the module, it is reimported:

__import__(module_name, level=0)
module = sys.modules[module_name]

This freshly imported module is then used to look up the function as an attribute:

obj2, parent = _getattribute(module, name)

obj2 would now be a new copy of the function, but since sdfsdf doesn't exist in this module, pickling fails here.


You can make this work, but you have to be consistent:

>>> import sys
>>> import pickle
>>> def hahaha(): return 1
>>> hahaha.__qualname__ = "sdfsdf"
>>> setattr(sys.modules["__main__"], "sdfsdf", hahaha)
>>> pickle.dumps(hahaha)
b'\x80\x04\x95\x17\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x06sdfsdf\x94\x93\x94.'

这篇关于pickle:它如何腌制一个函数?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-05 19:13