为什么用`pickle`转储比`json`快得多?

问题描述

这是针对Python 3.6的.

This is for Python 3.6.

编辑并删除了很多无关紧要的内容.

Edited and removed a lot of stuff that turned out to be irrelevant.

我曾经以为json比pickle快，而且关于Stack Overflow的其他答案和评论似乎也使很多其他人也相信这一点.

I had thought json was faster than pickle and other answers and comments on Stack Overflow make it seem like a lot of other people believe this as well.

我的测试是犹太洁食吗?差距比我预期的要大得多.在非常大的对象上进行测试，我得到相同的结果.

Is my test kosher? The disparity is much larger than I expected. I get the same results testing on very large objects.

import json
import pickle
import timeit

file_name = 'foo'
num_tests = 100000

obj = {1: 1}

command = 'pickle.dumps(obj)'
setup = 'from __main__ import pickle, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("pickle: %f seconds" % result)

command = 'json.dumps(obj)'
setup = 'from __main__ import json, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("json:   %f seconds" % result)

和输出:

pickle: 0.054130 seconds
json:   0.467168 seconds

推荐答案

我根据您的代码段尝试了几种方法，发现使用cPickle并将dumps方法的协议参数设置为:cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL)是最快的转储方法.

I have tried several methods based on your code snippet and found out that using cPickle with setting the protocol argument of the dumps method as: cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL) is the fastest dump method.

import msgpack
import json
import pickle
import timeit
import cPickle
import numpy as np

num_tests = 10

obj = np.random.normal(0.5, 1, [240, 320, 3])

command = 'pickle.dumps(obj)'
setup = 'from __main__ import pickle, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("pickle:  %f seconds" % result)

command = 'cPickle.dumps(obj)'
setup = 'from __main__ import cPickle, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("cPickle:   %f seconds" % result)


command = 'cPickle.dumps(obj, protocol=cPickle.HIGHEST_PROTOCOL)'
setup = 'from __main__ import cPickle, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("cPickle highest:   %f seconds" % result)

command = 'json.dumps(obj.tolist())'
setup = 'from __main__ import json, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("json:   %f seconds" % result)


command = 'msgpack.packb(obj.tolist())'
setup = 'from __main__ import msgpack, obj'
result = timeit.timeit(command, setup=setup, number=num_tests)
print("msgpack:   %f seconds" % result)

输出:

pickle         :   0.847938 seconds
cPickle        :   0.810384 seconds
cPickle highest:   0.004283 seconds
json           :   1.769215 seconds
msgpack        :   0.270886 seconds

这篇关于为什么用`pickle`转储比`json`快得多?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！