。
下面是一个示例:
df1 = mul_df(5000,30,400) # mul_df to create a big multilevel dataframe
#let df2, df3, df4 = df1, df1, df1 to minimize the memory usage,
#they can also be mul_df(5000,30,400)
df2, df3, df4 = df1, df1, df1
In [12]: timeit df1+df2+df3+df4
1 loops, best of 3: 993 ms per loop
我对993ms不满意,有没有办法加速??如果是,如何编写cython代码?谢谢。
Note:
mul_df()
是创建演示多级数据帧的函数。import itertools
import numpy as np
import pandas as pd
def mul_df(level1_rownum, level2_rownum, col_num, data_ty='float32'):
''' create multilevel dataframe, for example: mul_df(4,2,6)'''
index_name = ['STK_ID','RPT_Date']
col_name = ['COL'+str(x).zfill(3) for x in range(col_num)]
first_level_dt = [['A'+str(x).zfill(4)]*level2_rownum for x in range(level1_rownum)]
first_level_dt = list(itertools.chain(*first_level_dt)) #flatten the list
second_level_dt = ['B'+str(x).zfill(3) for x in range(level2_rownum)]*level1_rownum
dt = pd.DataFrame(np.random.randn(level1_rownum*level2_rownum, col_num), columns=col_name, dtype = data_ty)
dt[index_name[0]] = first_level_dt
dt[index_name[1]] = second_level_dt
rst = dt.set_index(index_name, drop=True, inplace=False)
return rst
更新:
我的奔腾双核[email protected],3.00GB内存,WindowXP,Python 2.7.4,Numpy 1.7.1,Pandas 0.11.0,numexpr 2.0.1(Anaconda 1.5.0(32位))上的数据
In [1]: from pandas.core import expressions as expr
In [2]: import numexpr as ne
In [3]: df1 = mul_df(5000,30,400)
In [4]: df2, df3, df4 = df1, df1, df1
In [5]: expr.set_use_numexpr(False)
In [6]: %timeit df1+df2+df3+df4
1 loops, best of 3: 1.06 s per loop
In [7]: expr.set_use_numexpr(True)
In [8]: %timeit df1+df2+df3+df4
1 loops, best of 3: 986 ms per loop
In [9]: %timeit DataFrame(ne.evaluate('df1+df2+df3+df4'),columns=df1.columns,index=df1.index,dtype='float32')
1 loops, best of 3: 388 ms per loop
最佳答案
method 1: On my machine not so bad (with numexpr
disabled)
In [41]: from pandas.core import expressions as expr
In [42]: expr.set_use_numexpr(False)
In [43]: %timeit df1+df2+df3+df4
1 loops, best of 3: 349 ms per loop
方法2:使用
numexpr
(如果安装了numexpr
,则默认启用此选项)In [44]: expr.set_use_numexpr(True)
In [45]: %timeit df1+df2+df3+df4
10 loops, best of 3: 173 ms per loop
方法3:直接使用
numexpr
In [34]: import numexpr as ne
In [46]: %timeit DataFrame(ne.evaluate('df1+df2+df3+df4'),columns=df1.columns,index=df1.index,dtype='float32')
10 loops, best of 3: 47.7 ms per loop
使用
numexpr
可以实现这些加速,因为:避免使用中间的临时数组(在本例中,可能是
quite inefficient in numpy, I suspect this is being evaluated like
((df1+df2)+df3)+df4
uses multi-cores as available
。使用direct(方法3)
numexpr
可以获得更多的加速。请注意,在pandas 0.13(本周将发布0.12)中,我们实现了一个函数
df1 + df2
,它实际上与我上面的示例完全一样。敬请关注(如果你有冒险精神,这将很快成为大师:https://github.com/pydata/pandas/pull/4037)In [5]: %timeit pd.eval('df1+df2+df3+df4')
10 loops, best of 3: 50.9 ms per loop
最后回答你的问题,
numexpr
在这里根本没有帮助;ne.evaluate(...)
在这类问题上是非常有效的(也就是说,有些情况下cython是有帮助的)。而且它们应该是一个单一的数据类型