python - 将参数值从文件传递到numba类

我正在python中建立一个numba模拟类，我想将早期模拟中存储在文件中的一些参数值传递给它。文件中保存了很多参数值，因此它们位于我正在使用pandas访问的文件中。

不幸的是，尝试在pandas中使用numba会引发错误。具体来说，似乎numba不满意我在其中调用pandas并分配值，但不确定如何键入它们。 （注意：如果我将这些链接的熊猫调用分成多行，则会将read_feather行标识为罪魁祸首）：

from collections import OrderedDict
from numba import jitclass, float32, int32
from pandas import read_feather

import os
PARAM_FILE = SOME_PATH_ON_MY_MACHINE

import numba
print(numba.__version__)

@jitclass(
    spec=OrderedDict(
        a=float32,
        b=float32,
    )
)
class MyParameters:
    def __init__(self,) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        self.a = simulated_parameters["a"]
        self.b = simulated_parameters["a"]

demo_bug = MyParameters()

### OUTPUT
0.45.1
Traceback (most recent call last):
  File "mwe-numba.py", line 42, in <module>
    demo_bug = MyParameters()
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/jitclass/base.py", line 126, in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 376, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 343, in error_rewrite
    reraise(type(e), e, None)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/six.py", line 658, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'read_feather': cannot determine Numba type of <class 'function'>

File "mwe-numba.py", line 32:
    def __init__(self,) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        ^

[1] During: resolving callee type: jitclass.MyParameters#10116c320<a:float32,b:float32,c:float32,tstar:int32,cstar:float32,loc:float32,scale:float32,shape:float32>
[2] During: typing of call at <string> (3)

[3] During: resolving callee type: jitclass.MyParameters#10116c320<a:float32,b:float32,c:float32,tstar:int32,cstar:float32,loc:float32,scale:float32,shape:float32>
[4] During: typing of call at <string> (3)


File "<string>", line 3:
<source missing, REPL/exec in use?>

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

为了变得聪明，我决定创建另一个类来保存参数信息，然后在其中放置read_feather。但是，那也不起作用

class MyParameters:
    def __init__(self) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        self.a = simulated_parameters["a"]
        self.b = simulated_parameters["a"]

@jitclass(
  OrderedDict(
    a=float32,
    b=float32,
  )
)
class MyModel:
  def __init__(self, param: MyParameters) -> None:
    self.a = param.a
    self.b = param.b

my_param = MyParameters()
print("That was no problem")
my_model = MyModel(param=my_param)

### OUTPUT
0.45.1
That was no problem
Traceback (most recent call last):
  File "also-error.py", line 40, in <module>
    my_model = MyModel(param=my_param)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/jitclass/base.py", line 126, in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 376, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 343, in error_rewrite
    reraise(type(e), e, None)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/six.py", line 658, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
[1] During: typing of argument at <string> (3)

File "<string>", line 3:
<source missing, REPL/exec in use?>

This error may have been caused by the following argument(s):
- argument 0: cannot determine Numba type of <class '__main__.MyParameters'>

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

我有什么办法可以将此代码放入numba中？我可以想到解决方法，但它们都有些笨拙。

最佳答案

此post突出显示numba最适合用于标量和矢量运算，而不是数据整理，因此不支持pandas。

另一个问题是您想在类dict中将kwargs实质上作为__init__传递，Github上的此issue解释了为什么会出现此问题。此通用参数处理需要在nopython模式函数内的Python模式下（例如，从kwarg中弹出项目）完成，此功能目前尚不存在。

这意味着当您在OrderedDict中进行解析时，您实际上必须告诉numba期待一个真正没有意义的python dict，但是numba从Dict开始具有0.43 functionality 。 SO post example。

但是，如果尝试这样做，将导致以下错误：

加载模块

from collections import OrderedDict
from numba import jitclass
from numba import types
from numba.typed import Dict

@jitclass(
    OrderedDict([
        ('params', Dict.empty(
            key_type=types.unicode_type,
            value_type=types.float64,
        ))
     ])
)
class MyModel:
    def __init__(self, params):
        self.a = params["a"]
        self.b = params["b"]

TypeError: spec values should be Numba type instances, got DictType[unicode_type,float64]({})

本质上，您必须告诉numba期望什么args，在您的情况下，我将执行以下操作：

假设您使用了read_feather，结果显示为dict：

simulated_parameters = {"a" : 2.0, "b" : 5.0}

现在初始化numba类：

@jitclass(
    OrderedDict([
        ('a', types.float32),
        ('b', types.float32)
    ])
)
class MyModel:
    def __init__(self, a, b):
        self.a = a
        self.b = b

foo = MyModel(simulated_parameters["a"], simulated_parameters["b"])

关于python - 将参数值从文件传递到numba类，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/58401413/