问题描述
我正在运行一个C ++应用程序,该应用程序尝试使用 https://来运行python docs.python.org/3.5/extending/embedding.html 函数调用.这是应用程序错误消息管道给我的错误.
I'm running a C++ application which tries to run python using the https://docs.python.org/3.5/extending/embedding.html function calls. This is the error that the application error message pipes are giving me.
原始错误是:/usr/local/lib/python3.5/site-packages/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so:未定义符号:PyExc_UserWarning
Original error was: /usr/local/lib/python3.5/site-packages/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so: undefined symbol: PyExc_UserWarning
我很困惑,因为这仅发生在将Python嵌入C ++中时,因为当我通过解释器使用它时,导入才起作用.我对快速增加或解决问题的答案更感兴趣,因为这增加了我的理解.我在下面列出了一些系统/问题信息,以及我正在考虑发布的有关同一主题的其他一些问题.任何指导表示赞赏!
I'm quite puzzled as this only occurs when embedding Python in C++ as the import works when I use it through the interpreter. I'm more interested in an answer that adds to my understanding than a quick do this or do that fix. I list some system/problem information below, and some other questions that I'm considering posting about the same topic. Any guidance is appreciated!
系统/问题信息:
- Ubuntu 16.04,64位
- 已编译并启用了共享的Python 3.5.5
- numpy导入在解释器(python3.exe和python3.5.exe)中工作
- 我已确保PySys_SetPath()设置与解释器输出相同的sys.path:
import sys
,sys.path
- 我可以导入其他模块,例如PIL和datetimeutil;但是,numpy和pandas不可导入(熊猫使用numpy或似乎使用)
- 嵌入式Python使用以下命令:
Py_Import_Import()
,Py_Initialize()
(我确定.它只被调用过一次.)等等,但是它没有在解释器上获得全局锁定. - 该应用程序是使用CMake构建系统构建的,该系统可以为我的系统编译为MakeFiles.
- 使用
pip3.5 install numpy
命令使用pip 9.0.0安装numpy-1.14.2 - 导致此错误的python脚本只有一行:
import numpy
... - 我没有要从中导入文件的.zip文件.
- C ++中嵌入的Python使用的.exe位于/usr/local/bin/python3(使用Py_GetProgramName()确定).该.exe链接到libpython3.5m.so.1.0,丢失的符号位于libpython3.5m.so.1.0(nm)中 在multiarray.cpython-35m-x86_64-linux-gnu.so上的
-
ldd显示:
- Ubuntu 16.04, 64 bit
- Compiled Python 3.5.5 with enabled-shared
- numpy import works in the interpreter (python3.exe, and python3.5.exe)
- I have made sure that the PySys_SetPath() sets the same sys.path as the output from the interpreter:
import sys
,sys.path
- I can import other modules like PIL, and datetimeutil; however, numpy and pandas are not importable (pandas uses numpy or seems to)
- The embedded Python uses the following commands:
Py_Import_Import()
,Py_Initialize()
(I made sure. It is only called once.), etc., but it does not get a global lock on the interpreter. - The application is built with a CMake build system which compiles to MakeFiles for my system.
- Installed numpy-1.14.2 using pip 9.0.0 using the
pip3.5 install numpy
command - The python script that causes this error has one line:
import numpy
... - I do not have a .zip file that I'm importing files from.
- The .exe used by the Python embedded in the C++ is located at /usr/local/bin/python3 (used Py_GetProgramName() to determine this). This .exe is linked to the libpython3.5m.so.1.0, and the missing symbol lives in libpython3.5m.so.1.0 (ran nm)
ldd on multiarray.cpython-35m-x86_64-linux-gnu.so shows:
ldd multiarray.cpython-35m-x86_64-linux-gnu.so
libopenblasp-r0-39a31c03.2.18.so =>/usr/local/lib/python3.5/site-packages/numpy/core/./../.libs/libopenblasp-r0-39a31c03.2.18.so (0x00007fdbe149b000)
libopenblasp-r0-39a31c03.2.18.so => /usr/local/lib/python3.5/site-packages/numpy/core/./../.libs/libopenblasp-r0-39a31c03.2.18.so (0x00007fdbe149b000)
libm.so.6 =>/lib/x86_64-linux-gnu/libm.so.6(0x00007fdbe1192000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdbe1192000)
libpthread.so.0 =>/lib/x86_64-linux-gnu/libpthread.so.0(0x00007fdbe0f75000) libc.so.6 =>/lib/x86_64-linux-gnu/libc.so.6(0x00007fdbe0bab000) /lib64/ld-linux-x86-64.so.2(0x00007fdbe3ed5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdbe0f75000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdbe0bab000) /lib64/ld-linux-x86-64.so.2 (0x00007fdbe3ed5000)
libgfortran-ed201abd.so.3.0.0 =>/usr/local/lib/python3.5/site-packages/numpy/core/./../.libs/libgfortran-ed201abd.so.3.0.0 (0x00007fdbe08b1000)
libgfortran-ed201abd.so.3.0.0 => /usr/local/lib/python3.5/site-packages/numpy/core/./../.libs/libgfortran-ed201abd.so.3.0.0 (0x00007fdbe08b1000)
我可以/可能尝试通过其他方式重新安装numpy,但无法跟踪为什么可能有效.
I could/might try reinstalling numpy through different means, but I'm having trouble tracking why that might work.
在这一点上,我假设我的知识存在一些漏洞.我看过很多类似的文章,内容涉及在将Python嵌入C ++时无法导入multiarray组件和numpy的问题.但是,它们都不符合我的具体情况,或者正如我所说的那样存在漏洞.这是我可能会问的子问题列表,如果在此设置中没有人看到明显与之有关的任何内容.当/如果我问了这些问题,我可能会用链接更新这些问题(抛光它们之后).
At this point, I'm assuming some hole in my knowledge exists. I have looked at a lot of similar posts regarding not being able to import the multiarray component and numpy when embedding Python in C++; however, either none of them match my specific case or as I stated there exists a hole. Here are a list of sub-questions that I will probably be asking if no one sees anything in this setup that is obviously concerning. I'll probably update the questions with links when/if I ask them (After I polish them).
- numpy multiarray.so如何链接到pythonX.X.so以进行符号解析? ldd似乎并没有暗示它曾经这样做过.在此链接中问了这个问题
- CMake问题非相关问题已在此在18年4月12日问,并在18年4月16日回答.
- 在.bashrc中设置PYTHONPATH似乎并没有更新Py_GetPath()返回的内容,我不得不在站点包中添加用于通过sys.path的另一种方法导入的内容.它可能只会更新不影响C ++的bash脚本环境变量.
- How does the numpy multiarray.so link to the pythonX.X.so for symbol resolution? The ldd does not seem to suggest that it ever does. Asked this question at this link
- CMake Question non-related issue resolved in this question asked on 4/12/18 and answered on 4/16/18.
- Setting PYTHONPATH in .bashrc does not seem to update what Py_GetPath() returns, I had to add in the site-packages for imports through a different methodology to sys.path. It may only update the bash script environment variable which doesn't effect the C++.
我现在不要求上述问题的答案,而是为我的知识鸿沟提供更多线索.
I'm not asking for an answer for the above question list at this point, rather I'm giving more clues to where my gap in knowledge may be.
感谢您抽出宝贵的时间阅读该问题.任何帮助表示赞赏.
Thank you for taking time from your day to read this question. Any help is appreciated.
好吧,我找到了解决方法,目前正在使用它.沙丘问题开始使我更加仔细地考虑未定义的符号,以及它可能是链接器/编译器错误还是numpy导入始终期望将那些符号已加载到内存中的环境.这使我试图安装不同版本的numpy,以查看是否有任何较旧的版本有所作为.他们没有,但是确实使错误引发的错误有所不同.当我用Google搜索时,出现了这个问题.接受的答案通过将这两行添加到pythonInterface.cpp中给了我一个解决方法:
Well, I found a work around, and I'm currently using it. Dunes question started making me think more closely about undefined symbols and how it could be a linker/compiler error or that the numpy import always expects an environment with those symbols already loaded into memory. This got me trying to install different versions of numpy to see if any of the older versions made a difference. They did not, but it did make the error thrown to be slightly different. When I googled that, this question appeared. The accepted answer gave me a work around by adding these two lines to the pythonInterface.cpp:
-
#include <dlfcn.h>
-
dlopen("libpython3.5m.so.1.0", RTLD_LAZY | RTLD_GLOBAL)
#include <dlfcn.h>
dlopen("libpython3.5m.so.1.0", RTLD_LAZY | RTLD_GLOBAL)
这些命令添加了要在cpython.multiarray.so中加载并可供其使用的共享库.
These commands add the shared library to be loaded in and available to the cpython.multiarray.so.
这不是理想的解决方案,因为它指向一个特定的.so,这可能因机器而异.它现在可以解决该问题,但是如果与pythonInterface.so链接的库发生更改,并且此行未更新,则在python调用过程中共享库不匹配的地方也可能导致错误.我相信如果已得到回答,因此我目前一直坚持提交或接受答案.谢谢!
This is not an ideal solution as pointing to a specific .so which may be different from machine to machine. It resolves the issue for now, but it also could lead to errors where mismatches of shared libraries can occur during the python call process if the linked library to the pythonInterface.so changes, and this line does not get updated. I believe a better answer can be achieved if this sub-question is answered, so I'm currently holding out on submitting or accepting an answer until then. Thanks!
推荐答案
根本原因
之所以会发生此错误,是因为numpy中的multiarray.cpython-35m-x86_64-linux-gnu.so
模块依赖于libpythonx.x.so
,因为它不是显式链接libpythonx.x.so
的原因.因此,如果您使用ldd -d multiarray.cpython-35m-x86_64-linux-gnu.so
,您将在列表中看不到python.
This error occurs because multiarray.cpython-35m-x86_64-linux-gnu.so
module in numpy depends on libpythonx.x.so
, be it is not explicit link the libpythonx.x.so
. So if you use ldd -d multiarray.cpython-35m-x86_64-linux-gnu.so
you will not see the python in the list.
Python没问题,因为python二进制文件依赖于libpython.x.x.so
,因此当numpy通过使用dlopen
加载multiarray.cpython-35m-x86_64-linux-gnu.so
时. libdl.so
将通过检查主程序python的依赖共享库来尝试解析未定义的符号.它将在libpython.x.x.so
中找到它.
Python doesn't have issue because python binary depends on libpython.x.x.so
, so when numpy load multiarray.cpython-35m-x86_64-linux-gnu.so
by using dlopen
. libdl.so
will try to resolve the undefined symbols by checking the dependent shared library of the main program which is python. It will find it in libpython.x.x.so
.
解决方案
了解根本原因后,解决方法非常简单,只需帮助libdl.so
就能找到libpython.x.x.so
.至少有两种方法可以实现:
After knowing the root cause the solution is very easy, just help libdl.so
to be able to find libpython.x.x.so
. There are at least two way to achieve that:
- 使用
dlopen("libpythonx.x.so", RTLD_GLOBAL)
.打开后,请使用RTLD_GLOBAL
标志,它会使libpythonx.x.so中的符号可用于随后加载的共享库的符号解析. - 在嵌入python的主程序中,将
libpythonx.x.so
添加到其依赖库中.
- Use
dlopen("libpythonx.x.so", RTLD_GLOBAL)
. After open this so useRTLD_GLOBAL
flag, it make symbol in libpythonx.x.so available for symbol resolution of subsequently loaded shared objects. - In main program which embed python, add the
libpythonx.x.so
into its dependency library.
这篇关于从C ++应用程序中的嵌入式Python调用时,Numpy导入在多数组扩展库上失败的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!