使用 cython --embed 时静态链接 python37.dll 和 vcruntime140.dll

问题描述

假设我正在cythonizing"这个test.py:

Let's say I'm "cythonizing" this test.py:

import json
print(json.dumps({'key': 'hello world'}))

与:

cython test.py --embed
call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl test.c /I C:\Python37\include /link C:\Python37\libs\python37.lib

如分发嵌入 Cython 编译的代码并使其在任何机器上运行所需的最少文件集，需要分发 python37.dll 和 vcruntime140.dll 和 Lib\ 的内容(作为 Lib\ 或打包成 python37.zip), 沿着 test.exe 文件.

As mentioned in Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine, it is necessary to distribute python37.dll and vcruntime140.dll and the content of Lib\ (either as Lib\ or packed into a python37.zip) as well, along the test.exe file.

问题:如何修改cl.exe ...命令让编译器静态链接python37.dll和vcruntime140.dll 在 test.exe 文件中?

Question: How to modify the cl.exe ... command to ask the compiler to statically link python37.dll and vcruntime140.dll inside the test.exe file?

(这样就不再需要分别发送 python37.dll 和 vcruntime140.dll)

(so that shipping python37.dll and vcruntime140.dll separately is no longer necessary)

推荐答案

虽然在 Linux 上创建静态链接的嵌入式 Python 可执行文件相对容易(例如参见 SO-post) 在 Windows 上要复杂得多.你可能不想这样做.

While creating a statically linked embeded-Python-executable is relatively easy on Linux (see for example this SO-post) it is much more complicated on Windows. And you probably don't want to do it.

结果也可能不是人们所期望的:由于与 Linux 共享对象相比 dll 的限制，生成的静态 python 版本将无法使用/加载任何其他 c 扩展，因为它支持- 在编译/链接期间输入.

Also the result might not be what one would expect: Due to limitations of dlls compared to Linux' shared objects, the resulting static python-version will not be able to use/load any other c-extensions, as the one backed-in during the compile/link time.

我也不建议从 vcruntime-dll 切换到它的静态版本 - 只有当一切(exe、c-extensions、其他依赖于 vcruntime 的 dll)是静态的时候才有意义链接到一个巨大的可执行文件中.

I also would not recommend to switch from vcruntime-dll to its static version - it would only make sense, when everything (exe, c-extensions, other dll which depend on vcruntime) is statically linked into one huge executable.

第一个障碍:虽然在 Linux 上，python 发行版通常已经提供了一个静态 Python 库，但 Windows 发行版只有 dll，不能静态链接.

The first stumbling block: While on Linux python distributions often have a static Python-library already shipped, Windows distributions have only the dll, which cannot be statically linked in.

因此需要在 Windows 上构建一个静态库.这个链接是一个很好的起点.

Thus one needs to build a static library on Windows. A good starting point is this link.

下载正确 Python 版本的源代码后 (git clone --depth=1 --branch v3.8.0 https://github.com/python/cpython.git) 你可以去到 cpython\PCBuild 并按照文档中的说明构建 cpython(可能因版本而异).

After downloading the source for the right Python version (git clone --depth=1 --branch v3.8.0 https://github.com/python/cpython.git) you can go to cpython\PCBuild and build cpython as explained in documentation (which might vary from version to version).

就我而言是

cd cpython/PCbuild
.\build.bat -e -p x64

不，我们有一个正常运行的 Python3.8 安装，可以在 cpython/PCbuild/amd64 中找到.创建文件夹 cpython/PCbuild/static_amd64 并添加以下 pyx 文件:

No we have a functioning Python3.8 installation, which can be found in cpython/PCbuild/amd64. Create folder cpython/PCbuild/static_amd64 and add the following pyx-file:

#hello.pyx
print("I'm standalone")

暂时将python38.dll复制到static_amd64.

现在让我们使用嵌入式 Python 解释器构建我们的程序:

Now let's build our program with embedded python interpreter:

cython --embed -3 hello.pyx
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl /c hello.c /Fohello.obj  /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib  /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\amd64"

启动后，hello_prog.exe 欺骗了我们，因为它并不是真正独立的.好消息是:它会找到所需的 Python 安装，例如此处.

After the start, hello_prog.exe lies to us, as it is not really standalone. The good news is: it finds the Python-installation which is needed as described for example here.

现在让我们创建一个静态的 python38-library.为此，我们在 cpython/PCbuild-folder 中打开 pcbuild.sln 并更改 pythoncore-project 的设置以在 PCbuild\amd64_static-folder 中生成静态库.重建它.

Now let's create a static python38-library. For that we open pcbuild.sln in cpython/PCbuild-folder and change pythoncore-project's setting to produce static library in PCbuild\amd64_static-folder. Rebuild it.

现在我们可以构建embedded-python-exe:

Now we can build the embedded-python-exe:

cl /c hello.c /Fohello.obj /D "Py_NO_ENABLE_SHARED" /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib "version.lib" "shlwapi.lib" "ws2_32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64"

与针对 dll 的构建相比，我们必须更改以下内容:

Compared to the build against dll we had to change the following:

• Py_NO_ENABLE_SHARED(即/DPy_NO_ENABLE_SHARED")添加到预处理器定义中，否则链接器将查找错误的符号.
• python-dll 带来的 Windows 依赖项(即 version.lib 等)现在需要显式传递给链接器(这可以在链接器命令中查找)-pythoncore 项目的行).
• lib 路径显示到静态文件夹，即 "/LIBPATH:\cpython\PCbuild\static_amd64" 现在.
• 可能还有其他较小的问题(不同的优化级别、链接时间代码生成、禁用整个程序优化等)，具体取决于您的具体工具链.

• Py_NO_ENABLE_SHARED (i.e. /D "Py_NO_ENABLE_SHARED") is added to preprocessor definitions, otherwise the linker will look for wrong symbols.
• the Windows dependencies (i.e. version.lib and so on) which were brought by the python-dll need to be passed to the linker explicitly now (this can be looked up in the linker-command-line of the pythoncore-project).
• lib path show to the static folder, i.e. "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64" now.
• there might be other smaller troubles (different optimization levels, link-time-code generation, disabling whole-program-optimization and so on) depending on your exact tool chain.

我们现在可以从 static_amd64 中删除 python38.dll 并且 hello_prog.exe 仍然有效.

We can delete the python38.dll from static_amd64 now and the hello_prog.exe still works.

在 Linux 上，这将是完成任务"，在 Windows 上，我们才刚刚开始......

On Linux, this would be "mission accomplished", on Windows we are just at the beginning...

确保 cpython-folder 有一个 DLLs-folder 和所有正确的 pyd 文件，否则从 PCbuild/amd64 创建和复制所有 pyd 文件-文件夹.

Make sure that cpython-folder has a DLLs-folder with all right pyd-files, otherwise create and copy all pyd-files from PCbuild/amd64-folder.

让我们的 pyx 文件稍微复杂一点:

Let's make our pyx-file a little bit more complicated:

import _decimal
print("I'm standalone")

_decimal 是 decimal 模块的快速实现，它是一个 C 扩展，可以在 DLL 文件夹中找到.

_decimal is a fast implementation of the decimal-module which is a C-extension and can be found in the DLL-folder.

cythonizing 和 build 之后，运行 hello_prog.exe 会导致以下错误消息:

After cythonizing and building it, running hello_prog.exe leads to the following error message:

import _decimal
ImportError: DLL load failed while importing _decimal: The specified module could not be found.

问题很容易找到:

dumpbin /DEPENDENTS ../amd64/_decimal.pyd
...
python38.dll
...

我们安装的扩展仍然依赖于python-dll.让我们针对静态库重建它们 - 我们需要将库路径从 amd64 更改为 static_amd64，添加预处理器定义 Py_NO_ENABLE_SHARED 和所有丢失的窗口-libraries(即version.lib"& Co.)并添加 /EXPORT:PyInit__decimal 到链接选项，否则，由于 Py_NO_ENABLE_SHARED 它变得不可见.结果不依赖于python-dll！我们将其复制到 DLLs 文件夹并...

The extensions of our installation still depends on the python-dll. Let's rebuild them against the static library - we need to change library path to from amd64 to static_amd64, to add preprocessor define Py_NO_ENABLE_SHARED and all missing windows-libraries (i.e ""version.lib"& Co.) and adding /EXPORT:PyInit__decimal to link options, otherwise, due to Py_NO_ENABLE_SHARED it becomes invisible. The result has no dependency on the python-dll! We copy it to the DLLs-folder and ...

hello_prog.exe
# crash/stopped worked

发生了什么?我们违反了一个定义规则 (ODR) 并最终得到了两个 Python 解释器:一个来自 hello_prog.exe，它被初始化，另一个来自 _decimal.pyd未初始化._decimal.pyd 说话"到它的未初始化的解释器，坏事发生了.

What is happening? We violated one definition rule (ODR) and ended up with two Python-interpreters: the one from the hello_prog.exe, which is initialized and the one from _decimal.pyd which is not initialized. _decimal.pyd "speaks" to its interpreter which is not initialized and bad things happens.

与 Linux 的区别在于共享对象和 dll 之间的区别:共享对象可以使用来自 exe 的符号(如果 exe 使用正确的选项构建)，但 dll 不能，因此必须依赖于 dll(我们不想要)或需要有自己的版本.

The difference to Linux is the difference between shared-objects and dlls: while shared-objects can use symbols from the exe (if the exe is built with right options) dll cannot and thus must either depend on a dll (which we don't want) or need to have its own version.

为了避免违反 ODR，我们只有一个出路:它必须直接链接到我们的 hello_word 可执行文件中.因此，让我们将 _decimal 的项目更改为静态库并在 static_amd64 文件夹中重建它.从DLL"文件夹中删除 pyd 并将 /WHOLEARCHIVE:_decimal.lib 添加到链接器命令行(整个存档，否则链接器将丢弃 _decimal.libcode> 因为它的任何符号都没有在某处被引用)，导致一个可执行文件，它有以下错误:

To avoid the violation of ODR we have only one way out: it must be linked directly into our hello_word-executable. So let's change the project for _decimal to static library and rebuild it in static_amd64-folder. Deleting the pyd from "DLLs"-folder and adding /WHOLEARCHIVE:_decimal.lib to the linker-command-line (whole archive while otherwise the linker would just discard _decimal.lib as none of its symbols is referenced somewhere), leads to an executable, which has the following error:

ModuleNotFoundError: No module named '_decimal'

这是意料之中的 - 我们需要告诉解释器，模块 _decimal 已回退，不应在 python 路径上搜索.

This is expected - we need to tell to the interpreter, that the module _decimal is backed in and should not be searched on the python path.

这个问题通常的解决方案是使用PyImport_AppendInittab 就在 Py_Initialize 之前，这意味着我们需要更改 cython 生成的 c 文件(可能有 解决方法，但由于多阶段初始化，这并不容易. 因此，嵌入 Python 的更明智的方法可能是 此处 或 此处 main 不是由 Cython 编写的).c 文件应如下所示:

The usual solution for this problem is to use PyImport_AppendInittab just before Py_Initialize, that means we need to change the c-file generated by cython (there might be workarounds, but due to multi-phase initialization it is not that easy. So probably a saner way to embed Python is the one presented here or here were main isn't written by Cython). The c-file should look as follows:

//decalare init-functions
extern  PyObject* PyInit__decimal();
...
int main(int argc, char** argv) {
...
    if (argc && argv)
        Py_SetProgramName(argv[0]);
    PyImport_AppendInittab("_decimal", PyInit__decimal); //  HERE WE GO
                                                         //  BEFORE Py_Initialize

    Py_Initialize();

现在构建所有东西会导致一个 exe 打印

Building everything now leads to an exe which prints

I'm standalone

这一次不是在骗我们！

现在我们必须为我们需要的所有其他内置扩展重复最后的步骤.

Now we have to repeat the last steps for all other built-in extension we need.

以上意味着静态构建的python-interpreter有一些限制:所有内置模块都需要备份到可执行文件中，我们不能使用numpy/scipy之类的库来扩展解释器(但可以直接执行)在编译/链接时).

The above means there are some restrictions for statically built python-interpreter: All built-in modules need to be backed into the executable and we cannot extend the interpreter latter on with libraries like numpy/scipy (but can to do it directly at the compile/link time).

摆脱 vcruntime-dll 更容易:以上所有步骤必须使用 /MT 选项代替MD-选项.但是，由于使用其他 dll(例如 _ctypes 需要 ffi-dll)，可能会出现一些问题，这些 dll 是使用 dll 版本构建的(因此我们再次拥有违反 ODR) - 所以我不推荐它.

Getting rid of vcruntime-dll is easier: all above steps must be done with /MT option instead of MD-option. However, there might be some problems due to usage of other dlls (e.g. _ctypes needs ffi-dll) which where built with the dll-version (and thus we once again have ODR-violated) - so I would not recommend it.

这篇关于使用 cython --embed 时静态链接 python37.dll 和 vcruntime140.dll的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！