如何使用正确的 dll 文件在 Cython C 扩展中启用第 3 方 C 库?

本文介绍了如何使用正确的 dll 文件在 Cython C 扩展中启用第 3 方 C 库?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个涉及使用 zstd 解压缩数据的 C 函数.我正在尝试使用 Cython 调用该函数.

使用此页面来自docs 作为指南，我可以毫无问题地编译和运行下面的代码.

(我实际上并没有在这里使用 zstd 库)

//hello.c#include #include 国际你好(){printf(你好，世界！
");void *next_in = malloc(0);void *next_out = malloc(0);返回0；}#你好.pyxcdef extern 来自hello.c":你好()cpdef int callHello():你好()# hello_wrapper.setup.py从 setuptools 导入设置，扩展从 Cython.Build 导入 cythonizeext_modules = [延期(hello_wrapper"，[hello_wrapper.pyx"]，图书馆=[zstd"]，library_dirs=[path/to/zstd/lib"],include_dirs=['path/to/zstd/include'],)]设置(ext_modules = cythonize(ext_modules, gdb_debug=True))

使用如下命令我得到了预期的输出:

>py hello_wrapper.setup.py build_ext --inplace>pyPython 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 位 (Intel)] on win32输入帮助"、版权"、信用"或许可证"想要查询更多的信息.>>>导入 hello_wrapper>>>hello_wrapper.callHello()你好世界！0

但是，当我修改 hello.c 以实际使用 zstd 库时:

//hello.c#include #include 国际你好(){printf(你好，世界！
");void *next_in = malloc(0);void *next_out = malloc(0);size_t const dSize = ZSTD_decompress(next_out, 0, next_in, 0);//添加的行返回0；}

虽然 hello_wrapper.setup.py 编译良好，但当我执行导入语句时，出现以下错误:

>>>导入 hello_wrapper回溯(最近一次调用最后一次):文件<stdin>"，第 1 行，在 <module> 中.导入错误:导入 hello_wrapper 时 DLL 加载失败:找不到指定的模块.

通过阅读这篇SO文章，我认为这个错误意味着我没有正确指向或可能首先创建 zstd.lib 所需的 DLL 文件以发挥其魔力.这个对吗?如果是这样，我该怎么做?如果不是，是什么问题?

解决方案

我们将 cython-extension 链接到 windows-dll，这意味着:

*.lib-file(即zstd.lib)需要在path/to/zstd/lib"中编译时间
*.dll 文件(即 zstd.dll)需要在导入模块时 Windows 可以找到的地方.

通常，Windows 不会查看 path/to/zstd/lib".所以我们得到了一个有点神秘的错误信息:

导入错误:DLL 加载失败:找不到指定的模块.

这并不意味着模块有问题 - 它只是碰巧依赖于找不到的 dll.

虽然 linux 有 -rpath-链接器的选项，"path/to/zstd/lib" 可以传递(它可以通过 runtime_library_dirs-argument 添加到 Extension)，Windows 上没有这个选项.Windows 的 dll 搜索算法可以在这里找到.简而言之，在(可能按此处所示的其他顺序)中搜索 dll

加载模块的目录.
当前目录.
系统目录(例如 C:WindowsSystem32)
windows 目录(例如 C:Windows)
在 PATH 变量中列出的目录
其他

将 dll 放入系统或 windows 目录听起来不太吸引人，这让我们有以下选择:

(最简单的?)复制编译扩展旁边的zstd.dll
将 zstd 路径添加到 PATH 变量，例如set PATH="path/to/zstd/lib";%PATH%

另一个选项有点棘手:鉴于

如果具有相同模块名称的 DLL 已经加载到内存中，则系统在解析之前只检查重定向和清单加载的 DLL，无论它在哪个目录中.系统不会不搜索 DLL.

我们可以使用 ctypes 来预加载"正确的 dll，将在导入包装器模块时使用(无需在光盘上搜索)，即:

import ctypes;ctypes.CDLL(path/to/zstd/lib/zstd.dll");# 我们预加载完整路径导入 hello_wrapper # 现在可以使用了！

如果扩展是在同一系统上构建和使用的(例如，通过 build_ext --inplace)，则上述内容适用.安装/分发有点麻烦(此SO-post涵盖了这一点)，一个想法是:

将 *.h-, *.lib- 和 *.dll- 文件放入'package_data'(似乎发生无论如何都是自动的)
可以在 setup.py 中设置正确的相对 library_path(或以编程方式设置的绝对路径)，因此可以通过以下方式找到 *.lib链接器.
dll 将放在安装中编译后的 *.pyd 文件旁边.

一个例子可能是以下或多或少的最小 setup.py，其中所有内容(pyx 文件、h 文件、lib 文件、dll 文件)都放入一个包/文件夹中src/zstd:

from setuptools import setup, Extension, find_packages从 Cython.Build 导入 cythonizeext_modules = [延期(zstd.zstdwrapper"，[src/zstd/zstdwrapper.pyx"]，图书馆=[zstd"]，library_dirs=[src/zstd"],include_dirs=[], # 在构建过程中自动设置为 src/zstd)]打印(find_packages(where='src'))设置(name = 'zstdwrapper',ext_modules = cythonize(ext_modules),包 = find_packages(where='src'),package_dir = {":src"}，)

现在它可以用 python setup.py install 安装或用于创建例如通过 python setup.py sdist 的源代码分发，然后可以通过 pip 安装.

I have a C function that involves decompressing data using zstd. I am attempting to call that function using Cython.

Using this page from the docs as a guide I can compile and run the code below with no problem.

(I don't actually use the zstd lib here)

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!
");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   return 0;
}

# Hello.pyx

cdef extern from "hello.c":
  int hello()

cpdef int callHello():
  hello()

# hello_wrapper.setup.py

from setuptools import setup, Extension
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "hello_wrapper",
        ["hello_wrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["path/to/zstd/lib"],
        include_dirs=['path/to/zstd/include'],
    )
]

setup(
    ext_modules = cythonize(ext_modules, gdb_debug=True)
)

Using the commands as follows I get the expected output:

>py hello_wrapper.setup.py build_ext --inplace
>py
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello_wrapper
>>> hello_wrapper.callHello()
Hello, World!
0

However when I modify hello.c to actually use the zstd library:

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!
");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   size_t const dSize = ZSTD_decompress(next_out, 0, next_in, 0); //the added line
   return 0;
}

While hello_wrapper.setup.py compiles fine, when I get to the import statement, I get the following error:

>>> import hello_wrapper
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing hello_wrapper: The specified module could not be found.

From reading This SO article, I gather that this error means I'm not correctly pointing to or perhaps creating in the first place the required DLL files for zstd.lib to work its magic. Is this correct? If so, how might I do that? If not, what is the problem?

解决方案

We link our cython-extension against a windows-dll, that means:

*.lib-file (i.e. zstd.lib) is needed in "path/to/zstd/lib" during the compile time
*.dll-file (i.e. zstd.dll) is needed somewhere where Windows can find it when the module is imported.

Normally, Windows will not look in the "path/to/zstd/lib". And so we get a somewhat cryptic error message:

Which doesn't mean there is something wrong with the module - it just happens to depend on a dll which cannot be found.

While linux has -rpath-option for the linker with which "path/to/zstd/lib" could be passed (it can be added with runtime_library_dirs-argument to Extension), there is no such option on Windows. The dll-search-algorithmus for Windows can be found here. In a nutshell, dll is searched in (possible in another order as presented here)

The directory from which the module is loaded.
The current directory.
system-directory (e.g. C:WindowsSystem32)
windows-directory(e.g. C:Windows)
directories that are listed in the PATH-variable
others

Putting the dll into system- or windows-directory doesn't sound too appealing, which leave us with the following options:

(the easiest?) to copy the zstd.dll next to the compiled extension
to add the zstd-path to the PATH-variable, e.g. set PATH="path/to/zstd/lib";%PATH%

Another option is somewhat more tricky: Given that

we can use ctypes to "preload" the right dll, which will be used (without the need to search for it on the disc) when the wrapper-module is imported, i.e.:

import ctypes;
ctypes.CDLL("path/to/zstd/lib/zstd.dll"); # we preload with the full path

import hello_wrapper  # works now!

The above applies if the extension is built and used on the same system (e.g. via build_ext --inplace). installation/distribution is somewhat more cumbersome (this is covered by this SO-post), one idea would be:

to put *.h-, *.lib- and *.dll-files into 'package_data' (it seems to happen automatically anyway)
the right relative library_path (or programmatically the absolute path) can be set in the setup.py so *.lib is found by the linker.
dll will be put next to the compiled *.pyd-file in the installation.

An example could be the following more or less minimal setup.py, where everything (pyx-file, h-files, lib-file, dll-file) are put into a package/folder src/zstd:

from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "zstd.zstdwrapper",
        ["src/zstd/zstdwrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["src/zstd"],
        include_dirs=[], # set automatically to src/zstd during the build
    )
]

print(find_packages(where='src'))

setup(
    name = 'zstdwrapper',
    ext_modules = cythonize(ext_modules),
    packages = find_packages(where='src'),
    package_dir = {"": "src"},
)

And now it can be installed with python setup.py install or used to create e.g. a source-distribution via python setup.py sdist which then can be installed via pip.

这篇关于如何使用正确的 dll 文件在 Cython C 扩展中启用第 3 方 C 库?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！