如何使用正确的dll文件在Cython C扩展中启用第三方C库？

问题描述

我有一个C函数，涉及使用zstd解压缩数据。我正在尝试使用Cython调用该函数。

I have a C function that involves decompressing data using zstd. I am attempting to call that function using Cython.

作为指南，我可以毫无问题地编译并运行下面的代码。

Using this page from the docs as a guide I can compile and run the code below with no problem.

（我不会实际上是在这里使用zstd库）

(I don't actually use the zstd lib here)

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!\n");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   return 0;
}

# Hello.pyx

cdef extern from "hello.c":
  int hello()

cpdef int callHello():
  hello()

# hello_wrapper.setup.py

from setuptools import setup, Extension
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "hello_wrapper",
        ["hello_wrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["path/to/zstd/lib"],
        include_dirs=['path/to/zstd/include'],
    )
]

setup(
    ext_modules = cythonize(ext_modules, gdb_debug=True)
)

使用以下命令，我得到预期的输出：

Using the commands as follows I get the expected output:

>py hello_wrapper.setup.py build_ext --inplace
>py
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello_wrapper
>>> hello_wrapper.callHello()
Hello, World!
0

但是当我修改 hello.c 实际使用zstd库：

However when I modify hello.c to actually use the zstd library:

// hello.c
#include <stdio.h>
#include <zstd.h>

int hello() {
   printf("Hello, World!\n");
   void *next_in = malloc(0);
   void *next_out = malloc(0);
   size_t const dSize = ZSTD_decompress(next_out, 0, next_in, 0); //the added line
   return 0;
}

hello_wrapper.setup.py 编译时很好，当我进入import语句时，出现以下错误：

While hello_wrapper.setup.py compiles fine, when I get to the import statement, I get the following error:

>>> import hello_wrapper
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing hello_wrapper: The specified module could not be found.

从阅读，我认为这个错误意味着我没有正确指向或可能没有首先创建zstd.lib所需的DLL文件来发挥其魔力。这样对吗？如果是这样，我该怎么办？如果不是，那是什么 问题？

From reading This SO article, I gather that this error means I'm not correctly pointing to or perhaps creating in the first place the required DLL files for zstd.lib to work its magic. Is this correct? If so, how might I do that? If not, what is the problem?

推荐答案

我们将cython扩展名链接到Windows- dll，即：

We link our cython-extension against a windows-dll, that means:

• *。lib -file（即 zstd .lib ）在编译期间需要 path / to / zstd / lib


• *。dll -file（即 zstd.dll ）是需要的，Windows可以在该位置找到模块

• *.lib-file (i.e. zstd.lib) is needed in "path/to/zstd/lib" during the compile time
• *.dll-file (i.e. zstd.dll) is needed somewhere where Windows can find it when the module is imported.

通常，Windows不会在 path / to / zstd / lib 。因此，我们得到了一些含糊不清的错误消息：

Normally, Windows will not look in the "path/to/zstd/lib". And so we get a somewhat cryptic error message:

这并不意味着模块有问题-它恰好取决于无法找到的dll。

Which doesn't mean there is something wrong with the module - it just happens to depend on a dll which cannot be found.

在linux下具有，其中<$ c可以传递$ c> path / to / zstd / lib （可以将 runtime_library_dirs 参数添加到扩展），Windows上没有此类选项。 Windows的dll搜索算法。简而言之，将搜索dll（可能以此处显示的另一个顺序）

While linux has -rpath-option for the linker with which "path/to/zstd/lib" could be passed (it can be added with runtime_library_dirs-argument to Extension), there is no such option on Windows. The dll-search-algorithmus for Windows can be found here. In a nutshell, dll is searched in (possible in another order as presented here)

• 从中加载模块的目录。


• 当前目录。


• 系统目录（例如 C：\Windows\System32 ）


• windows目录（例如 C：\Windows ）


• PATH中列出的目录-variable


• 其他

• The directory from which the module is loaded.
• The current directory.
• system-directory (e.g. C:\Windows\System32)
• windows-directory(e.g. C:\Windows)
• directories that are listed in the PATH-variable
• others

将dll放入系统目录或Windows目录听起来不太吸引人，

Putting the dll into system- or windows-directory doesn't sound too appealing, which leave us with the following options:

• （最简单？）来复制 zstd.dll 在已编译扩展名旁边


• ，将zstd路径添加到 PATH 变量中，例如 set PATH = path / to / zstd / lib;％PATH％

• (the easiest?) to copy the zstd.dll next to the compiled extension
• to add the zstd-path to the PATH-variable, e.g. set PATH="path/to/zstd/lib";%PATH%

另一个选项有些棘手：

Another option is somewhat more tricky: Given that

我们可以使用 ctypes 来预加载正确的dll，在导入包装模块时将使用该dll（无需在光盘上搜索），即：

we can use ctypes to "preload" the right dll, which will be used (without the need to search for it on the disc) when the wrapper-module is imported, i.e.:

import ctypes;
ctypes.CDLL("path/to/zstd/lib/zstd.dll"); # we preload with the full path

import hello_wrapper  # works now!

---

如果扩展程序是在同一系统上构建和使用的，则上述适用 build_ext --inplace ）。安装/分发比较麻烦（此对此进行了介绍），一个想法是：

---

The above applies if the extension is built and used on the same system (e.g. via build_ext --inplace). installation/distribution is somewhat more cumbersome (this is covered by this SO-post), one idea would be:

• 放入 *。h -， *。lib -和 *。dll -文件放入 package_data中（反正似乎自动发生）


• 正确的相对 library_path （或以编程方式定义的绝对路径）可以在 setup.py 中设置，因此 *。lib 由链接器找到。


• dll将放在编译后的 *。pyd 文件旁边。

• to put *.h-, *.lib- and *.dll-files into 'package_data' (it seems to happen automatically anyway)
• the right relative library_path (or programmatically the absolute path) can be set in the setup.py so *.lib is found by the linker.
• dll will be put next to the compiled *.pyd-file in the installation.

一个示例可能是以下或多或少的最小 setup.py ，其中一切（pyx文件，h文件，lib文件，dll文件）都放入包/文件夹 src / zstd ：

An example could be the following more or less minimal setup.py, where everything (pyx-file, h-files, lib-file, dll-file) are put into a package/folder src/zstd:

from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize

ext_modules = [
    Extension(
        "zstd.zstdwrapper",
        ["src/zstd/zstdwrapper.pyx"],
        libraries=["zstd"],
        library_dirs=["src/zstd"],
        include_dirs=[], # set automatically to src/zstd during the build
    )
]

print(find_packages(where='src'))

setup(
    name = 'zstdwrapper',
    ext_modules = cythonize(ext_modules),
    packages = find_packages(where='src'),
    package_dir = {"": "src"},
)

现在可以安装了使用 python setup.py install 或用于创建例如通过 python setup.py sdist 分发源，然后可以通过 pip 安装。

And now it can be installed with python setup.py install or used to create e.g. a source-distribution via python setup.py sdist which then can be installed via pip.

这篇关于如何使用正确的dll文件在Cython C扩展中启用第三方C库？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！