本文介绍了如何构建和分发依赖于第三方libFoo.so的Python/Cython软件包的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我编写了一个依赖于某些C扩展的Python模块.这些C扩展又依赖于几个已编译的C库.我希望能够分发与所有依赖项捆绑在一起的该模块.

I've written a Python module that depends on some C extensions. Those C extensions depend in turn on several compiled C libraries. I'd like to be able to distribute this module bundled with all the dependencies.

我整理了一个最小的示例(可以在GitHub上完整找到它 ).

I've put together a minimal example (it can be found on GitHub in its entirety).

目录结构为:

$ tree .
.
├── README.md
├── poc
│   ├── __init__.py
│   ├── cython_extensions
│   │   ├── __init__.py
│   │   ├── cvRoberts_dns.c
│   │   ├── cvRoberts_dns.h
│   │   ├── helloworld.c
│   │   ├── helloworld.pxd
│   │   ├── helloworld.pyx
│   │   ├── test.c
│   │   └── test.h
│   ├── do_stuff.c
│   └── do_stuff.pyx
└── setup.py

setup.py构建扩展,并链接到必要的库(在本例中为libsundials_cvodelibsundials_nvectorserial):

setup.py builds the extensions, and links against the necessary libraries (libsundials_cvode, libsundials_nvectorserial in this case):

from setuptools import setup, find_packages
from setuptools.extension import Extension
from Cython.Build import cythonize


ext_module_dostuff = Extension(
    'poc.do_stuff',
    ['poc/do_stuff.pyx'],
)

ext_module_helloworld = Extension(
    'poc.cython_extensions.helloworld',
    ['poc/cython_extensions/helloworld.pyx', 'poc/cython_extensions/test.c', 'poc/cython_extensions/cvRoberts_dns.c'],
    include_dirs = ['/usr/local/include'],
    libraries = ['m', 'sundials_cvodes', 'sundials_nvecserial'],
    library_dirs = ['/usr/local/lib'],
)

cython_ext_modules = [
   ext_module_dostuff,
   ext_module_helloworld
]


setup (
  name = "poc",
  ext_modules = cythonize(cython_ext_modules),
  packages=['poc', 'poc.cython_extensions'],
)

这一切都很好,但是它确实要求最终用户首先安装sundials(在实际情况下,还需要安装其他几个非常挑剔的库才能启动和运行).

This is all well and good, but it does require that the end user first install sundials (and, in the actual case, several other libraries that are extremely finicky to get up and running).

理想情况下,我希望只能在开发计算机上进行设置,创建包含适当共享库的发行版,并提供某种捆绑软件.

Ideally, I'd like to be able to set this up only on development machines, create a distribution that includes the appropriate shared libraries, and ship some sort of bundle.

给出到目前为止我发现的各种教程,示例和SO帖子.我被引导相信自己在正确的道路上.但是,还有最后一步,我只是不理解.

Given the various tutorials, examples and SO posts I've found so far. I'm led to believe I'm on the right track. However, there's some sort of final step that I'm just not groking.

感谢您的帮助:-).

推荐答案

您可能已经知道,建议使用已编译组件分发Python模块的方法是使用车轮格式.似乎没有任何标准的跨平台方式将第三方本机库捆绑到一个轮子中.但是,有用于此目的的特定于平台的工具.

As you probably know, the recommended way of distributing a Python module with compiled components is to use the wheel format. There doesn't appear to be any standard cross-platform way of bundling third-party native libraries into a wheel. However, there are platform-specific tools for this purpose.

auditwheel 修改现有的Linux wheel文件以添加任何未包含的第三方库在基本的"许多Linux "标准中.以下是在全新安装的Ubuntu 17.10上如何将其与项目一起使用的演练:

auditwheel modifies an existing Linux wheel file to add any third-party libraries which are not included in the basic "manylinux" standard. Here's an walkthrough of how to use it with your project on a clean install of Ubuntu 17.10:

首先,安装基本的Python开发工具以及带有其标头的第三方库:

First, install basic Python development tools, and the third-party library with its headers:

root@ubuntu-17:~# apt-get install cython python-pip unzip
root@ubuntu-17:~# apt-get install libsundials-serial-dev

然后将您的项目构建到wheel文件中:

Then build your project into a wheel file:

root@ubuntu-17:~# cd cython-example/
root@ubuntu-17:~/cython-example# python setup.py bdist_wheel
[...]
root@ubuntu-17:~/cython-example# cd dist/
root@ubuntu-17:~/cython-example/dist# ll
total 80
drwxr-xr-x 2 root root  4096 Nov  8 11:28 ./
drwxr-xr-x 7 root root  4096 Nov  8 11:28 ../
-rw-r--r-- 1 root root 70135 Nov  8 11:28 poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Archive:  poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   116648  2017-11-08 11:28   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
      105  2017-11-08 11:28   poc-0.0.0.dist-info/WHEEL
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
      793  2017-11-08 11:28   poc-0.0.0.dist-info/RECORD
---------                     -------
   180382                     10 files

wheel文件现在可以在本地安装并经过测试:

The wheel file can now be installed locally and tested:

root@ubuntu-17:~/cython-example/dist# pip install poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

现在,我们安装auditwheel工具.它需要Python 3,但能够处理Python 2或3的轮子.

Now we install the auditwheel tool. It requires Python 3, but it's capable of processing wheels for Python 2 or 3.

root@ubuntu-17:~/cython-example/dist# apt-get install python3-pip
root@ubuntu-17:~/cython-example/dist# pip3 install auditwheel

auditwheel使用另一个名为patchelf的工具来完成其工作.不幸的是,Ubuntu 17.10随附的patchelf版本缺少错误修正,而没有 auditwheel不起作用.因此,我们必须从源代码构建它(脚本摘录自 manylinux Docker映像):

auditwheel uses another tool called patchelf to do its job. Unfortunately, the version of patchelf included with Ubuntu 17.10 is missing a bugfix without which auditwheel will not work. So we'll have to build it from source (script taken from the manylinux Docker image):

root@ubuntu-17:~# apt-get install autoconf
root@ubuntu-17:~# PATCHELF_VERSION=6bfcafbba8d89e44f9ac9582493b4f27d9d8c369
root@ubuntu-17:~# curl -sL -o patchelf.tar.gz https://github.com/NixOS/patchelf/archive/$PATCHELF_VERSION.tar.gz
root@ubuntu-17:~# tar -xzf patchelf.tar.gz
root@ubuntu-17:~# (cd patchelf-$PATCHELF_VERSION && ./bootstrap.sh && ./configure && make && make install)

现在,我们可以检查方向盘需要哪些第三方库:

Now we can check which third-party libraries the wheel requires:

root@ubuntu-17:~/cython-example/dist# auditwheel show poc-0.0.0-cp27-cp27mu-linux_x86_64.whl

poc-0.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.4',
'GLIBC_2.2.5', 'GLIBC_2.3.4'}

The following external shared libraries are required by the wheel:
{
    "libblas.so.3": "/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1",
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.26.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libgfortran.so.4": "/usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0",
    "liblapack.so.3": "/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.26.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.26.so",
    "libquadmath.so.0": "/usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0",
    "libsundials_cvodes.so.2": "/usr/lib/libsundials_cvodes.so.2.0.0",
    "libsundials_nvecserial.so.0": "/usr/lib/libsundials_nvecserial.so.0.0.2"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libblas.so.3, libgfortran.so.4, liblapack.so.3, libquadmath.so.0,
libsundials_cvodes.so.2, libsundials_nvecserial.so.0

并创建一个将它们捆绑在一起的新轮子:

And create a new wheel which bundles them:

root@ubuntu-17:~/cython-example/dist# auditwheel repair poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Repairing poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libsundials_nvecserial.so.0.0.2 -> poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
Grafting: /usr/lib/libsundials_cvodes.so.2.0.0 -> poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
Grafting: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 -> poc/.libs/liblapack-549933c4.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 -> poc/.libs/libblas-52fa99c8.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0 -> poc/.libs/libgfortran-2df4b07d.so.4.0.0
Grafting: /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0 -> poc/.libs/libquadmath-0d7c3070.so.0.0.0
Setting RPATH: poc/cython_extensions/helloworld.so to "$ORIGIN/../.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

Fixed-up wheel written to /root/cython-example/dist/wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Archive:  wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
     1400  2017-11-08 12:08   poc-0.0.0.dist-info/RECORD
      110  2017-11-08 12:08   poc-0.0.0.dist-info/WHEEL
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   131712  2017-11-08 12:08   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
   230744  2017-11-08 12:08   poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
  7005072  2017-11-08 12:08   poc/.libs/liblapack-549933c4.so.3.7.1
   264024  2017-11-08 12:08   poc/.libs/libquadmath-0d7c3070.so.0.0.0
  2039960  2017-11-08 12:08   poc/.libs/libgfortran-2df4b07d.so.4.0.0
    17736  2017-11-08 12:08   poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
   452432  2017-11-08 12:08   poc/.libs/libblas-52fa99c8.so.3.7.1
---------                     -------
 10206026                     16 files

如果我们卸载第三方库,则先前安装的滚轮将停止工作:

If we uninstall the third-party libraries, the previously-installed wheel will stop working:

root@ubuntu-17:~/cython-example/dist# apt-get remove libsundials-serial-dev && apt-get autoremove
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "poc/do_stuff.pyx", line 1, in init poc.do_stuff
ImportError: libsundials_cvodes.so.2: cannot open shared object file: No such file or directory

但是带有捆绑库的轮子可以正常工作:

But the wheel with the bundled libraries will work fine:

root@ubuntu-17:~/cython-example/dist# pip uninstall poc
[...]
root@ubuntu-17:~/cython-example/dist# pip install wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

在OSX上,使用delocate.

用于OSX的

delocate 显然与auditwheel非常相似.不幸的是,我没有OSX机器可提供演练.

On OSX, use delocate.

delocate for OSX apparently works very similarly to auditwheel. Unfortunately I don't have an OSX machine available to provide a walkthrough.

使用这两种工具的一个项目是SciPy. 此存储库,尽管名称如此,但包含所有平台(不仅仅是Mac)的官方SciPy构建过程.具体来说,比较 Linux构建脚本(使用auditwheel ),使用 OSX构建脚本(使用delocate ).

One project which uses both tools is SciPy. This repository, despite its name, contains the official SciPy build process for all platforms, not just Mac. Specifically, compare the Linux build script (which uses auditwheel), with the OSX build script (which uses delocate).

要查看此过程的结果,您可能需要从PyPI下载并解压缩一些 SciPy轮子.例如,scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl包含以下内容:

To see the result of this process, you might want to download and unzip some of the SciPy wheels from PyPI. For example, scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl contains the following:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so
  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

scipy-1.0.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl包含以下内容:

   273072  2017-10-25 07:03   scipy/.dylibs/libgcc_s.1.dylib
  1550456  2017-10-25 07:03   scipy/.dylibs/libgfortran.3.dylib
   279932  2017-10-25 07:03   scipy/.dylibs/libquadmath.0.dylib

这篇关于如何构建和分发依赖于第三方libFoo.so的Python/Cython软件包的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!

09-05 10:44
查看更多