加快distutils的构建过程

问题描述

我编程Python的C ++扩展，我使用distutils编译项目。随着项目的发展，重建需要更长时间。有没有办法加快构建过程？

我读取并行构建（如 make -j ）不可能与distutils。有没有更好的替代distutils可能更快？

我也注意到它是重新编译所有的对象文件每次我调用 python setup.py build ，即使我只更改了一个源文件。如果是这样，或者我可能在这里做错了？

如果它有帮助，这里有一些文件，我尝试编译：

谢谢！

解决方案

1. 尝试使用环境变量 CC =ccache gcc code>，这将加速构建显着时源没有改变。 （奇怪的是，distutils使用 CC 也可以用于c ++源文件）。



2. 因为你有一个由多个编译对象文件组成的扩展，你可以使用monkey-patch distutils来编译（它们是独立的） - 将其放入你的setup.py（根据需要调整 N = 2 ）：

    ＃monkey-patch用于并行编译
    def parallelCCompile（self，sources，output_dir = None，macros = None，include_dirs = None，debug = 0，extra_preargs = None ，extra_postargs = None，depends = None）：
   ＃这些行是从distutils.ccompiler.CCompiler直接复制的
   宏，对象，extra_postargs，pp_opts，build = self._setup_compile（output_dir，macros，include_dirs， source，depends，extra_postargs）
    cc_args = self._get_cc_args（pp_opts，debug，extra_preargs）
   ＃并行代码
    N = 2并行编译数
    import multiprocessing.pool 
    def _single_compile（obj）：
    try：src，ext = build [obj] 
    except KeyError：return 
    self._compile（obj，src，ext，cc_args，extra_postargs，pp_opts ）
   ＃转换为列表，imap按需求计算
    list（multiprocessing.pool.ThreadPool（N）.imap（_single_compile，objects））
   返回对象
    import distutils .ccompiler 
    distutils.ccompiler.CCompiler.compile = parallelCCompile 
     




I am programming a C++ extension for Python and I am using distutils to compile the project. As the project grows, rebuilding it takes longer and longer. Is there a way to speed up the build process?

I read that parallel builds (as with make -j) are not possible with distutils. Are there any good alternatives to distutils which might be faster?

I also noticed that it's recompiling all object files every time I call python setup.py build, even when I only changed one source file. Should this be the case or might I be doing something wrong here?

In case it helps, here are some of the files which I try to compile: https://gist.github.com/2923577

Thanks!

解决方案

1. Try building with environment variable CC="ccache gcc", that will speed up build significantly when the source has not changed. (strangely, distutils uses CC also for c++ source files). Install the ccache package, of course.

2. Since you have a single extension which is assembled from multiple compiled object files, you can monkey-patch distutils to compile those in parallel (they are independent) - put this into your setup.py (adjust the N=2 as you wish):

   # monkey-patch for parallel compilation
   def parallelCCompile(self, sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None):
       # those lines are copied from distutils.ccompiler.CCompiler directly
       macros, objects, extra_postargs, pp_opts, build = self._setup_compile(output_dir, macros, include_dirs, sources, depends, extra_postargs)
       cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
       # parallel code
       N=2 # number of parallel compilations
       import multiprocessing.pool
       def _single_compile(obj):
           try: src, ext = build[obj]
           except KeyError: return
           self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
       # convert to list, imap is evaluated on-demand
       list(multiprocessing.pool.ThreadPool(N).imap(_single_compile,objects))
       return objects
   import distutils.ccompiler
   distutils.ccompiler.CCompiler.compile=parallelCCompile
   

这篇关于加快distutils的构建过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！