如何组织一个包含多个包的 python 项目，以便包中的每个文件仍然可以单独运行?

问题描述

TL;DR

这是一个示例存储库，其设置如第一张图(下图)所述:https://github.com/Poddster/package_problems

Here's an example repository that is set up as described in the first diagram (below): https://github.com/Poddster/package_problems

如果您可以让它在项目组织方面看起来像第二张图并且仍然可以运行以下命令，那么您已经回答了这个问题:

If you could please make it look like the second diagram in terms of project organisation and can still run the following commands, then you've answered the question:

$ git clone https://github.com/Poddster/package_problems.git
$ cd package_problems
<do your magic here>

$ nosetests

$ ./my_tool/my_tool.py
$ ./my_tool/t.py
$ ./my_tool/d.py

 (or for the above commands, $ cd ./my_tool/ && ./my_tool.py is also acceptable)

或者:给我一个不同的项目结构，允许我将相关文件('包')组合在一起，单独运行所有文件，将文件导入同一包中的其他文件，并将包/文件导入到其他包的文件.

Alternatively: Give me a different project structure that allows me to group together related files ('package'), run all of the files individually, import the files into other files in the same package, and import the packages/files into other package's files.

我有一堆 python 文件.它们中的大多数在可从命令行调用时很有用，即它们都使用 argparse 和 if __name__ == "__main__" 来做有用的事情.

I have a bunch of python files. Most of them are useful when callable from the command line i.e. they all use argparse and if __name__ == "__main__" to do useful things.

目前我有这个目录结构，一切正常:

Currently I have this directory structure, and everything is working fine:

.
├── config.txt
├── docs/
│   ├── ...
├── my_tool.py
├── a.py
├── b.py
├── c.py
├── d.py
├── e.py
├── README.md
├── tests
│   ├── __init__.py
│   ├── a.py
│   ├── b.py
│   ├── c.py
│   ├── d.py
│   └── e.py
└── resources
    ├── ...

一些脚本import 来自其他脚本的东西来完成它们的工作.但是没有脚本仅仅是一个库，它们都是可调用的.例如我可以调用 ./my_tool.py、./a.by、./b.py、./c.py 等，他们会为用户做有用的事情.

Some of the scripts import things from other scripts to do their work. But no script is merely a library, they are all invokable. e.g. I could invoke ./my_tool.py, ./a.by, ./b.py, ./c.py etc and they would do useful things for the user.

my_tool.py"是利用所有其他脚本的主脚本.

"my_tool.py" is the main script that leverages all of the other scripts.

但是我想改变项目的组织方式.该项目本身代表了一个可供用户使用的整个程序，并将按原样分发，但我知道它的一部分将在以后的不同项目中有用，所以我想尝试将当前文件封装到一个包中.在不久的将来，我还将在同一个项目中添加其他包.

However I want to change the way the project is organised. The project itself represents an entire program useable by the user, and will be distributed as such, but I know that parts of it will be useful in different projects later so I want to try and encapsulate the current files into a package. In the immediate future I will also add other packages to this same project.

为了促进这一点，我决定将项目重新组织为以下内容:

To facilitate this I've decided to re-organise the project to something like the following:

.
├── config.txt
├── docs/
│   ├── ...
├── my_tool
│   ├── __init__.py
│   ├── my_tool.py
│   ├── a.py
│   ├── b.py
│   ├── c.py
│   ├── d.py
│   ├── e.py
│   └── tests
│       ├── __init__.py
│       ├── a.py
│       ├── b.py
│       ├── c.py
│       ├── d.py
│       └── e.py
├── package2
│   ├── __init__.py
│   ├── my_second_package.py
|   ├── ...
├── README.md
└── resources
    ├── ...

但是，我想不出满足以下条件的项目组织:

However, I can't figure out an project organisation that satisfies the following criteria:

1. 所有脚本都可以在命令行上调用(作为 my_toola.py 或 cd my_tool && a.py)
2. 测试实际运行:)
3. package2 中的文件可以做 import my_tool

主要问题在于包和测试使用的导入语句.

The main problem is with the import statements used by the packages and the tests.

目前，包括测试在内的所有包都只需执行 import <module> 即可正确解析.但是，当在它周围晃动东西时，它就不起作用了.

Currently, all of the packages, including the tests, simply do import <module> and it's resolved correctly. But when jiggering things around it doesn't work.

请注意，支持 py2.7 是一项要求，因此所有文件的顶部都有 from __future__ import absolute_import, ....

Note that supporting py2.7 is a requirement so all of the files have from __future__ import absolute_import, ... at the top.

如果我如上所示移动文件，但将所有导入语句保留为当前状态:

If I move the files around as shown above, but leave all of the import statements as they currently are:

1. $ ./my_tool/*.py 可以正常工作并且它们都运行正常
2. $nosetests 从顶层目录运行不起作用.测试无法导入包脚本.
3. pycharm 在编辑这些文件时以红色突出显示导入语句:(

1. $ ./my_tool/*.py works and they all run properly
2. $ nosetests run from the top directory doesn't work. The tests fail to import the packages scripts.
3. pycharm highlights import statements in red when editing those files :(

2

如果我将测试脚本更改为:

2

If I then change the test scripts to do:

from my_tool import x

1. $ ./my_tool/*.py 仍然有效，并且它们都运行正常
2. $nosetests 从顶层目录运行不起作用.然后测试可以导入正确的脚本，但是当测试脚本导入它们时，脚本本身的导入会失败.
3. pycharm 在主脚本中仍以红色突出显示导入语句:(

1. $ ./my_tool/*.py still works and they all run properly
2. $ nosetests run from the top directory doesn't work. Then tests can import the correct scripts, but the imports in the scripts themselves fail when the test scripts import them.
3. pycharm highlights import statements in red in the main scripts still :(

3

如果我保持相同的结构并将 everything 更改为 from my_tool import 那么:

1. $ ./my_tool/*.py 导致 ImportErrors
2. $ nosetests 一切正常.
3. pycharm 不会抱怨任何事情

1. $ ./my_tool/*.py results in ImportErrors
2. $ nosetests runs everything ok.
3. pycharm doesn't complain about anything

例如1 个:

Traceback (most recent call last):
  File "./my_tool/a.py", line 34, in <module>
    from my_tool import b
ImportError: cannot import name b

4

我也试过 from .import x 但这只是以 ValueError: Attempted relative import in non-package 来直接运行脚本.

4

I also tried from . import x but that just ends up with ValueError: Attempted relative import in non-package for the direct running of scripts.

我不能只使用 python -m pkg.tests.core_test as

a) 我没有 ma​​in.py.我想我可以拥有一个?
b) 我希望能够运行所有脚本，而不仅仅是主脚本?

a) I don't have main.py. I guess I could have one?
b) I want to be able to run all of the scripts, not just main?

我试过了:

if __name__ == '__main__' and __package__ is None:
    from os import sys, path
    sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))

但这没有帮助.

我也试过了:

__package__ = "my_tool"
from . import b

但收到了:

SystemError: Parent module 'loading_tool' not loaded, cannot perform relative import

在 from 之前添加 import my_tool.import b 以 ImportError: cannot import name b

adding import my_tool before from . import b just ends up back with ImportError: cannot import name b

什么是正确的魔法咒语和目录布局来完成所有这些工作?

What's the correct set of magical incantations and directory layout to make all of this work?

推荐答案

一旦你转移到你想要的配置，你用来加载特定于 my_tool 的模块的绝对导入不再起作用.

Once you move to your desired configuration, the absolute imports you are using to load the modules that are specific to my_tool no longer work.

在创建 my_tool 子目录并将文件移入其中后，您需要进行三处修改:

You need three modifications after you create the my_tool subdirectory and move the files into it:

1. 创建 my_tool/__init__.py.(您似乎已经这样做了，但为了完整起见，我想提一下.)

1. Create my_tool/__init__.py. (You seem to already do this but I wanted to mention it for completeness.)

在 my_tool 下的文件中:更改 import 语句以从当前包中加载模块.所以在 my_tool.py 改变:

In the files directly under in my_tool: change the import statements to load the modules from the current package. So in my_tool.py change:

import c
import d
import k
import s

到:

from . import c
from . import d
from . import k
from . import s

您需要对所有其他文件进行类似的更改.(您提到尝试设置 __package__ 然后进行相对导入，但不需要设置 __package__.)

You need to make a similar change to all your other files. (You mention having tried setting __package__ and then doing a relative import but setting __package__ is not needed.)

在位于 my_tool/tests 的文件中:将导入要测试的代码的 import 语句更改为从一个包中加载的相对导入在层次结构中.所以在 test_my_tool.py 改变:

In the files located in my_tool/tests: change the import statements that import the code you want to test to relative imports that load from one package up in the hierarchy. So in test_my_tool.py change:

import my_tool

到:

from .. import my_tool

对于所有其他测试文件也是如此.

Similarly for all the other test files.

通过上面的修改，我可以直接运行模块了:

With the modifications above, I can run modules directly:

$ python -m my_tool.my_tool
C!
D!
F!
V!
K!
T!
S!
my_tool!
my_tool main!
|main tool!||detected||tar edit!||installed||keys||LOL||ssl connect||parse ASN.1||config|

$ python -m my_tool.k
F!
V!
K!
K main!
|keys||LOL||ssl connect||parse ASN.1|

我可以运行测试:

$ nosetests
........
----------------------------------------------------------------------
Ran 8 tests in 0.006s

OK

请注意，我可以同时使用 Python 2.7 和 Python 3 运行上述程序.

Note that I can run the above both with Python 2.7 and Python 3.

与其让my_tool下的各个模块直接可执行，我建议使用合适的setup.py文件来声明入口点，让setup.py 在安装包时创建这些入口点.由于您打算分发此代码，因此无论如何您都应该使用 setup.py 来正式打包它.

Rather than make the various modules under my_tool be directly executable, I suggest using a proper setup.py file to declare entry points and let setup.py create these entry points when the package is installed. Since you intend to distribute this code, you should use a setup.py to formally package it anyway.

1. 修改可以从命令行调用的模块，以my_tool/my_tool.py为例，而不是这样:

if __name__ == "__main__":
    print("my_tool main!")
    print(do_something())

你有:

def main():
    print("my_tool main!")
    print(do_something())

if __name__ == "__main__":
    main()

创建一个包含正确 entry_points 的 setup.py 文件.例如:

from setuptools import setup, find_packages

setup(
    name="my_tool",
    version="0.1.0",
    packages=find_packages(),
    entry_points={
        'console_scripts': [
            'my_tool = my_tool.my_tool:main'
        ],
    },
    author="",
    author_email="",
    description="Does stuff.",
    license="MIT",
    keywords=[],
    url="",
    classifiers=[
    ],
)

上面的文件指示 setup.py 创建一个名为 my_tool 的脚本，该脚本将调用模块 my_tool 中的 main 方法.my_tool.在我的系统上，一旦安装了软件包，就会有一个位于 /usr/local/bin/my_tool 的脚本调用 my_tool.my_tool 中的 main 方法.它产生与运行 python -m my_tool.my_tool 相同的输出，如上所示.

The file above instructs setup.py to create a script named my_tool that will invoke the main method in the module my_tool.my_tool. On my system, once the package is installed, there is a script located at /usr/local/bin/my_tool that invokes the main method in my_tool.my_tool. It produces the same output as running python -m my_tool.my_tool, which I've shown above.

这篇关于如何组织一个包含多个包的 python 项目，以便包中的每个文件仍然可以单独运行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！