自动化Python软件包发布过程

问题描述

我刚刚启动了一个开源Python项目，希望有一天它会流行起来.目前要发布新版本，我需要做一些事情.

I've just started an open source Python project that I hope might be popular one day. At the moment to release a new version I have to do a few things.

1. 测试所有东西.
2. 编辑 mypackage.VERSION 变量，该变量从 __ init __
导入 setup.py
3. 使用 python setup.py sdist bdist_wheel
构建包和轮子
4. 将变更日志条目写入 CHANGELOG 文件
5. 提交我的更改，回显一些更改日志
6. 标记为发布的标签，再次复制该变更日志条目.
7. 拖动我的内置文件，以便人们可以从发行版中下载它们
8. 使用Twine将软件包推到PyPI上
9. 通过PyPI在我的登台服务器上再次测试.

1. Test all the things.
2. Edit mypackage.VERSION variable, which setup.py imports from __init__
3. Build packages and wheels with python setup.py sdist bdist_wheel
4. Write a changelog entry to CHANGELOG file
5. Commit my changes, echo some of that changelog
6. Tag that commit as a release, copy that changelog entry over again.
7. Drag in my built files so people can download them from the release
8. Use Twine to push the packages up onto PyPI
9. Test again on my staging server via PyPI.

如果我必须在9个要点上总结我对项目的所有不满，我想我们会在一个非常相似的列表中找到.切入的是，过去我编了一个新版本号并写了commit/changelog消息，这很痛苦，很枯燥.

If I had to sum up everything I hate about my project in nine bullet points, I think we'd be looking at a very similar list. The thing that cuts is that past me making up a new version number and writing the commit/changelog message, this is painfully dull.

我是否可以通过某种方式使这些任务自动化，例如，让GitHub CI仅凭我的提交就可以完成一切?

Can I automate any of these tasks in such a way that I might be able to, for example, let GitHub CI do everything just from my commits?

我已经拥有十年的Python经验和一点CI，但是对于打包Python并与PyPI进行积极的交互我还是很陌生.我怀疑我不是唯一一个在这里因手动重复而发疯的人，我只是在寻找可以简化此过程的工具(或服务).

I already have a decade of Python experience, and a bit of CI, but I'm very new to packaging Python and actively interacting with PyPI. I suspect I'm not the only person driven crazy by the manual repetition here, I'm just looking for tools (or services) that can make this process easier.

推荐答案

以下是我对您的清单的看法.您可以实现一定范围的自动化，我将尝试提供一个合理的起点，然后提供一些提示，告诉您如何从那里进一步发展.

The following is my own opinionated take on your list. There is a certain range of automation you can achieve, and I'll try to provide a reasonable starting point, and then some hints on how you can go further from there.

采用此部分应该已经摆脱了大多数烦人的手动工作，并且您可以根据需要使自动化程度越来越高.如果您不愿意维护大量的CI代码，则应从此处开始.

Adopting this part should already get rid of most of the annoying manual work, and you can automate away more and more as the need arises. If you're not comfortable maintaining a good amount of CI code, you should start here.

您需要的是一个CI(如您已经提到的)和一个程序包管理器.您无法解决的事情是使用git推送您的更改和新标签，因此第5步和第6步的部分内容仍为手动操作.

Things you'll need are a CI (as you already noted) and a package manager. Something you won't get around is pushing your changes and a new tag with git, so parts of step 5 and 6 remain manual.

我将使用诗歌保持简洁，因为我喜欢它，但还有其他选项.这将处理第2、3、7、8步以及未列出的第10步，更新我的依赖关系并测试它们的兼容性"，一旦发现有问题，这将令人讨厌.

I'll use poetry to keep things concise and because I like it, but there are also other options. This will take care of steps 2, 3, 7, 8, and the unlisted step 10, "update my dependencies and test them for compatibility", which is incredibly annoying as soon as it turns out to be a problem.

使用诗歌时的坏消息是，您需要将所有打包配置都移到新文件 pyproject.toml 中.好消息是，您不需要单独的 setup.py ， setup.cfg ， MANIFEST.in 或从 pyproject.toml 是包装和其他工具的临时标准，诗歌还具有演练移植所有相关信息.

The bad news when using poetry is that you'll need to move all packaging configuration into a new file, pyproject.toml. The good news is, that you don't need a separate setup.py, setup.cfg, MANIFEST.in, or requirements.txt any more, since pyproject.toml is a provisional standard for packaging and other tools, and poetry also has a walkthrough on how to port over all the relevant info.

设置完成后，新的部署工作流程将为:

Once the setup is ready, the new deployment workflow would be:

$ poetry update           # update dependencies, may be skipped
$ poetry version          # bump version
Bumping version from 1.1.2 to 1.1.3
# finalize git stuff, e.g. add -u, commit -m 'v1.1.3', tag v1.1.3, push
$ poetry publish --build  # build and publish to PyPI
Building my_django_lib (1.1.3)
 - Building sdist
 - Built my_django_lib-1.1.3.tar.gz

 - Building wheel
 - Built my_django_lib-1.1.3-py3-none-any.whl

Publishing my_django_lib (1.1.3) to PyPI
 - Uploading my_django_lib-1.1.3-py3-none-any.whl 100%
 - Uploading my_django_lib-1.1.3.tar.gz 100%

这应该已经比您目前正在做的要短得多.如果您始终执行完全相同的git命令，不怕自动执行推送，并妥善保管 .gitignore 文件，请随时在〜/中添加类似此功能的内容.bashrc 并调用它:

This should already be a lot shorter than what you're currently doing. If you always execute the exact same git commands, are not afraid to automate a push, and take good care of your .gitignore file, feel free to add something like this function to your ~/.bashrc and call it instead:

git_cord () {
  version=$(grep pyproject.toml -e '(?<=^version = ")(.*)(?=")' -Po)
  git add -u
  git commit -m "${version}"
  git tag "${version}"
  git push -u origin "${version}"
}

开始使用gitlab-CI

CI原则上可以处理部署过程中的所有事宜，包括版本变更和发布.但是第一个要求您的CI可以推送到您的存储库(这会带来令人讨厌的副作用)，而第二个要求它可以将其发布到您的PyPI(这是有风险的，并且使调试CI变得很痛苦).我认为喜欢手工完成这两个步骤并不稀奇，因此这种最小的方法只能处理步骤1和9.之后可以进行更广泛的测试和构建工作.

Getting started with gitlab-CI

The CI can in principle handle everything surrounding the deployment process, including version bumping and publishing. But the first requires that your CI can push to your repo (which has annoying side effects) and the latter that it can publish to your PyPI (which is risky, and makes debugging the CI a pain). I think it's not unusual to prefer to do those two steps by hand, so this minimal approach will only handle step 1 and 9. More extensive testing and build jobs can be included afterwards.

配置项的正确设置取决于您计划使用的配置项.github的列表很长，所以我将重点放在gitlab的内置CI上.它是免费的，几乎没有什么魔力(这使得它具有可移植性)，并且CI运行器的二进制文件打开，免费，并已实际记录，因此您可以在本地调试配置项，或者在免费运行商无法为您效劳的情况下启动并连接新运行商.

The correct setup of a CI depends on which one you plan to use. The list for github is long, so I'll instead focus on gitlab's builtin CI. It's free, has very little magic (which makes it comparably portable), and the binaries for the CI runners are open, free, and actually documented, so you can debug your CI locally or start and connect new runners if the free ones don't cut it for you.

这是一个小的 .gitlab-ci.yml ，您可以将其放入项目根目录以运行测试.流水线中的每个作业(跳过设置和安装命令)也应该在您的开发环境中可执行，保持这种方式可以提高维护人员的经验.

Here is a small .gitlab-ci.yml that you can put into you project root in order to run the tests. Every single job in the pipeline (skipping setup and install commands) should also be executable in your dev environment, keeping it that way makes for a better maintainer-experience.

image: python:3.7-alpine

stages:
  - build
  - test

packaging:
  stage: build
  script:
    - pip install poetry
    - poetry build
  artifacts:
    paths:
      - dist

pytest:
  stage: test
  script:
    - pip install dist/*.whl
    - pip install pytest
    - pytest

像这样设置 build 和 test 阶段可一次处理第1步和第9步，同时还针对已安装的软件包而不是源文件运行测试套件.尽管只有在您的项目中有src布局，它才能正常工作，这会使本地资源无法从项目根目录导入.此处和此处.

Setting up the build and test stage like this handles steps 1 and 9 in one swoop, while also running the test suite against the installed package instead of your source files. Though it will only work properly if you have have a src-layout in your project, which makes local sources unimportable from the project root. Some info on why that would be a good idea here and here.

诗歌可以创建一个src-layout模板，您可以使用 poetry new my_django_lib --src 将代码移入其中.

Poetry can create a src-layout template you can move your code into with poetry new my_django_lib --src.

虽然那里有工具可以根据提交消息自动创建变更日志，但保持良好的变更日志是其中的一项得益于手工护理的好处.因此，我的建议是第4步不是自动化的.

While there are tools out there that automatically create a changelog from commit messages, keeping a good changelog is one of those things that benefit greatly from being cared for by hand. So, my advice is no automation for step 4.

一种思考方式是，手动 CHANGELOG 文件包含与您的用户相关的信息，并且仅应包含新功能，重要的错误修正和弃用之类的信息.

One way to think about it is that the manual CHANGELOG file contains information that is relevant to your users, and should only feature information like new features, important bugfixes, and deprecations.

更细粒度的信息可能对MR，提交消息或问题讨论有所帮助，这些信息可能对贡献者或插件编写者很重要，不应将其纳入 CHANGELOG 中.您可以尝试以某种方式收集它，但是导航这样的 AUTOLOG 可能和筛选刚才提到的主要资源一样麻烦.

More fine grained information that might be important for contributors or plugin writers would be located in MRs, commit messages, or issue discussions, and should not make it into the CHANGELOG. You can try to collect it somehow, but navigating such an AUTOLOG is probably about as cumbersome as sifting through the primary sources I just mentioned.

因此，简而言之，可以跳过第5步和第6步中与变更日志相关的部分.

So in short, the changelog-related parts of step 5 and 6 can be skipped.

添加CD不会发生太大变化，只是您不必再手动释放.如果CI下降，越野车出现，或者您不想等待管道发布修补程序，您仍然可以以诗歌形式发布.

Adding CD doesn't change too much, except that you don't have to release by hand any more. You can still release with poetry in case the CI is down, buggy, or you don't want to wait for the pipeline to release a hotfix.

这将通过以下方式改变工作流程:

This would alter the workflow in the following way:

• 日常工作
  • 编写代码(目前无法避免)
  • 记录提交消息和/或MR中的进度(即使是我自己的更改，我也更喜欢MR，并且压缩合并时的所有提交)
  • 推送到gitlab/合并MRs
  • 创建标签，运行诗歌版本，也许运行 poetry update
  • 在 CHANGELOG
  中编写发行说明
  • 推送到gitlab

  如果您提供秘密 PYPI_USER 和 PYPI_PASSWORD :

  stages:
    - build
    - test
    - release
  
  [...]  # packaging and pytest unchanged
  
  upload:
    stage: release
    only:
      - tags
      # Or alternatively "- /^v\d+\.\d+\.\d+/" if you also use non-release
      # tags, the regex only matches tags that look like this: "v1.12.0"
    script:
      - pip install poetry
      - poetry publish -u ${PYPI_USER} -p ${PYPI_PASSWORD} dist/*
  

  ---

  一些有用的链接:

  ---

  Some useful links:

  • .gitlab-ci.yml 文档
  • 预定义变量列表，这是大多数gitlab CI的位置含糊不清的谎言
  • 我的的长版本.gitlab-ci.yml 模板，以及可能对您有用或无用的其他阶段.它期望代码的src布局.
    • lint :类型检查，覆盖率和代码样式
    • 安全性:检查您自己的代码和您的依存关系以获取易用性
    • release.docs :公共gitlab页面部分，在其中提供自动创建文档的文档根据您的文档字符串
    • build 阶段从 poetry.lock 文件创建一个操舵室，该文件可用于稍后安装依赖项以支持PyPI.这样可以更快一点，可以节省网络带宽，并且可以在调试时断言使用特定版本，但是这样做可能会过大，并且需要使用诗歌预发行版本.
    • .gitlab-ci.yml documentation
    • list of predefined variables, this is where most of gitlab CI's obscurities lie
    • the long version of my .gitlab-ci.yml template, with additional stages that may or may not be useful to you. It expects a src layout of your code.
      • lint: type checking, coverage, and code style
      • security: checking your own code and your dependencies for valnuarabilities
      • release.docs: public gitlab pages section where docs are served that are created automatically based on your docstrings
      • The build stage creates a wheelhouse from the poetry.lock file that can be used for installing dependencies later in favor of PyPI. This is a little faster, saves network bandwidth, and asserts the use of specific versions if you want to debug, but might be overkill and requires the use of a poetry pre-release.

      这篇关于自动化Python软件包发布过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！