I’ve been working for the last couple of days in a small command line tool in Python, and I took the opportunity to check out a little bit Poetry, which seems to help in package and distribute Python modules.
Enter pyproject.toml
A very promising development in the Python ecosystem are the new pyproject.toml files, presented in PEP 518. This file aims to replace the old setup.py with a config file, to avoid executing arbitrary code, as well as clarify the usage.
Poetry generates a new project, and includes the corresponding pyproject.toml.
[tool.poetry] name = "std_encode" version = "0.2.1" description = "Encode and decode files through the standard input/output" homepage = "https://github.com/jaimebuelta/std_encode" repository = "https://github.com/jaimebuelta/std_encode" authors = ["Jaime Buelta "] license = "MIT" readme = "README.md" [tool.poetry.dependencies] python = ">=2.7, !=3.0, !=3.1, !=3.2, !=3.3, !=3.4, =0.12"] build-backend = "poetry.masonry.api"
Most of it is generated automatically by Poetry, there are a couple of interesting bits:
Python compatibility
[tool.poetry.dependencies] python = ">=2.7, !=3.0, !=3.1, !=3.2, !=3.3, !=3.4, <4"
This makes it compatible with Python 2.7 and Python 3.5 and later.
Including documentation automatically
The added README.md will be automatically included in the package.
Easy dependencies management
The dependencies are clearly stated, in particular, the difference between dev dependencies and regular dependencies. Poetry creates also a poetry.lock file that includes the versions, etc.
Scripts and entry points
This package creates command line tools. This is easy to do describing scripts.
[tool.poetry.scripts] std_decode = 'std_encode:console.run_sd' std_encode = 'std_encode:console.run_se'
They’ll call the function run_sd and run_se on the console.py file.
Testing the code with cram and tox
Cramtastic!
As the module is aimed as a command line tool, the best way of testing it is through command line actions. A great tool for that is cram. It allows to describe a test file as a series of command line actions and the returned standard output. For example:
Setup $ . $TESTDIR/setup.sh Run test $ echo 'test line' | std_decode test line
Any line starting with $ is a command, and any line following, the result, as it will appear in the console. There’s a plugin for pytest, so it can be integrated in a bigger test suite with other Python tests.
Ensuring installation and tests with tox
To run the tests, the process should be:
- Generate a package with your changes.
- Install it in a virtual environment.
- Run all the cram tests, that will call the installed command line scripts.
The best way of doing this is to use tox, that also adds the possibility of running it over different Python versions.
To do so, we create a tox.ini file
[tox]
isolated_build = true
envlist = py37,py27
[testenv]
whitelist_externals = poetry
commands =
poetry install -v
poetry run pytest {posargs} tests/
Which defines two environments to run the tests, Python 2.7 and 3.7, and for each poetry installs the package and then runs the tests using pytest.
Running
$ tox
Runs the whole suite, but while testing, to speed up development, you can do instead
$ tox -e py37 -- -k my_test
The parameter -e runs only in one environment, and anything after the -- will be transferred to pytest to select only a subset of tests, or any other possibility.
Locally, this allow to run and iterate on the package. But we also want to run the test remotely in CI fashion.
CI and deployment with Travis
Travis-CI is a great tool to setup in your open source repo. Enabling your GitHub repo can be done very quickly. But after enabling it to our repo, we need to define the .travis.yml file with info.
language: python
python:
- '2.7'
- '3.5'
- '3.6'
matrix:
include:
- python: 3.7
dist: xenial
before_install:
- pip install poetry
install:
- poetry install -v
- pip install tox-travis
script:
- tox
before_deploy:
- poetry config http-basic.pypi $PYPI_USER $PYPI_PASSWORD
- poetry build
deploy:
provider: script
script: poetry publish
on:
tags: true
condition: "$TRAVIS_PYTHON_VERSION == 3.7"
env:
global:
- secure: [REDACTED]
- secure: [REDACTED]
The first part defines the different build to run, for versions of Python 2.7, 3.5, 3.6 and 3.7.
Version 3.7 requires to be executed in Ubuntu Xenial (16.04), as by default travis uses Trusty (14.04), which doesn’t support Python 3.7.
language: python
python:
- '2.7'
- '3.5'
- '3.6'
matrix:
include:
- python: 3.7
dist: xenial
The next part describes how to run the tests. The package tox-travis is installed to seamless integrate both. This makes travis run versions of Python that are not included in tox.ini.
before_install: - pip install poetry install: - poetry install -v - pip install tox-travis script: - tox
Finally, a deployment part is added.
before_deploy:
- poetry config http-basic.pypi $PYPI_USER $PYPI_PASSWORD
- poetry build
deploy:
provider: script
script: poetry publish
on:
tags: true
condition: "$TRAVIS_PYTHON_VERSION == 3.7"
The deploy is configured to happen only if a git tag is set up, and if the build is using Python 3.7. The last condition can be removed, but then the package will be uploaded several times. Poetry ignores it if that’s the case, but it’s just wasteful.
The package is build before deploying it with poetry publish.
To properly configure the access to PyPI, we need to store in secure variables our login and password. To do so, install the travis command line tool, and encrypt the secrets, including the variable name.
$ travis encrypt PYPI_PASSWORD=<PASSWORD> --add env.global $ travis encrypt PYPI_USER=<USER> --add env.global
The line
poetry config http-basic.pypi $PYPI_USER $PYPI_PASSWORD
will configure poetry to use these credentials and upload the packages correctly.
Release flow
After all this in place, to prepare a new release of the package, the flow will be like this:
- Set up the new functionality and commits. Travis will run the tests to ensure that the build works as expected. This may include bumping the dependencies with
poetry update. - Once everything is ready, create a new commit with the new version information. This normally includes:
- Run
poetry version {patch|minor|major|...}to bump the version. - Set up any manual changes, like release notes, documentation updates or internal version references.
- Run
- Commit and verify that the build is green in travis.
- Create a new tag (or GitHub release) with the version. Remember to push the tag to GitHub.
- Travis will upload the new version automatically to PyPI.
- Spread the word! Your package deserves to be known!
The future, suggestions and things to keep an eye to
There are a couple of elements that could be a little bit easier in the process. As pyproject.toml and poetry are quite new there are a couple of rough edges that could be improved.
Tags, versions and releases
Poetry has a version command to bump the version, but its only effect is to change the pyproject.toml file. I’d love to see an integration to update more elements, including internal versions like the one in __init__.py that gets generated automatically, or ask for release notes and append them to a standard document.
There’s also no integration with generating a git tag or GitHub release in the same command. You need to perform all these commands manually, while it seems like they should be part of the same action.
Something like:
$ poetry version Generating version 3.4 Append release notes? [y/n]: Opening editor to add release notes Saved A new git commit and tag (v3.4) will be generated with the following changes: pyproject.toml - version: 3.3 + version: 3.4 src/package/__init__.py - __version__ = "3.3" + __version__ = "3.4" RELEASE_NOTES.md + Version 3.4 [2018-10-28] + === + New features and bugfixes Continue? [y/n/c(change tag name)] Creating and pushing... Waiting for CI confirmation CI build is green Creating new tag. Done. Create a new release in GitHub [y/n]
This is a wishlist, obviously, but I think it will fit the flow of a lot of GitHub releases to PyPI.
Travis work with Python 3.7 and Poetry
I’m pretty sure that travis will update the support for Python 3.7 quite soon. Having to define a different environment feels awkward, though I understand the underlying technical issues with it. It’s not a big deal, but I imagine that they’ll fix it so the definition is the same wether you work on 3.6 or 3.7. 3.7 was released 4 months ago at this time.
The other possible improvement is to add pyproject.toml support. At the moment setup.py uploads to PyPI is natively supported, so adding support for pyproject.toml will be amazing. I imagine it will be added if more projects uses this way of packaging more and more.
Final words
Having a CI running properly, and a deployment flow is actually a lot of work. Even doing it with great tools like the ones discussed here, there’s a lot of details to keep into account and polishing bits that need to be considered. It took me around a full day of experimentation to get this setup, even if I worked previously with travis (I configured it for ffind some time ago).
Poetry is also a very promising tool, and I’ll keep checking it. The packaging world in Python is complicated, but there has been a lot of work recently to improve it.
