An Opinionated tox.ini File

This post assumes some knowledge of tox. For a beginner’s introduction see my tox tutorial.

Below is an excerpted version of the tox.ini file from hypothesis/h as of November 2018 (yes we’re still using Python 2). It demonstrates a particular way of using tox in which all of the tox commands are of the form tox -e pyXY-<task>. That is: use a particular version of Python (the pyXY part) to run a particular development task (the <task>) part:

tox -e py27-dev runs the app in the development server.
tox -e py36-dev runs the app in the development server using Python 3.
tox -e py27-tests or tox -e py36-tests runs the tests in Python 2 or 3.
tox -e py36-lint runs the linter and prints out any warnings.
tox -e py36-docs builds the docs and serves them locally with live reloading.
tox -e py36-coverage prints the coverage report.
Not shown below, but tox -e py36-format formats the code using Black.
Other tasks, such as deploying a new release, could be added too.

We see tox as a general, virtualenv-based development task automation tool, and don’t just use it for running tests. We use tox to standardise and automate any development task that benefits from being run in its own isolated venv.

Using tox’s factors and conditionals (see below) you can implement this pyXY-<task> approach in a simple and concise tox.ini, without duplication or boilerplate:

[tox]
envlist = py27-tests
skipsdist = true
# Enable the venv_update extension so that changes to requirements files are
# automatically detected.
requires = tox-pip-extensions
tox_pip_extensions_ext_venv_update = true

[testenv]
skip_install = true
passenv =
  dev: AUTHORITY
  dev: BOUNCER_URL
  dev: CLIENT_OAUTH_ID
  …
  {tests,functests}: TEST_DATABASE_URL
  {tests,functests}: ELASTICSEARCH_URL
  …
  functests: BROKER_URL
  codecov: CI TRAVIS*
deps =
  tests: coverage
  {tests,functests}: pytest
  {tests,functests}: factory-boy
  tests: mock
  tests: hypothesis
  lint: flake8
  lint: flake8-future-import
  coverage: coverage
  codecov: codecov
  functests: webtest
  docs: sphinx-autobuild
  docs: sphinx
  docs: sphinx_rtd_theme
  {tests,functests}: -r requirements.txt
  dev: ipython
  dev: ipdb
  dev: -r requirements-dev.in
whitelist_externals =
  dev: sh
changedir =
  docs: docs
commands =
  dev: sh bin/hypothesis --dev init
  dev: {posargs:sh bin/hypothesis devserver}
  lint: flake8 h
  lint: flake8 tests
  tests: coverage run …
  functests: pytest -Werror {posargs:tests/functional/}
  docs: sphinx-autobuild …
  coverage: -coverage combine
  coverage: coverage report
  codecov: codecov

How it Works

The tox.ini file is written entirely using tox’s factors and conditionals, and doesn’t use any of the separate [testenv:NAME] file sections that you see in most tox.ini files. When you pass an envlist to tox like tox -e py36-tests you’re telling tox to run a single testenv, named py36-tests, but the testenv name contains two factors py36 and tox.

Builtin factors: The py36 factor is a tox builtin factor that sets the Python version to 3.6. tox comes with builtin factors for all the Python versions: py27, py37, etc. You’ll notice that we don’t define the pyXY‘s anywhere in our tox.ini file. They’re builtin, so we can just go ahead and use them with -e on the command line.

Custom factors: The tests part in -e py36-tests is a custom factor, defined by us in our tox.ini file. Some of the lines in our tox.ini use tests: as a conditional, meaning those lines should only be applied if tests is in the testenv name. For example the coverage and mock dependencies will only be installed if tests is involved (for example when running tox -e py27-tests or tox -e py36-tests), whereas flake8 will only be installed if lint is in the tox -e command:

deps =
tests: coverage
tests: mock
lint: flake8
…

By using tests: and lint: as conditionals in the tox.ini file we implicitly define tests and lint factors in addition to the builtin pyXY factors, and can use them in commands like tox -e py36-lint.

What tox does when you give it a command like tox -e py36-tests is:

First, it invokes the builtin py36 factor, which sets the version of Python to be used to 3.6.
Second, it parses the tox.ini file looking for lines that match any factor in the given testenv name py36-tests:
1. Any line that doesn’t begin with a factor: prefix is unconditional, and is always applied whenever tox runs.
2. Any line that begins with tests: matches the tests conditional, so it’s applied. Any tests: dependencies will be installed, any tests: commands will be run, and so on.
3. A line like {tests,functests}: pytest matches either tests or functests, so the pytest dependency will be installed for tox -e py36-tests too. More complex conditionals are possible too.
After collecting all the matching lines tox runs the testenv:
1. Any matched environment variables are passed through to the test environment
2. Matched dependencies are installed in the order that they were given in the tox.ini file
3. Matched commands are run in file order

Here’s a view of the tox.ini file with the lines that apply when tox -e py36-tests is run highlighted. The non-highlighted lines are ignored:

[tox]
envlist = py27-tests
skipsdist = true
requires = tox-pip-extensions
tox_pip_extensions_ext_venv_update = true

[testenv]
skip_install = true
passenv =
dev: AUTHORITY
dev: BOUNCER_URL
dev: CLIENT_OAUTH_ID
…
{tests,functests}: TEST_DATABASE_URL
{tests,functests}: ELASTICSEARCH_URL
…
functests: BROKER_URL
codecov: CI TRAVIS*
deps =
tests: coverage
{tests,functests}: pytest
{tests,functests}: factory-boy
tests: mock
tests: hypothesis
lint: flake8
lint: flake8-future-import
coverage: coverage
codecov: codecov
functests: webtest
docs: sphinx-autobuild
docs: sphinx
docs: sphinx_rtd_theme
{tests,functests}: -r requirements.txt
dev: ipython
dev: ipdb
dev: -r requirements-dev.in
whitelist_externals =
dev: sh
changedir =
docs: docs
commands =
dev: sh bin/hypothesis --dev init
dev: {posargs:sh bin/hypothesis devserver}
lint: flake8 h
lint: flake8 tests
tests: coverage run …
functests: pytest -Werror {posargs:tests/functional/}
docs: sphinx-autobuild …
coverage: -coverage combine
coverage: coverage report
codecov: codecov

Benefits

Automating our development tasks like this has a bunch of advantages:

It simplifies things for developers, especially new developers. You don’t need to learn about virtualenv and create and activate a development virtualenv and install dependencies. You don’t need virtualenv management tools like virtualenvwrapper. You just run tox.
Tasks for running the tests, linting, releasing, etc can be defined in one place in tox.ini and run exactly the same everywhere: on any developer’s machine or on CI.
tox isolates every task using a Python virtualenv, PATH isolation and environment variable isolation, so problems caused by differences between environments are minimised.
Because tox uses a different virtualenv for each task (it creates one virtualenv for the dev server, another for the tests, another for building the docs, and so on) the chances of conflicts between dependencies are reduced. You don’t need your documentation tools installed in your devserver environment.
When Python 3.8 comes out and it’s time to test your app in it, you can easily run the dev server with tox -e py38-dev or the tests with tox -e py38-tests, without having to modify tox.ini first.

tox isn’t just for tests! Consider using it to standardise all of your project’s development tasks. For details of all the tox concepts used here and more see my tox tutorial.