Openstack projects use tox to manage virtual environments and run unit tests which I talked about here.

In this example I am using the oslo.config repo to look at the various tox configs in openstack use. The Governance Project Testing Interface is a starting point to read about project guidelines.

Get the current codebase

$ git clone git://git.openstack.org/openstack/oslo.config
$ cd oslo.config/
$ git rev-parse HEAD
7b1e157aeea426c58e3d4c9a76a231f0e5bc8241

The last line helps me identify the specific git commit I am working with. When moving between branches or when looking at a repo that may be a few days old, if I need to recreate this exact codebase all I need is this. For example, to look at a prior version at 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d7 for example.

$ git checkout 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72
Note: checking out '3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72'.
...
HEAD is now at 3ab4039... Merge "Added Raw Value Loading to Test Fixture"
$ git rev-parse HEAD
3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72

To revert back to the current repo master.

$ git checkout master
Previous HEAD position was 3ab4039... Merge "Added Raw Value Loading to Test Fixture"
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ git rev-parse HEAD
7b1e157aeea426c58e3d4c9a76a231f0e5bc8241

NOTE: You don’t need to specify the full commit hash. In this example 3ab4039 also works.

tox configuration

The tox.ini file contains various config sections.

• [tox] are global options
• [testenv] are the default options for each virtual environment
• [testenv:NAME] are the specific test environments

tox.ini

$ cat tox.ini
[tox]
distribute = False
envlist = py33,py34,py26,py27,pep8

[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'

[testenv:pep8]
commands = flake8

[testenv:cover]
setenv = VIRTUAL_ENV={envdir}
commands =
  python setup.py testr --coverage

[testenv:venv]
commands = {posargs}

[testenv:docs]
commands = python setup.py build_sphinx

[flake8]
show-source = True
exclude = .tox,dist,doc,*.egg,build

NOTE: These file differ between projects. See later for an example comparison with python-openstackclient, nova and horizon.

Prerequisites

The default [testenv] options first refer to requirements.txt and test-requirements.txt which define the specific packages and required versions. Either minimum (e.g. netaddr>=0.7.12), range (e.g. stevedore>=1.3.0,<1.4.0) or more specific (e.g. pbr>=0.6,!=0.7,<1.0).

requirements.txt

$ cat requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.

pbr>=0.6,!=0.7,<1.0
argparse
netaddr>=0.7.12
six>=1.9.0
stevedore>=1.3.0,<1.4.0  # Apache-2.0

test-requirements.txt

$ cat test-requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.

hacking>=0.10.0,<0.11

discover
fixtures>=0.3.14
python-subunit>=0.0.18
testrepository>=0.0.18
testscenarios>=0.4
testtools>=0.9.36,!=1.2.0
oslotest>=1.5.1,<1.6.0  # Apache-2.0

# when we can require tox>= 1.4, this can go into tox.ini:
#  [testenv:cover]
#  deps = {[testenv]deps} coverage
coverage>=3.6

# this is required for the docs build jobs
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
oslosphinx>=2.5.0,<2.6.0  # Apache-2.0

# Required only for tests
oslo.i18n>=1.5.0,<1.6.0  # Apache-2.0

# mocking framework
mock>=1.0

Style Guidelines (PEP8)

The first test we look at is pep8 run by flake8. This starts by reviewing the code with Style Guide for Python Code and any specific Openstack Style Guidelines.

$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='3973315668'
pep8 runtests: commands[0] | flake8
_________ summary ___________
  pep8: commands succeeded
  congratulations :)

As with all unit tests you are wanting to see "The bar is green, the code is clean". An example of a failing test would look like:

$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='820640265'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:51:31: E702 multiple statements on one line (semicolon)
        self.choices = choices; self.quotes = quotes
                              ^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
________ summary _________
ERROR:   pep8: commands failed


$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='1937373059'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:52:13: E113 unexpected indentation
            self.quotes = quotes
            ^
./oslo_config/types.py:52:13: E901 IndentationError: unexpected indent
            self.quotes = quotes
            ^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
__________ summary __________
ERROR:   pep8: commands failed

Running tests

To run all tests for a given Python version you just specify said version.

$ tox -e py27
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1822382852'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpbHjMgm
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpLA0oO0
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpMqT_s_
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpyJLbu8
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpF5KG5t
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpebkBDp
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpscXbNV
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpTv0jAn
Ran 1182 tests in 0.475s (-0.068s)
PASSED (id=4)
Slowest Tests
Test id                                                                                                Runtime (s)
-----------------------------------------------------------------------------------------------------  -----------
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_value_no_colon                               0.029
oslo_config.tests.test_cfg.ConfigFileReloadTestCase.test_conf_files_reload_default                     0.024
oslo_config.tests.test_cfg.SubCommandTestCase.test_sub_command_resparse                                0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_ignore_dname                                 0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_spaces_use_dgroup_and_dname                  0.016
tests.test_cfg.MultipleDeprecatedCliOptionsTestCase.test_conf_file_override_use_deprecated_multi_opts  0.015
oslo_config.tests.test_cfg.OverridesTestCase.test_default_override                                     0.014
oslo_config.tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_default_wrong_type               0.014
oslo_config.tests.test_cfg.RequiredOptsTestCase.test_missing_required_group_opt                        0.012
tests.test_generator.GeneratorTestCase.test_generate(long_help,output_file)                            0.011
________ summary _______
  py27: commands succeeded
  congratulations :)

You can pass a specific test or tests via command line identifying the names by looking at the test classes.

$ ls -l oslo_config/tests/[^_]*.py
-rw-rw-r-- 1 rbradfor rbradfor  12788 Apr 30 12:46 oslo_config/tests/test_cfgfilter.py
-rw-rw-r-- 1 rbradfor rbradfor 144538 Apr 30 12:46 oslo_config/tests/test_cfg.py
-rw-rw-r-- 1 rbradfor rbradfor   4938 Apr 30 12:46 oslo_config/tests/test_fixture.py
-rw-rw-r-- 1 rbradfor rbradfor  16479 Apr 30 12:46 oslo_config/tests/test_generator.py
-rw-rw-r-- 1 rbradfor rbradfor   3865 Apr 30 12:46 oslo_config/tests/test_iniparser.py
-rw-rw-r-- 1 rbradfor rbradfor  13259 Apr 30 12:46 oslo_config/tests/test_types.py

NOTE: This project has a top level /tests directory which uses the old import API and I am informed is being removed for liberty.

$ tox -e py27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1505218584'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
Ran 186 (-996) tests in 0.100s (-0.334s)
PASSED (id=6)
Slowest Tests
Test id                                                                                     Runtime (s)
------------------------------------------------------------------------------------------  -----------
tests.test_types.BooleanTypeTests.test_other_values_produce_error                           0.001
oslo_config.tests.test_types.DictTypeTests.test_equal                                       0.001
oslo_config.tests.test_types.BooleanTypeTests.test_not_equal_to_other_class                 0.000
tests.test_types.IntegerTypeTests.test_positive_values_are_valid                            0.000
tests.test_types.DictTypeTests.test_dict_of_dicts                                           0.000
oslo_config.tests.test_types.ListTypeTests.test_not_equal_with_non_equal_custom_item_types  0.000
tests.test_types.IntegerTypeTests.test_with_max_and_min                                     0.000
oslo_config.tests.test_types.FloatTypeTests.test_exponential_format                         0.000
tests.test_types.BooleanTypeTests.test_yes                                                  0.000
tests.test_types.ListTypeTests.test_repr                                                    0.000
________ summary ________
  py27: commands succeeded
  congratulations :)
$ echo $?
0

$ tox -epy27 -- '(test_types|test_generator)'

A failing test is going to produce the following.

$ tox -epy27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='3672144590'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 186 (-117) tests in 0.102s (-0.046s)
FAILED (id=8, failures=2 (+2))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types'
________ summary ________
ERROR:   py27: commands failed
$ echo $?
1

Testr

This is a wrapper for the underlying testr command (found in the command line of the [testenv] section). We can reproduce what this runs manually with.

$ source .tox/py27/bin/activate
(py27)$  python setup.py testr
running testr
...
Ran 1182 tests in 0.443s (-0.025s)
PASSED (id=5)

The current tox.ini config includes the --slowest argument which is self explaining.

One benefit of running this specifically is when writing failing tests (i.e. the Test Driven Development (TDD) approach to agile software development). You do not really want to run all tests in order to see a failure. The -f option helps.

$ testr run
...
Ran 1182 (+637) tests in 2.058s (+1.064s)
FAILED (id=12, failures=2 (+1))

$ testr run -- -f
...
Ran 545 (-637) tests in 1.075s (-0.900s)
FAILED (id=13, failures=1 (-1))

$ testr run test_types -- -f
...
Ran 34 (-152) tests in 0.030s (-0.000s)
FAILED (id=18, failures=1 (-1))

NOTE: It takes a bit to realize the syntax of tox and testr and handling doubledash? -- placement. When you work it out you realize you can reproduce this with tox directly using:

$ tox -e py27 -- test_types -- -f
...
Ran 151 (+117) tests in 0.125s (+0.120s)
FAILED (id=19, failures=2 (+1))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types -- -f'
________ summary ________
ERROR:   py27: commands failed

The reason for dropping into an activated virtual environment and running testr manually is because tox will destroy and recreate your virtual environment each time the command is executed, which is time consuming.

The Testr source can be found at testrepository, identified by (py27)$ more `which testr`.

Testr syntax

Testr has multiple options and commands you can read about via various help options:

$ testr help
$ testr quickstart
$ testr commands
$ testr help run

Usage: testr run [options] testfilters* doubledash? testargs*
...

While debugging several testr commands were useful.

List all tests

$ testr list-tests
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_bad
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_good
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_bad_choice_value
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_bad_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_empty_value
...

(py27)$ testr list-tests | wc -l
1183

1183 - 1 corresponds to the 1182 test run.

Last run

This enables you to review the last run tests (in a separate thread) and also get a correct error response code.

(py27)$ testr last
Ran 1182 tests in 0.575s (+0.099s)
PASSED (id=27)
(py27)$ echo $?
0

(py27)$ testr last
======================================================================
FAIL: oslo_config.tests.test_types.IPAddressTypeTests.test_ipv4_address
tags: worker-6
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-7
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 1182 tests in 0.445s (-0.130s)
FAILED (id=28, failures=2 (+2))
(py27)$ echo $?
1

Code Coverage

The tox.ini also provides a section for code coverage.

$ tox -e cover
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
cover inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
cover runtests: PYTHONHASHSEED='546795877'
cover runtests: commands[0] | python setup.py testr --coverage
running testr
...
Ran 1182 tests in 0.493s (-0.046s)
PASSED (id=26)
_________ summary _________
  cover: commands succeeded
  congratulations :)

Which is a wrapper for:

$ python setup.py testr --coverage
...
Ran 1182 tests in 0.592s (+0.116s)
PASSED (id=27)

These commands produces a /cover directory (which is not currently in .gitignore). The contents are HTML. I suspect there is likely an option for a more CLI readable format however for simplicity we publish these to an available running web server.

Apache Setup

In order to view what code coverage produces I configured Apache with a separate port and vhost in this devstack environment.

$ echo "ServerName "`hostname` | sudo tee /etc/apache2/conf-enabled/servername.conf
$ echo "Listen 81


    DocumentRoot /var/www/html
    
        Options FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    

    LogLevel warn
    ErrorLog \${APACHE_LOG_DIR}/localhost.error.log
    CustomLog \${APACHE_LOG_DIR}/localhost.access.log combined
" | sudo tee /etc/apache2/sites-enabled/localhost.conf
$ sudo apache2ctl graceful

Then I simply copied the projects coverage output as a quick hack to view.

$ sudo cp -r cover/ /var/www/html/
$ sudo apt-get install lynx-cur
$ lynx http://localhost:81/cover

                             Module                            statements missing excluded coverage
   Total                                                       12         0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/__init__  6          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/cfg       1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/cfgfilter 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/fixture   1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/generator 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/iniparser 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/types     1          0       0        100%

   coverage.py v3.7.1

Documentation

The last testenv setup in oslo.config is for documentation.

$ tox -e docs
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
docs create: /home/rbradfor/oslo.config/.tox/docs
docs installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
docs inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
docs runtests: PYTHONHASHSEED='4293391351'
docs runtests: commands[0] | python setup.py build_sphinx
running build_sphinx
creating /home/rbradfor/oslo.config/doc/build
creating /home/rbradfor/oslo.config/doc/build/doctrees
creating /home/rbradfor/oslo.config/doc/build/html
Running Sphinx v1.2.3
loading pickled environment... not yet created
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [html]: all source files
updating environment: 15 added, 0 changed, 0 removed
reading sources... [100%] types
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] types
writing additional files... genindex py-modindex search
copying static files... WARNING: html_static_path entry u'/home/rbradfor/oslo.config/doc/source/static' does not exist
done
copying extra files... done
dumping search index... done
dumping object inventory... done
build succeeded, 1 warning.
creating /home/rbradfor/oslo.config/doc/build/man
Running Sphinx v1.2.3
loading pickled environment... done
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... osloconfig.1 { cfg opts types configopts cfgfilter helpers fixture parser exceptions namespaces styleguide generator faq contributing }
build succeeded.
___________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________
  docs: commands succeeded
  congratulations :)

This creates a /doc directory (in .gitignore) which I copied to my previously configured web container to view in HTML.

$ sudo cp -r doc/ /var/www/html/
$ lynx http://localhost:81/doc/build/html

Other tox.ini configuration

As I navigate around other Openstack projects I have noticed some differences. These include:

Alternative global settings

[tox]
minversion = 1.6
skipdist = True

More detailed [testenv]

[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'

[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --testr-args='{posargs}'
whitelist_externals = bash

Some fancy output coloring.

[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
         NOSE_WITH_OPENSTACK=1
         NOSE_OPENSTACK_COLOR=1
         NOSE_OPENSTACK_RED=0.05
         NOSE_OPENSTACK_YELLOW=0.025
         NOSE_OPENSTACK_SHOW_ELAPSED=1
# Note the hash seed is set to 0 until horizon can be tested with a
# random hash seed successfully.
         PYTHONHASHSEED=0
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = /bin/bash run_tests.sh -N --no-pep8 {posargs}

[testenv]
usedevelop = True
# tox is silly... these need to be separated by a newline....
whitelist_externals = bash
                      find
install_command = pip install -U --force-reinstall {opts} {packages}
# Note the hash seed is set to 0 until nova can be tested with a
# random hash seed successfully.
setenv = VIRTUAL_ENV={envdir}
         OS_TEST_PATH=./nova/tests/unit
         LANGUAGE=en_US
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands =
  find . -type f -name "*.pyc" -delete
  bash tools/pretty_tox.sh '{posargs}'
# there is also secret magic in pretty_tox.sh which lets you run in a fail only
# mode. To do this define the TRACE_FAILONLY environmental variable.

Alternative [testenv:NAME] sections

[testenv:functional]
commands = bash -x {toxinidir}/functional/harpoon.sh

[testenv:debug]
commands = oslo_debug_helper -t openstackclient/tests {posargs}

[tox:jenkins]
downloadcache = ~/cache/pip

[testenv:jshint]
commands = nodeenv -p
           npm install jshint -g
           /bin/bash run_tests.sh -N --jshint

[testenv:genconfig]
commands =
  bash tools/config/generate_sample.sh -b . -p nova -o etc/nova

Different Style guidelines

[flake8]
show-source = True
exclude = .tox,dist,doc,*.egg,build

[flake8]
show-source = True
exclude =  .venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build,tools

[testenv:pep8]
commands = flake8

[testenv:pep8]
commands =
  /bin/bash run_tests.sh -N --pep8
  /bin/bash run_tests.sh -N --makemessages --check-only

Different Code Coverage

[testenv:cover]
commands = python setup.py testr --coverage --testr-args='{posargs}'

[testenv:cover]
# Also do not run test_coverage_ext tests while gathering coverage as those
# tests conflict with coverage.
commands =
  coverage erase
  python setup.py testr --coverage \
    --testr-args='{posargs}'
  coverage combine
  coverage html --include='nova/*' --omit='nova/openstack/common/*' -d covhtml -i

Different Docs

[testenv:docs]
commands = python setup.py build_sphinx

[testenv:docs]
commands =
  python setup.py build_sphinx
  bash -c '! find doc/ -type f -name *.json | xargs -t -n1 python -m json.tool 2>&1 > /dev/null | grep -B1 -v ^python'

Additional sections

[testenv:pip-missing-reqs]
# do not install test-requirements as that will pollute the virtualenv for
# determining missing packages
# this also means that pip-missing-reqs must be installed separately, outside
# of the requirements.txt files
deps = pip_missing_reqs
       -rrequirements.txt
commands=pip-missing-reqs -d --ignore-file=nova/tests/* nova

[hacking]
import_exceptions = oslo_log._i18n

What's Next

In a followup blog I will be talking about debugging with pdb and how to use this with tox.

References

• Running openstack tests with tox
• Installing Python 3.3 on Ubuntu 14.04.2 LTS
• OpenStack Testr docs
• tox reference documentation