Openstack projects use tox to manage virtual environments and run unit tests which I talked about here.
In this example I am using the oslo.config repo to look at the various tox configs in openstack use. The Governance Project Testing Interface is a starting point to read about project guidelines.
Get the current codebase
$ git clone git://git.openstack.org/openstack/oslo.config $ cd oslo.config/ $ git rev-parse HEAD 7b1e157aeea426c58e3d4c9a76a231f0e5bc8241
The last line helps me identify the specific git commit I am working with. When moving between branches or when looking at a repo that may be a few days old, if I need to recreate this exact codebase all I need is this. For example, to look at a prior version at 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d7 for example.
$ git checkout 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72 Note: checking out '3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72'. ... HEAD is now at 3ab4039... Merge "Added Raw Value Loading to Test Fixture" $ git rev-parse HEAD 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72
To revert back to the current repo master.
$ git checkout master Previous HEAD position was 3ab4039... Merge "Added Raw Value Loading to Test Fixture" Switched to branch 'master' Your branch is up-to-date with 'origin/master'. $ git rev-parse HEAD 7b1e157aeea426c58e3d4c9a76a231f0e5bc8241
NOTE: You don’t need to specify the full commit hash. In this example 3ab4039 also works.
tox configuration
The tox.ini file contains various config sections.
- [tox] are global options
- [testenv] are the default options for each virtual environment
- [testenv:NAME] are the specific test environments
tox.ini
$ cat tox.ini
[tox]
distribute = False
envlist = py33,py34,py26,py27,pep8
[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'
[testenv:pep8]
commands = flake8
[testenv:cover]
setenv = VIRTUAL_ENV={envdir}
commands =
python setup.py testr --coverage
[testenv:venv]
commands = {posargs}
[testenv:docs]
commands = python setup.py build_sphinx
[flake8]
show-source = True
exclude = .tox,dist,doc,*.egg,build
NOTE: These file differ between projects. See later for an example comparison with python-openstackclient, nova and horizon.
Prerequisites
The default [testenv] options first refer to requirements.txt and test-requirements.txt which define the specific packages and required versions. Either minimum (e.g. netaddr>=0.7.12), range (e.g. stevedore>=1.3.0,<1.4.0) or more specific (e.g. pbr>=0.6,!=0.7,<1.0).
requirements.txt
$ cat requirements.txt # The order of packages is significant, because pip processes them in the order # of appearance. Changing the order has an impact on the overall integration # process, which may cause wedges in the gate later. pbr>=0.6,!=0.7,<1.0 argparse netaddr>=0.7.12 six>=1.9.0 stevedore>=1.3.0,<1.4.0 # Apache-2.0
test-requirements.txt
$ cat test-requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
hacking>=0.10.0,<0.11
discover
fixtures>=0.3.14
python-subunit>=0.0.18
testrepository>=0.0.18
testscenarios>=0.4
testtools>=0.9.36,!=1.2.0
oslotest>=1.5.1,<1.6.0 # Apache-2.0
# when we can require tox>= 1.4, this can go into tox.ini:
# [testenv:cover]
# deps = {[testenv]deps} coverage
coverage>=3.6
# this is required for the docs build jobs
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
oslosphinx>=2.5.0,<2.6.0 # Apache-2.0
# Required only for tests
oslo.i18n>=1.5.0,<1.6.0 # Apache-2.0
# mocking framework
mock>=1.0
Style Guidelines (PEP8)
The first test we look at is pep8 run by flake8. This starts by reviewing the code with Style Guide for Python Code and any specific Openstack Style Guidelines.
$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='3973315668'
pep8 runtests: commands[0] | flake8
_________ summary ___________
pep8: commands succeeded
congratulations :)
As with all unit tests you are wanting to see "The bar is green, the code is clean". An example of a failing test would look like:
$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='820640265'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:51:31: E702 multiple statements on one line (semicolon)
self.choices = choices; self.quotes = quotes
^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
________ summary _________
ERROR: pep8: commands failed
$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='1937373059'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:52:13: E113 unexpected indentation
self.quotes = quotes
^
./oslo_config/types.py:52:13: E901 IndentationError: unexpected indent
self.quotes = quotes
^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
__________ summary __________
ERROR: pep8: commands failed
Running tests
To run all tests for a given Python version you just specify said version.
$ tox -e py27
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1822382852'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpbHjMgm
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpLA0oO0
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpMqT_s_
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpyJLbu8
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpF5KG5t
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpebkBDp
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpscXbNV
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --load-list /tmp/tmpTv0jAn
Ran 1182 tests in 0.475s (-0.068s)
PASSED (id=4)
Slowest Tests
Test id Runtime (s)
----------------------------------------------------------------------------------------------------- -----------
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_value_no_colon 0.029
oslo_config.tests.test_cfg.ConfigFileReloadTestCase.test_conf_files_reload_default 0.024
oslo_config.tests.test_cfg.SubCommandTestCase.test_sub_command_resparse 0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_ignore_dname 0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_spaces_use_dgroup_and_dname 0.016
tests.test_cfg.MultipleDeprecatedCliOptionsTestCase.test_conf_file_override_use_deprecated_multi_opts 0.015
oslo_config.tests.test_cfg.OverridesTestCase.test_default_override 0.014
oslo_config.tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_default_wrong_type 0.014
oslo_config.tests.test_cfg.RequiredOptsTestCase.test_missing_required_group_opt 0.012
tests.test_generator.GeneratorTestCase.test_generate(long_help,output_file) 0.011
________ summary _______
py27: commands succeeded
congratulations :)
You can pass a specific test or tests via command line identifying the names by looking at the test classes.
$ ls -l oslo_config/tests/[^_]*.py -rw-rw-r-- 1 rbradfor rbradfor 12788 Apr 30 12:46 oslo_config/tests/test_cfgfilter.py -rw-rw-r-- 1 rbradfor rbradfor 144538 Apr 30 12:46 oslo_config/tests/test_cfg.py -rw-rw-r-- 1 rbradfor rbradfor 4938 Apr 30 12:46 oslo_config/tests/test_fixture.py -rw-rw-r-- 1 rbradfor rbradfor 16479 Apr 30 12:46 oslo_config/tests/test_generator.py -rw-rw-r-- 1 rbradfor rbradfor 3865 Apr 30 12:46 oslo_config/tests/test_iniparser.py -rw-rw-r-- 1 rbradfor rbradfor 13259 Apr 30 12:46 oslo_config/tests/test_types.py
NOTE: This project has a top level /tests directory which uses the old import API and I am informed is being removed for liberty.
$ tox -e py27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1505218584'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
Ran 186 (-996) tests in 0.100s (-0.334s)
PASSED (id=6)
Slowest Tests
Test id Runtime (s)
------------------------------------------------------------------------------------------ -----------
tests.test_types.BooleanTypeTests.test_other_values_produce_error 0.001
oslo_config.tests.test_types.DictTypeTests.test_equal 0.001
oslo_config.tests.test_types.BooleanTypeTests.test_not_equal_to_other_class 0.000
tests.test_types.IntegerTypeTests.test_positive_values_are_valid 0.000
tests.test_types.DictTypeTests.test_dict_of_dicts 0.000
oslo_config.tests.test_types.ListTypeTests.test_not_equal_with_non_equal_custom_item_types 0.000
tests.test_types.IntegerTypeTests.test_with_max_and_min 0.000
oslo_config.tests.test_types.FloatTypeTests.test_exponential_format 0.000
tests.test_types.BooleanTypeTests.test_yes 0.000
tests.test_types.ListTypeTests.test_repr 0.000
________ summary ________
py27: commands succeeded
congratulations :)
$ echo $?
0
$ tox -epy27 -- '(test_types|test_generator)'
A failing test is going to produce the following.
$ tox -epy27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='3672144590'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
self.assertConvertedValue('192.168.0.1', '192.168.0.2')
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
self.assertEqual(expected, self.type_instance(s))
File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 186 (-117) tests in 0.102s (-0.046s)
FAILED (id=8, failures=2 (+2))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types'
________ summary ________
ERROR: py27: commands failed
$ echo $?
1
Testr
This is a wrapper for the underlying testr command (found in the command line of the [testenv] section). We can reproduce what this runs manually with.
$ source .tox/py27/bin/activate (py27)$ python setup.py testr running testr ... Ran 1182 tests in 0.443s (-0.025s) PASSED (id=5)
The current tox.ini config includes the --slowest argument which is self explaining.
One benefit of running this specifically is when writing failing tests (i.e. the Test Driven Development (TDD) approach to agile software development). You do not really want to run all tests in order to see a failure. The -f option helps.
$ testr run ... Ran 1182 (+637) tests in 2.058s (+1.064s) FAILED (id=12, failures=2 (+1))
$ testr run -- -f ... Ran 545 (-637) tests in 1.075s (-0.900s) FAILED (id=13, failures=1 (-1))
$ testr run test_types -- -f ... Ran 34 (-152) tests in 0.030s (-0.000s) FAILED (id=18, failures=1 (-1))
NOTE: It takes a bit to realize the syntax of tox and testr and handling doubledash? -- placement. When you work it out you realize you can reproduce this with tox directly using:
$ tox -e py27 -- test_types -- -f
...
Ran 151 (+117) tests in 0.125s (+0.120s)
FAILED (id=19, failures=2 (+1))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types -- -f'
________ summary ________
ERROR: py27: commands failed
The reason for dropping into an activated virtual environment and running testr manually is because tox will destroy and recreate your virtual environment each time the command is executed, which is time consuming.
The Testr source can be found at testrepository, identified by (py27)$ more `which testr`.
Testr syntax
Testr has multiple options and commands you can read about via various help options:
$ testr help $ testr quickstart $ testr commands $ testr help run Usage: testr run [options] testfilters* doubledash? testargs* ...
While debugging several testr commands were useful.
List all tests
$ testr list-tests
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_bad
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_good
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_bad_choice_value
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_bad_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_empty_value
...
(py27)$ testr list-tests | wc -l
1183
1183 - 1 corresponds to the 1182 test run.
Last run
This enables you to review the last run tests (in a separate thread) and also get a correct error response code.
(py27)$ testr last Ran 1182 tests in 0.575s (+0.099s) PASSED (id=27) (py27)$ echo $? 0
(py27)$ testr last
======================================================================
FAIL: oslo_config.tests.test_types.IPAddressTypeTests.test_ipv4_address
tags: worker-6
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
self.assertConvertedValue('192.168.0.1', '192.168.0.2')
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
self.assertEqual(expected, self.type_instance(s))
File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-7
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
self.assertConvertedValue('192.168.0.1', '192.168.0.2')
File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
self.assertEqual(expected, self.type_instance(s))
File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 1182 tests in 0.445s (-0.130s)
FAILED (id=28, failures=2 (+2))
(py27)$ echo $?
1
Code Coverage
The tox.ini also provides a section for code coverage.
$ tox -e cover
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
cover inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
cover runtests: PYTHONHASHSEED='546795877'
cover runtests: commands[0] | python setup.py testr --coverage
running testr
...
Ran 1182 tests in 0.493s (-0.046s)
PASSED (id=26)
_________ summary _________
cover: commands succeeded
congratulations :)
Which is a wrapper for:
$ python setup.py testr --coverage ... Ran 1182 tests in 0.592s (+0.116s) PASSED (id=27)
These commands produces a /cover directory (which is not currently in .gitignore). The contents are HTML. I suspect there is likely an option for a more CLI readable format however for simplicity we publish these to an available running web server.
Apache Setup
In order to view what code coverage produces I configured Apache with a separate port and vhost in this devstack environment.
$ echo "ServerName "`hostname` | sudo tee /etc/apache2/conf-enabled/servername.conf $ echo "Listen 81DocumentRoot /var/www/html " | sudo tee /etc/apache2/sites-enabled/localhost.conf $ sudo apache2ctl gracefulOptions FollowSymLinks MultiViews AllowOverride All Order allow,deny allow from all LogLevel warn ErrorLog \${APACHE_LOG_DIR}/localhost.error.log CustomLog \${APACHE_LOG_DIR}/localhost.access.log combined
Then I simply copied the projects coverage output as a quick hack to view.
$ sudo cp -r cover/ /var/www/html/ $ sudo apt-get install lynx-cur $ lynx http://localhost:81/cover
Module statements missing excluded coverage
Total 12 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/__init__ 6 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/cfg 1 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/cfgfilter 1 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/fixture 1 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/generator 1 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/iniparser 1 0 0 100%
.tox/py27/lib/python2.7/site-packages/oslo/config/types 1 0 0 100%
coverage.py v3.7.1
Documentation
The last testenv setup in oslo.config is for documentation.
$ tox -e docs
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
docs create: /home/rbradfor/oslo.config/.tox/docs
docs installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
docs inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
docs runtests: PYTHONHASHSEED='4293391351'
docs runtests: commands[0] | python setup.py build_sphinx
running build_sphinx
creating /home/rbradfor/oslo.config/doc/build
creating /home/rbradfor/oslo.config/doc/build/doctrees
creating /home/rbradfor/oslo.config/doc/build/html
Running Sphinx v1.2.3
loading pickled environment... not yet created
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [html]: all source files
updating environment: 15 added, 0 changed, 0 removed
reading sources... [100%] types
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] types
writing additional files... genindex py-modindex search
copying static files... WARNING: html_static_path entry u'/home/rbradfor/oslo.config/doc/source/static' does not exist
done
copying extra files... done
dumping search index... done
dumping object inventory... done
build succeeded, 1 warning.
creating /home/rbradfor/oslo.config/doc/build/man
Running Sphinx v1.2.3
loading pickled environment... done
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... osloconfig.1 { cfg opts types configopts cfgfilter helpers fixture parser exceptions namespaces styleguide generator faq contributing }
build succeeded.
___________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________
docs: commands succeeded
congratulations :)
This creates a /doc directory (in .gitignore) which I copied to my previously configured web container to view in HTML.
$ sudo cp -r doc/ /var/www/html/ $ lynx http://localhost:81/doc/build/html
Other tox.ini configuration
As I navigate around other Openstack projects I have noticed some differences. These include:
Alternative global settings
[tox] minversion = 1.6 skipdist = True
More detailed [testenv]
[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'
[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = python setup.py testr --testr-args='{posargs}'
whitelist_externals = bash
Some fancy output coloring.
[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
NOSE_WITH_OPENSTACK=1
NOSE_OPENSTACK_COLOR=1
NOSE_OPENSTACK_RED=0.05
NOSE_OPENSTACK_YELLOW=0.025
NOSE_OPENSTACK_SHOW_ELAPSED=1
# Note the hash seed is set to 0 until horizon can be tested with a
# random hash seed successfully.
PYTHONHASHSEED=0
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = /bin/bash run_tests.sh -N --no-pep8 {posargs}
[testenv]
usedevelop = True
# tox is silly... these need to be separated by a newline....
whitelist_externals = bash
find
install_command = pip install -U --force-reinstall {opts} {packages}
# Note the hash seed is set to 0 until nova can be tested with a
# random hash seed successfully.
setenv = VIRTUAL_ENV={envdir}
OS_TEST_PATH=./nova/tests/unit
LANGUAGE=en_US
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
find . -type f -name "*.pyc" -delete
bash tools/pretty_tox.sh '{posargs}'
# there is also secret magic in pretty_tox.sh which lets you run in a fail only
# mode. To do this define the TRACE_FAILONLY environmental variable.
Alternative [testenv:NAME] sections
[testenv:functional]
commands = bash -x {toxinidir}/functional/harpoon.sh
[testenv:debug]
commands = oslo_debug_helper -t openstackclient/tests {posargs}
[tox:jenkins]
downloadcache = ~/cache/pip
[testenv:jshint]
commands = nodeenv -p
npm install jshint -g
/bin/bash run_tests.sh -N --jshint
[testenv:genconfig]
commands =
bash tools/config/generate_sample.sh -b . -p nova -o etc/nova
Different Style guidelines
[flake8] show-source = True exclude = .tox,dist,doc,*.egg,build
[flake8] show-source = True exclude = .venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build,tools
[testenv:pep8] commands = flake8
[testenv:pep8] commands = /bin/bash run_tests.sh -N --pep8 /bin/bash run_tests.sh -N --makemessages --check-only
Different Code Coverage
[testenv:cover]
commands = python setup.py testr --coverage --testr-args='{posargs}'
[testenv:cover]
# Also do not run test_coverage_ext tests while gathering coverage as those
# tests conflict with coverage.
commands =
coverage erase
python setup.py testr --coverage \
--testr-args='{posargs}'
coverage combine
coverage html --include='nova/*' --omit='nova/openstack/common/*' -d covhtml -i
Different Docs
[testenv:docs] commands = python setup.py build_sphinx
[testenv:docs] commands = python setup.py build_sphinx bash -c '! find doc/ -type f -name *.json | xargs -t -n1 python -m json.tool 2>&1 > /dev/null | grep -B1 -v ^python'
Additional sections
[testenv:pip-missing-reqs]
# do not install test-requirements as that will pollute the virtualenv for
# determining missing packages
# this also means that pip-missing-reqs must be installed separately, outside
# of the requirements.txt files
deps = pip_missing_reqs
-rrequirements.txt
commands=pip-missing-reqs -d --ignore-file=nova/tests/* nova
[hacking] import_exceptions = oslo_log._i18n
What's Next
In a followup blog I will be talking about debugging with pdb and how to use this with tox.