Disabling the temporary authorization token in devstack keystone

While building my own OpenStack cloud on physical servers I realized that Keystone uses a temporary authorization token in the Create the service entity and API endpoint and Create projects, users, and roles steps.

The Verify operation step makes reference to removing this mechanism however my current devstack installations have not done this.

To verify this I use the SERVICE_TOKEN as defined in my devstack/local.conf and the Keystone Admin URL.

$ openstack --os-token=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee --os-url=http://controller:35357/v2.0 user list
+----------------------------------+----------------------------------+
| ID                               | Name                             |
+----------------------------------+----------------------------------+
| 554209509f5b47e286e0379bcbf66762 | admin                            |
| 59ac0457a80d449c9dac3b66848f2b5b | demo                             |
| 8aab962698f9460692efb8d3aab35886 | verify_tempest_config-1304647972 |
| 8b602467cd9045888687987067cbd3f6 | alt_demo                         |
| a134c3b33e94475fb5398643dd816053 | glance                           |
| c68c68579ec0437094a14dfbc4728224 | cinder                           |
| e65bd34ca85a429ea5c56bf980f77d67 | nova                             |
+----------------------------------+----------------------------------+

Removing the configuration settings as documented from /etc/keystone/keystone-paste.ini as documented DOES NOT disable this level of access.

NOTE: This edit removes the admin_token_auth option from the pipeline setting in the [pipeline:public_api], [pipeline:admin_api] and [pipeline:api_v3] sections.

$ sudo sed -ie "s/ admin_token_auth / /" /etc/keystone/keystone-paste.ini
$ openstack --os-token=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee --os-url=http://controller:35357/v2.0 user list
+----------------------------------+----------------------------------+
| ID                               | Name                             |
+----------------------------------+----------------------------------+
| 554209509f5b47e286e0379bcbf66762 | admin                            |
| 59ac0457a80d449c9dac3b66848f2b5b | demo                             |
| 8aab962698f9460692efb8d3aab35886 | verify_tempest_config-1304647972 |
| 8b602467cd9045888687987067cbd3f6 | alt_demo                         |
| a134c3b33e94475fb5398643dd816053 | glance                           |
| c68c68579ec0437094a14dfbc4728224 | cinder                           |
| e65bd34ca85a429ea5c56bf980f77d67 | nova                             |
+----------------------------------+----------------------------------+

An additional (and not presently documented step) of restarting apache is needed to invalidate this access.

$ sudo service apache2 restart
$ openstack --os-token=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee --os-url=http://controller:35357/v2.0 user list
ERROR: openstack Could not find token: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee (Disable debug mode to suppress these details.) (HTTP 401) (Request-ID: req-617961b7-012a-4d61-bdfb-aa738b8f788f)

The results for the command as shown can be produced by using the user/password credentials with the Keystone public URL.

$ openstack --os-username=admin --os-password=passwd --os-project-name=admin --os-auth-url=http://localhost:5000/ user list
+----------------------------------+----------------------------------+
| ID                               | Name                             |
+----------------------------------+----------------------------------+
| 554209509f5b47e286e0379bcbf66762 | admin                            |
| 59ac0457a80d449c9dac3b66848f2b5b | demo                             |
| 8aab962698f9460692efb8d3aab35886 | verify_tempest_config-1304647972 |
| 8b602467cd9045888687987067cbd3f6 | alt_demo                         |
| a134c3b33e94475fb5398643dd816053 | glance                           |
| c68c68579ec0437094a14dfbc4728224 | cinder                           |
| e65bd34ca85a429ea5c56bf980f77d67 | nova                             |
+----------------------------------+----------------------------------+

Understanding the different Openstack tox configs

Openstack projects use tox to manage virtual environments and run unit tests which I talked about here.

In this example I am using the oslo.config repo to look at the various tox configs in openstack use. The Governance Project Testing Interface is a starting point to read about project guidelines.

Get the current codebase

$ git clone git://git.openstack.org/openstack/oslo.config
$ cd oslo.config/
$ git rev-parse HEAD
7b1e157aeea426c58e3d4c9a76a231f0e5bc8241

The last line helps me identify the specific git commit I am working with. When moving between branches or when looking at a repo that may be a few days old, if I need to recreate this exact codebase all I need is this. For example, to look at a prior version at 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d7 for example.

$ git checkout 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72
Note: checking out '3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72'.
...
HEAD is now at 3ab4039... Merge "Added Raw Value Loading to Test Fixture"
$ git rev-parse HEAD
3ab403925e9cb2928ba8e893c4d0f4a6f4b27d72

To revert back to the current repo master.

$ git checkout master
Previous HEAD position was 3ab4039... Merge "Added Raw Value Loading to Test Fixture"
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ git rev-parse HEAD
7b1e157aeea426c58e3d4c9a76a231f0e5bc8241

NOTE: You don’t need to specify the full commit hash. In this example 3ab4039 also works.

tox configuration

The tox.ini file contains various config sections.

  • [tox] are global options
  • [testenv] are the default options for each virtual environment
  • [testenv:NAME] are the specific test environments

tox.ini

$ cat tox.ini
[tox]
distribute = False
envlist = py33,py34,py26,py27,pep8

[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'

[testenv:pep8]
commands = flake8

[testenv:cover]
setenv = VIRTUAL_ENV={envdir}
commands =
  python setup.py testr --coverage

[testenv:venv]
commands = {posargs}

[testenv:docs]
commands = python setup.py build_sphinx

[flake8]
show-source = True
exclude = .tox,dist,doc,*.egg,build

NOTE: These file differ between projects. See later for an example comparison with python-openstackclient, nova and horizon.

Prerequisites

The default [testenv] options first refer to requirements.txt and test-requirements.txt which define the specific packages and required versions. Either minimum (e.g. netaddr>=0.7.12), range (e.g. stevedore>=1.3.0,<1.4.0) or more specific (e.g. pbr>=0.6,!=0.7,<1.0).

requirements.txt

$ cat requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.

pbr>=0.6,!=0.7,<1.0
argparse
netaddr>=0.7.12
six>=1.9.0
stevedore>=1.3.0,<1.4.0  # Apache-2.0

test-requirements.txt

$ cat test-requirements.txt
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.

hacking>=0.10.0,<0.11

discover
fixtures>=0.3.14
python-subunit>=0.0.18
testrepository>=0.0.18
testscenarios>=0.4
testtools>=0.9.36,!=1.2.0
oslotest>=1.5.1,<1.6.0  # Apache-2.0

# when we can require tox>= 1.4, this can go into tox.ini:
#  [testenv:cover]
#  deps = {[testenv]deps} coverage
coverage>=3.6

# this is required for the docs build jobs
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
oslosphinx>=2.5.0,<2.6.0  # Apache-2.0

# Required only for tests
oslo.i18n>=1.5.0,<1.6.0  # Apache-2.0

# mocking framework
mock>=1.0

Style Guidelines (PEP8)

The first test we look at is pep8 run by flake8. This starts by reviewing the code with Style Guide for Python Code and any specific Openstack Style Guidelines.

$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='3973315668'
pep8 runtests: commands[0] | flake8
_________ summary ___________
  pep8: commands succeeded
  congratulations :)

As with all unit tests you are wanting to see "The bar is green, the code is clean". An example of a failing test would look like:

$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='820640265'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:51:31: E702 multiple statements on one line (semicolon)
        self.choices = choices; self.quotes = quotes
                              ^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
________ summary _________
ERROR:   pep8: commands failed


$ tox -e pep8
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
pep8 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
pep8 runtests: PYTHONHASHSEED='1937373059'
pep8 runtests: commands[0] | flake8
./oslo_config/types.py:52:13: E113 unexpected indentation
            self.quotes = quotes
            ^
./oslo_config/types.py:52:13: E901 IndentationError: unexpected indent
            self.quotes = quotes
            ^
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/pep8/bin/flake8'
__________ summary __________
ERROR:   pep8: commands failed

Running tests

To run all tests for a given Python version you just specify said version.

$ tox -e py27
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1822382852'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpbHjMgm
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpLA0oO0
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpMqT_s_
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpyJLbu8
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpF5KG5t
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpebkBDp
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpscXbNV
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ .  --load-list /tmp/tmpTv0jAn
Ran 1182 tests in 0.475s (-0.068s)
PASSED (id=4)
Slowest Tests
Test id                                                                                                Runtime (s)
-----------------------------------------------------------------------------------------------------  -----------
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_value_no_colon                               0.029
oslo_config.tests.test_cfg.ConfigFileReloadTestCase.test_conf_files_reload_default                     0.024
oslo_config.tests.test_cfg.SubCommandTestCase.test_sub_command_resparse                                0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_dict_ignore_dname                                 0.016
tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_spaces_use_dgroup_and_dname                  0.016
tests.test_cfg.MultipleDeprecatedCliOptionsTestCase.test_conf_file_override_use_deprecated_multi_opts  0.015
oslo_config.tests.test_cfg.OverridesTestCase.test_default_override                                     0.014
oslo_config.tests.test_cfg.ConfigFileOptsTestCase.test_conf_file_list_default_wrong_type               0.014
oslo_config.tests.test_cfg.RequiredOptsTestCase.test_missing_required_group_opt                        0.012
tests.test_generator.GeneratorTestCase.test_generate(long_help,output_file)                            0.011
________ summary _______
  py27: commands succeeded
  congratulations :)

You can pass a specific test or tests via command line identifying the names by looking at the test classes.

$ ls -l oslo_config/tests/[^_]*.py
-rw-rw-r-- 1 rbradfor rbradfor  12788 Apr 30 12:46 oslo_config/tests/test_cfgfilter.py
-rw-rw-r-- 1 rbradfor rbradfor 144538 Apr 30 12:46 oslo_config/tests/test_cfg.py
-rw-rw-r-- 1 rbradfor rbradfor   4938 Apr 30 12:46 oslo_config/tests/test_fixture.py
-rw-rw-r-- 1 rbradfor rbradfor  16479 Apr 30 12:46 oslo_config/tests/test_generator.py
-rw-rw-r-- 1 rbradfor rbradfor   3865 Apr 30 12:46 oslo_config/tests/test_iniparser.py
-rw-rw-r-- 1 rbradfor rbradfor  13259 Apr 30 12:46 oslo_config/tests/test_types.py

NOTE: This project has a top level /tests directory which uses the old import API and I am informed is being removed for liberty.

$ tox -e py27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='1505218584'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
Ran 186 (-996) tests in 0.100s (-0.334s)
PASSED (id=6)
Slowest Tests
Test id                                                                                     Runtime (s)
------------------------------------------------------------------------------------------  -----------
tests.test_types.BooleanTypeTests.test_other_values_produce_error                           0.001
oslo_config.tests.test_types.DictTypeTests.test_equal                                       0.001
oslo_config.tests.test_types.BooleanTypeTests.test_not_equal_to_other_class                 0.000
tests.test_types.IntegerTypeTests.test_positive_values_are_valid                            0.000
tests.test_types.DictTypeTests.test_dict_of_dicts                                           0.000
oslo_config.tests.test_types.ListTypeTests.test_not_equal_with_non_equal_custom_item_types  0.000
tests.test_types.IntegerTypeTests.test_with_max_and_min                                     0.000
oslo_config.tests.test_types.FloatTypeTests.test_exponential_format                         0.000
tests.test_types.BooleanTypeTests.test_yes                                                  0.000
tests.test_types.ListTypeTests.test_repr                                                    0.000
________ summary ________
  py27: commands succeeded
  congratulations :)
$ echo $?
0
$ tox -epy27 -- '(test_types|test_generator)'

A failing test is going to produce the following.

$ tox -epy27 -- test_types
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
py27 create: /home/rbradfor/oslo.config/.tox/py27
py27 installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
py27 inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
py27 runtests: PYTHONHASHSEED='3672144590'
py27 runtests: commands[0] | python setup.py testr --slowest --testr-args=test_types
running testr
...
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 186 (-117) tests in 0.102s (-0.046s)
FAILED (id=8, failures=2 (+2))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types'
________ summary ________
ERROR:   py27: commands failed
$ echo $?
1

Testr

This is a wrapper for the underlying testr command (found in the command line of the [testenv] section). We can reproduce what this runs manually with.

$ source .tox/py27/bin/activate
(py27)$  python setup.py testr
running testr
...
Ran 1182 tests in 0.443s (-0.025s)
PASSED (id=5)

The current tox.ini config includes the --slowest argument which is self explaining.

One benefit of running this specifically is when writing failing tests (i.e. the Test Driven Development (TDD) approach to agile software development). You do not really want to run all tests in order to see a failure. The -f option helps.

$ testr run
...
Ran 1182 (+637) tests in 2.058s (+1.064s)
FAILED (id=12, failures=2 (+1))
$ testr run -- -f
...
Ran 545 (-637) tests in 1.075s (-0.900s)
FAILED (id=13, failures=1 (-1))
$ testr run test_types -- -f
...
Ran 34 (-152) tests in 0.030s (-0.000s)
FAILED (id=18, failures=1 (-1))

NOTE: It takes a bit to realize the syntax of tox and testr and handling doubledash? -- placement. When you work it out you realize you can reproduce this with tox directly using:

$ tox -e py27 -- test_types -- -f
...
Ran 151 (+117) tests in 0.125s (+0.120s)
FAILED (id=19, failures=2 (+1))
error: testr failed (1)
ERROR: InvocationError: '/home/rbradfor/oslo.config/.tox/py27/bin/python setup.py testr --slowest --testr-args=test_types -- -f'
________ summary ________
ERROR:   py27: commands failed

The reason for dropping into an activated virtual environment and running testr manually is because tox will destroy and recreate your virtual environment each time the command is executed, which is time consuming.

The Testr source can be found at testrepository, identified by (py27)$ more `which testr`.

Testr syntax

Testr has multiple options and commands you can read about via various help options:

$ testr help
$ testr quickstart
$ testr commands
$ testr help run

Usage: testr run [options] testfilters* doubledash? testargs*
...

While debugging several testr commands were useful.

List all tests

$ testr list-tests
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ . --list
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_bad
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_choice_good
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_bad_choice_value
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_bad_default
oslo_config.tests.test_cfg.ChoicesTestCase.test_conf_file_choice_empty_value
...

(py27)$ testr list-tests | wc -l
1183

1183 - 1 corresponds to the 1182 test run.

Last run

This enables you to review the last run tests (in a separate thread) and also get a correct error response code.

(py27)$ testr last
Ran 1182 tests in 0.575s (+0.099s)
PASSED (id=27)
(py27)$ echo $?
0
(py27)$ testr last
======================================================================
FAIL: oslo_config.tests.test_types.IPAddressTypeTests.test_ipv4_address
tags: worker-6
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
======================================================================
FAIL: oslo_config.tests.test_types.IPv4AddressTypeTests.test_ipv4_address
tags: worker-7
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 386, in test_ipv4_address
    self.assertConvertedValue('192.168.0.1', '192.168.0.2')
  File "/home/rbradfor/oslo.config/oslo_config/tests/test_types.py", line 27, in assertConvertedValue
    self.assertEqual(expected, self.type_instance(s))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: '192.168.0.2' != '192.168.0.1'
Ran 1182 tests in 0.445s (-0.130s)
FAILED (id=28, failures=2 (+2))
(py27)$ echo $?
1

Code Coverage

The tox.ini also provides a section for code coverage.

$ tox -e cover
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
cover inst-nodeps: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
cover runtests: PYTHONHASHSEED='546795877'
cover runtests: commands[0] | python setup.py testr --coverage
running testr
...
Ran 1182 tests in 0.493s (-0.046s)
PASSED (id=26)
_________ summary _________
  cover: commands succeeded
  congratulations :)

Which is a wrapper for:

$ python setup.py testr --coverage
...
Ran 1182 tests in 0.592s (+0.116s)
PASSED (id=27)

These commands produces a /cover directory (which is not currently in .gitignore). The contents are HTML. I suspect there is likely an option for a more CLI readable format however for simplicity we publish these to an available running web server.

Apache Setup

In order to view what code coverage produces I configured Apache with a separate port and vhost in this devstack environment.

$ echo "ServerName "`hostname` | sudo tee /etc/apache2/conf-enabled/servername.conf
$ echo "Listen 81


    DocumentRoot /var/www/html
    
        Options FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    

    LogLevel warn
    ErrorLog \${APACHE_LOG_DIR}/localhost.error.log
    CustomLog \${APACHE_LOG_DIR}/localhost.access.log combined
" | sudo tee /etc/apache2/sites-enabled/localhost.conf
$ sudo apache2ctl graceful

Then I simply copied the projects coverage output as a quick hack to view.

$ sudo cp -r cover/ /var/www/html/
$ sudo apt-get install lynx-cur
$ lynx http://localhost:81/cover
                             Module                            statements missing excluded coverage
   Total                                                       12         0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/__init__  6          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/cfg       1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/cfgfilter 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/fixture   1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/generator 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/iniparser 1          0       0        100%
   .tox/py27/lib/python2.7/site-packages/oslo/config/types     1          0       0        100%

   coverage.py v3.7.1

Documentation

The last testenv setup in oslo.config is for documentation.

$ tox -e docs
GLOB sdist-make: /home/rbradfor/oslo.config/setup.py
docs create: /home/rbradfor/oslo.config/.tox/docs
docs installdeps: -r/home/rbradfor/oslo.config/requirements.txt, -r/home/rbradfor/oslo.config/test-requirements.txt
docs inst: /home/rbradfor/oslo.config/.tox/dist/oslo.config-1.10.0.zip
docs runtests: PYTHONHASHSEED='4293391351'
docs runtests: commands[0] | python setup.py build_sphinx
running build_sphinx
creating /home/rbradfor/oslo.config/doc/build
creating /home/rbradfor/oslo.config/doc/build/doctrees
creating /home/rbradfor/oslo.config/doc/build/html
Running Sphinx v1.2.3
loading pickled environment... not yet created
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [html]: all source files
updating environment: 15 added, 0 changed, 0 removed
reading sources... [100%] types
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] types
writing additional files... genindex py-modindex search
copying static files... WARNING: html_static_path entry u'/home/rbradfor/oslo.config/doc/source/static' does not exist
done
copying extra files... done
dumping search index... done
dumping object inventory... done
build succeeded, 1 warning.
creating /home/rbradfor/oslo.config/doc/build/man
Running Sphinx v1.2.3
loading pickled environment... done
Using openstack theme from /home/rbradfor/oslo.config/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [man]: all source files
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
writing... osloconfig.1 { cfg opts types configopts cfgfilter helpers fixture parser exceptions namespaces styleguide generator faq contributing }
build succeeded.
___________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________
  docs: commands succeeded
  congratulations :)

This creates a /doc directory (in .gitignore) which I copied to my previously configured web container to view in HTML.

$ sudo cp -r doc/ /var/www/html/
$ lynx http://localhost:81/doc/build/html

Other tox.ini configuration

As I navigate around other Openstack projects I have noticed some differences. These include:

Alternative global settings

[tox]
minversion = 1.6
skipdist = True

More detailed [testenv]

[testenv]
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --slowest --testr-args='{posargs}'
[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = python setup.py testr --testr-args='{posargs}'
whitelist_externals = bash

Some fancy output coloring.

[testenv]
usedevelop = True
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
         NOSE_WITH_OPENSTACK=1
         NOSE_OPENSTACK_COLOR=1
         NOSE_OPENSTACK_RED=0.05
         NOSE_OPENSTACK_YELLOW=0.025
         NOSE_OPENSTACK_SHOW_ELAPSED=1
# Note the hash seed is set to 0 until horizon can be tested with a
# random hash seed successfully.
         PYTHONHASHSEED=0
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands = /bin/bash run_tests.sh -N --no-pep8 {posargs}
[testenv]
usedevelop = True
# tox is silly... these need to be separated by a newline....
whitelist_externals = bash
                      find
install_command = pip install -U --force-reinstall {opts} {packages}
# Note the hash seed is set to 0 until nova can be tested with a
# random hash seed successfully.
setenv = VIRTUAL_ENV={envdir}
         OS_TEST_PATH=./nova/tests/unit
         LANGUAGE=en_US
deps = -r{toxinidir}/requirements.txt
       -r{toxinidir}/test-requirements.txt
commands =
  find . -type f -name "*.pyc" -delete
  bash tools/pretty_tox.sh '{posargs}'
# there is also secret magic in pretty_tox.sh which lets you run in a fail only
# mode. To do this define the TRACE_FAILONLY environmental variable.

Alternative [testenv:NAME] sections

[testenv:functional]
commands = bash -x {toxinidir}/functional/harpoon.sh

[testenv:debug]
commands = oslo_debug_helper -t openstackclient/tests {posargs}

[tox:jenkins]
downloadcache = ~/cache/pip

[testenv:jshint]
commands = nodeenv -p
           npm install jshint -g
           /bin/bash run_tests.sh -N --jshint

[testenv:genconfig]
commands =
  bash tools/config/generate_sample.sh -b . -p nova -o etc/nova

Different Style guidelines

[flake8]
show-source = True
exclude = .tox,dist,doc,*.egg,build
[flake8]
show-source = True
exclude =  .venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build,tools
[testenv:pep8]
commands = flake8
[testenv:pep8]
commands =
  /bin/bash run_tests.sh -N --pep8
  /bin/bash run_tests.sh -N --makemessages --check-only

Different Code Coverage

[testenv:cover]
commands = python setup.py testr --coverage --testr-args='{posargs}'
[testenv:cover]
# Also do not run test_coverage_ext tests while gathering coverage as those
# tests conflict with coverage.
commands =
  coverage erase
  python setup.py testr --coverage \
    --testr-args='{posargs}'
  coverage combine
  coverage html --include='nova/*' --omit='nova/openstack/common/*' -d covhtml -i

Different Docs

[testenv:docs]
commands = python setup.py build_sphinx
[testenv:docs]
commands =
  python setup.py build_sphinx
  bash -c '! find doc/ -type f -name *.json | xargs -t -n1 python -m json.tool 2>&1 > /dev/null | grep -B1 -v ^python'

Additional sections

[testenv:pip-missing-reqs]
# do not install test-requirements as that will pollute the virtualenv for
# determining missing packages
# this also means that pip-missing-reqs must be installed separately, outside
# of the requirements.txt files
deps = pip_missing_reqs
       -rrequirements.txt
commands=pip-missing-reqs -d --ignore-file=nova/tests/* nova
[hacking]
import_exceptions = oslo_log._i18n

What's Next

In a followup blog I will be talking about debugging with pdb and how to use this with tox.

References

Running openstack tests with tox

Recently the OSC (python-openstackclient) project removed run_tests.sh #177066 and tools/install_venv.py scripts #177086.

As I was very new to OpenStack development practices this threw me because of reading several OpenStack documentation pages including Getting the code that specifically mentions in Hacking on your laptop and running unit tests an example Setting Up a Developer Environment, and consulting with a friend that is a ATC this is the way I learned to setup virtual environments and running tests.

The Testing OpenStack Projects documentation also refers to run_tests.sh however caveats “There is an older convention, as follows. Most projects have a shell script, named “run_tests.sh”, that runs the unit tests of that project.” (i.e. devil in the details).

With run_tests.sh and tools/install_venv.py removed what is the *correct* way?

Setting up a virtual environment with tox

First you create the tox virtual environments. The tox configuration is held in tox.ini and this supports multiple environments for compatibility testing. You can see the environments created with.

$ grep envlist tox.ini
envlist = py33,py34,py26,py27,pep8

I have chosen to specify the Python 2.7 version. I am running Ubuntu 12.04.2 LTS which has Python 2.7 and Python 3.4. The HACKING.rst docs makes reference to “OpenStackClient strives to be Python 3.3 compatible.” Side Note: Python 3.4 fails to work with the openstackclient codebase, See later for issues I am seeing.

I setup the 2.7 virtual environment without running any tests.

$ tox -e py27 --notest
py27 create: /home/rbradfor/tmp/python-openstackclient/.tox/py27
py27 installdeps: -r/home/rbradfor/tmp/python-openstackclient/requirements.txt, -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt
py27 develop-inst: /home/rbradfor/tmp/python-openstackclient
_________ summary ____________
  py27: skipped tests
  congratulations :)

I can then reference the openstack binary directly in this virtual environment with:

$ .tox/py27/bin/openstack --version
openstack 1.1.0

You can use this virtual environment without requiring any pathing by activating.

$ source .tox/py27/bin/activate
$ openstack

This actually adds the applicable /bin directory to PATH and not PYTHONPATH.

$ which openstack
/home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/openstack
$ openstack --version
openstack 1.1.0

Running Tests

There are now several ways I can run individual or full tests.

$ tox -epy27 -- test_shell
py27 create: /home/rbradfor/tmp/python-openstackclient/.tox/py27
py27 installdeps: -r/home/rbradfor/tmp/python-openstackclient/requirements.txt, -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt
py27 develop-inst: /home/rbradfor/tmp/python-openstackclient
py27 runtests: PYTHONHASHSEED='2928878700'
py27 runtests: commands[0] | python setup.py testr --testr-args=test_shell
running testr
...
Ran 32 tests in 0.148s (+0.029s)
PASSED (id=6)
_____ summary _________________
  py27: commands succeeded
  congratulations :)

NOTE: It seems every second invocation of this fails. This is what I see.

$ tox -epy27 -- test_shell
py27 recreate: /home/rbradfor/tmp/python-openstackclient/.tox/py27
ERROR: invocation failed (exit code 3), logfile: /home/rbradfor/tmp/python-openstackclient/.tox/py27/log/py27-0.log
ERROR: actionid=py27
msg=getenv
cmdargs=['/usr/bin/python', '-m', 'virtualenv', '--setuptools', '--python', '/home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/python2.7', 'py27']
env={'LESSOPEN': '| /usr/bin/lesspipe %s', 'SSH_CLIENT': '192.168.1.2 60030 22', 'LOGNAME': 'rbradfor', 'USER': 'rbradfor', 'PATH': '/home/rbradfor/tmp/python-openstackclient/.tox/py27/bin:/home/rbradfor/tmp/python-openstackclient/.tox/py27/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'HOME': '/home/rbradfor', 'PS1': '(py27)\\[\\e]0;\\u@\\h: \\w\\a\\]${debian_chroot:+($debian_chroot)}\\u@\\h:\\w\\$ ', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'SHLVL': '1', 'PYTHONHASHSEED': '4072653076', 'XDG_RUNTIME_DIR': '/run/user/1000', 'VIRTUAL_ENV': '/home/rbradfor/tmp/python-openstackclient/.tox/py27', 'XDG_SESSION_ID': '12', '_': '/usr/local/bin/tox', 'SSH_CONNECTION': '192.168.1.2 60030 192.168.1.60 22', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'SSH_TTY': '/dev/pts/2', 'OLDPWD': '/home/rbradfor', 'PWD': '/home/rbradfor/tmp/python-openstackclient', 'MAIL': '/var/mail/rbradfor', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'}
The executable /home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/python2.7 (from --python=/home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/python2.7) does not exist

ERROR: InvocationError: /usr/bin/python -m virtualenv --setuptools --python /home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/python2.7 py27 (see /home/rbradfor/tmp/python-openstackclient/.tox/py27/log/py27-0.log)
______ summary ___________
ERROR:   py27: InvocationError: /usr/bin/python -m virtualenv --setuptools --python /home/rbradfor/tmp/python-openstackclient/.tox/py27/bin/python2.7 py27 (see /home/rbradfor/tmp/python-openstackclient/.tox/py27/log/py27-0.log)

Using the tox.ini command syntax.

$ python setup.py testr --slowest --testr-args=test_shell
...

PASSED (id=5)
Slowest Tests
Test id                                                                             Runtime (s)
----------------------------------------------------------------------------------  -----------
openstackclient.tests.test_shell.TestShellHelp.test_help_options                    0.026
openstackclient.tests.test_shell.TestShellPasswordAuth.test_only_url_flow           0.015
openstackclient.tests.test_shell.TestShellPasswordAuth.test_only_project_id_flow    0.009
openstackclient.tests.test_shell.TestShellTokenAuth.test_empty_auth                 0.009
openstackclient.tests.test_shell.TestShellTokenEndpointAuth.test_only_url           0.008
openstackclient.tests.test_shell.TestShellPasswordAuth.test_only_auth_type_flow     0.007
openstackclient.tests.test_shell.TestShellPasswordAuth.test_only_project_name_flow  0.007
openstackclient.tests.test_shell.TestShellPasswordAuth.test_only_trust_id_flow      0.007
openstackclient.tests.test_shell.TestShellTokenAuthEnv.test_only_auth_url           0.007
openstackclient.tests.test_shell.TestShellTokenAuthEnv.test_only_token              0.007

References

The following is recommended reading.

Thanks to Jeremy Stanley (fungi) and Doug Hellmann from the openstack-dev mailing list for setting me on the correct path.

Problems with Python 3.4 on Ubuntu 14.04.2 LTS

I was unable to run tests with Python 3.x. I have not spent the time to investigate why there are issues with libyaml which is not listed as core dependency in requirements.txt.
UPDATE: Seems this also is a simple problem. I did not have the -dev package installed.

$ sudo apt-get install -y python3.4-dev

And it’s all fine. Thanks Doug for that insight.

My next objective is to install Python 3.3 also as this is referenced as the baseline compatibility of the project.

$ git rev-parse HEAD
416d840dc4cb00026bac2512b259ce88a0e4a765

$ tox -epy34 -- notests
py34 create: /home/rbradfor/tmp/python-openstackclient/.tox/py34
py34 installdeps: -r/home/rbradfor/tmp/python-openstackclient/requirements.txt, -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile: /home/rbradfor/tmp/python-openstackclient/.tox/py34/log/py34-1.log
ERROR: actionid=py34
msg=getenv
cmdargs=[local('/home/rbradfor/tmp/python-openstackclient/.tox/py34/bin/pip'), 'install', '-U', '-r/home/rbradfor/tmp/python-openstackclient/requirements.txt', '-r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt']
env={'XDG_RUNTIME_DIR': '/run/user/1000', 'VIRTUAL_ENV': '/home/rbradfor/tmp/python-openstackclient/.tox/py34', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'SSH_CLIENT': '192.168.1.2 60030 22', 'LOGNAME': 'rbradfor', 'USER': 'rbradfor', 'HOME': '/home/rbradfor', 'PATH': '/home/rbradfor/tmp/python-openstackclient/.tox/py34/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'XDG_SESSION_ID': '12', '_': '/usr/local/bin/tox', 'SSH_CONNECTION': '192.168.1.2 60030 192.168.1.60 22', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'SHLVL': '1', 'SSH_TTY': '/dev/pts/2', 'OLDPWD': '/home/rbradfor', 'PWD': '/home/rbradfor/tmp/python-openstackclient', 'PYTHONHASHSEED': '1330227753', 'MAIL': '/var/mail/rbradfor', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'}
Collecting pbr!=0.7,<1.0,>=0.6 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 4))
  Using cached pbr-0.10.8-py2.py3-none-any.whl
Collecting six>=1.9.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 5))
  Using cached six-1.9.0-py2.py3-none-any.whl
Collecting Babel>=1.3 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 7))
  Using cached Babel-1.3.tar.gz
Collecting cliff>=1.10.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 8))
  Using cached cliff-1.12.0.tar.gz
Collecting cliff-tablib>=1.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 9))
  Using cached cliff-tablib-1.1.tar.gz
Collecting os-client-config (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 10))
  Using cached os-client-config-0.8.0.tar.gz
Collecting oslo.config>=1.9.3 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 11))
  Using cached oslo.config-1.11.0-py2.py3-none-any.whl
Collecting oslo.i18n>=1.5.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 12))
  Using cached oslo.i18n-1.6.0-py2.py3-none-any.whl
Collecting oslo.utils>=1.4.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 13))
  Using cached oslo.utils-1.5.0-py2.py3-none-any.whl
Collecting oslo.serialization>=1.4.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 14))
  Using cached oslo.serialization-1.5.0-py2.py3-none-any.whl
Collecting python-glanceclient>=0.15.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached python_glanceclient-0.18.0-py2.py3-none-any.whl
Collecting python-keystoneclient>=1.1.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 16))
  Using cached python_keystoneclient-1.4.0-py2.py3-none-any.whl
Collecting python-novaclient>=2.22.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 17))
  Using cached python_novaclient-2.24.1-py2.py3-none-any.whl
Collecting python-cinderclient>=1.1.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 18))
  Using cached python_cinderclient-1.2.0-py2.py3-none-any.whl
Collecting python-neutronclient<3,>=2.3.11 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 19))
  Using cached python_neutronclient-2.5.0-py2.py3-none-any.whl
Collecting requests!=2.4.0,>=2.2.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 20))
  Using cached requests-2.6.2-py2.py3-none-any.whl
Collecting stevedore>=1.3.0 (from -r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 21))
  Using cached stevedore-1.4.0-py2.py3-none-any.whl
Collecting hacking<0.11,>=0.10.0 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 4))
  Using cached hacking-0.10.1-py2.py3-none-any.whl
Collecting coverage>=3.6 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 6))
  Using cached coverage-3.7.1.tar.gz
Collecting discover (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 7))
  Using cached discover-0.4.0.tar.gz
Collecting fixtures>=0.3.14 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 8))
  Using cached fixtures-1.0.0.tar.gz
Collecting mock>=1.0 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 9))
  Using cached mock-1.0.1.tar.gz
Collecting oslosphinx>=2.5.0 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 10))
  Using cached oslosphinx-2.5.0-py2.py3-none-any.whl
Collecting oslotest>=1.5.1 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 11))
  Using cached oslotest-1.6.0-py2.py3-none-any.whl
Collecting requests-mock>=0.6.0 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 12))
  Using cached requests_mock-0.6.0-py2.py3-none-any.whl
Collecting sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 13))
  Using cached Sphinx-1.2.3-py3-none-any.whl
Collecting testrepository>=0.0.18 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 14))
  Using cached testrepository-0.0.20.tar.gz
Collecting testtools!=1.2.0,>=0.9.36 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached testtools-1.7.1-py2.py3-none-any.whl
Collecting WebOb>=1.2.3 (from -r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 16))
  Using cached WebOb-1.4.1.tar.gz
Requirement already up-to-date: pip in ./.tox/py34/lib/python3.4/site-packages (from pbr!=0.7,<1.0,>=0.6->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 4))
Collecting pytz>=0a (from Babel>=1.3->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 7))
  Using cached pytz-2015.2-py2.py3-none-any.whl
Collecting argparse (from cliff>=1.10.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 8))
  Using cached argparse-1.3.0-py2.py3-none-any.whl
Collecting cmd2>=0.6.7 (from cliff>=1.10.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 8))
  Using cached cmd2-0.6.8.tar.gz
Collecting PrettyTable<0.8,>=0.7 (from cliff>=1.10.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 8))
  Using cached prettytable-0.7.2.tar.bz2
Collecting pyparsing>=2.0.1 (from cliff>=1.10.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 8))
  Using cached pyparsing-2.0.3-py2.py3-none-any.whl
Collecting tablib (from cliff-tablib>=1.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 9))
  Using cached tablib-0.10.0-py2.py3-none-any.whl
Collecting PyYAML>=3.1.0 (from os-client-config->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 10))
  Using cached PyYAML-3.11.tar.gz
Collecting netaddr>=0.7.12 (from oslo.config>=1.9.3->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 11))
  Using cached netaddr-0.7.14-py2.py3-none-any.whl
Collecting iso8601>=0.1.9 (from oslo.utils>=1.4.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 13))
  Using cached iso8601-0.1.10-py33-none-any.whl
Collecting netifaces>=0.10.4 (from oslo.utils>=1.4.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 13))
  Using cached netifaces-0.10.4.tar.gz
Collecting msgpack-python>=0.4.0 (from oslo.serialization>=1.4.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 14))
  Using cached msgpack-python-0.4.6.tar.gz
Collecting pyOpenSSL>=0.11 (from python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached pyOpenSSL-0.15.1-py2.py3-none-any.whl
Collecting warlock<2,>=1.0.1 (from python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached warlock-1.1.0.tar.gz
Collecting simplejson>=2.2.0 (from python-novaclient>=2.22.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 17))
  Using cached simplejson-3.6.5.tar.gz
Collecting flake8==2.2.4 (from hacking<0.11,>=0.10.0->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 4))
  Using cached flake8-2.2.4.tar.gz
Collecting pep8==1.5.7 (from hacking<0.11,>=0.10.0->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 4))
  Using cached pep8-1.5.7-py2.py3-none-any.whl
Collecting mccabe==0.2.1 (from hacking<0.11,>=0.10.0->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 4))
  Using cached mccabe-0.2.1.tar.gz
Collecting pyflakes==0.8.1 (from hacking<0.11,>=0.10.0->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 4))
  Using cached pyflakes-0.8.1-py2.py3-none-any.whl
Collecting testscenarios>=0.4 (from oslotest>=1.5.1->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 11))
  Using cached testscenarios-0.4.tar.gz
Collecting python-subunit>=0.0.18 (from oslotest>=1.5.1->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 11))
  Using cached python-subunit-1.1.0.tar.gz
Collecting mox3>=0.7.0 (from oslotest>=1.5.1->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 11))
  Using cached mox3-0.7.0.tar.gz
Collecting docutils>=0.10 (from sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 13))
  Using cached docutils-0.12.tar.gz
Collecting Jinja2>=2.3 (from sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 13))
  Using cached Jinja2-2.7.3.tar.gz
Collecting Pygments>=1.2 (from sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 13))
  Using cached Pygments-2.0.2-py3-none-any.whl
Collecting unittest2>=1.0.0 (from testtools!=1.2.0,>=0.9.36->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached unittest2-1.0.1-py2.py3-none-any.whl
Collecting traceback2 (from testtools!=1.2.0,>=0.9.36->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached traceback2-1.4.0-py2.py3-none-any.whl
Collecting extras (from testtools!=1.2.0,>=0.9.36->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached extras-0.0.3.tar.gz
Collecting python-mimeparse (from testtools!=1.2.0,>=0.9.36->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached python-mimeparse-0.1.4.tar.gz
Collecting cryptography>=0.7 (from pyOpenSSL>=0.11->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached cryptography-0.8.2.tar.gz
Collecting jsonschema<3,>=0.7 (from warlock<2,>=1.0.1->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached jsonschema-2.4.0-py2.py3-none-any.whl
Collecting jsonpatch<2,>=0.10 (from warlock<2,>=1.0.1->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached jsonpatch-1.9.tar.gz
Collecting markupsafe (from Jinja2>=2.3->sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 13))
  Using cached MarkupSafe-0.23.tar.gz
Collecting linecache2 (from traceback2->testtools!=1.2.0,>=0.9.36->-r /home/rbradfor/tmp/python-openstackclient/test-requirements.txt (line 15))
  Using cached linecache2-1.0.0-py2.py3-none-any.whl
Collecting pyasn1 (from cryptography>=0.7->pyOpenSSL>=0.11->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached pyasn1-0.1.7.tar.gz
Collecting setuptools (from cryptography>=0.7->pyOpenSSL>=0.11->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached setuptools-15.2-py2.py3-none-any.whl
Collecting cffi>=0.8 (from cryptography>=0.7->pyOpenSSL>=0.11->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached cffi-0.9.2.tar.gz
Collecting jsonpointer>=1.5 (from jsonpatch<2,>=0.10->warlock<2,>=1.0.1->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached jsonpointer-1.7.tar.gz
Collecting pycparser (from cffi>=0.8->cryptography>=0.7->pyOpenSSL>=0.11->python-glanceclient>=0.15.0->-r /home/rbradfor/tmp/python-openstackclient/requirements.txt (line 15))
  Using cached pycparser-2.12.tar.gz
Installing collected packages: pbr, six, pytz, Babel, argparse, pyparsing, cmd2, PrettyTable, stevedore, cliff, tablib, cliff-tablib, PyYAML, os-client-config, netaddr, oslo.config, oslo.i18n, iso8601, netifaces, oslo.utils, msgpack-python, oslo.serialization, pyasn1, setuptools, pycparser, cffi, cryptography, pyOpenSSL, requests, jsonschema, jsonpointer, jsonpatch, warlock, python-keystoneclient, python-glanceclient, simplejson, python-novaclient, python-cinderclient, python-neutronclient, pyflakes, pep8, mccabe, flake8, hacking, coverage, discover, linecache2, traceback2, unittest2, extras, python-mimeparse, testtools, fixtures, mock, oslosphinx, testscenarios, python-subunit, mox3, testrepository, oslotest, requests-mock, docutils, markupsafe, Jinja2, Pygments, sphinx, WebOb
  Running setup.py install for Babel
  Running setup.py install for cmd2
  Running setup.py install for PrettyTable
  Running setup.py install for cliff
  Running setup.py install for cliff-tablib
  Running setup.py install for PyYAML
    Complete output from command /home/rbradfor/tmp/python-openstackclient/.tox/py34/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-p19auoc2/PyYAML/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xlgu_evx-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/rbradfor/tmp/python-openstackclient/.tox/py34/include/site/python3.4/PyYAML:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.4
    creating build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/representer.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/tokens.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/constructor.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/reader.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/__init__.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/error.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/scanner.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/loader.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/parser.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/nodes.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/serializer.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/cyaml.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/emitter.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/events.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/dumper.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/resolver.py -> build/lib.linux-x86_64-3.4/yaml
    copying lib3/yaml/composer.py -> build/lib.linux-x86_64-3.4/yaml
    running build_ext
    creating build/temp.linux-x86_64-3.4
    checking if libyaml is compilable
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -I/home/rbradfor/tmp/python-openstackclient/.tox/py34/include/python3.4m -c build/temp.linux-x86_64-3.4/check_libyaml.c -o build/temp.linux-x86_64-3.4/check_libyaml.o
    checking if libyaml is linkable
    x86_64-linux-gnu-gcc -pthread build/temp.linux-x86_64-3.4/check_libyaml.o -lyaml -o build/temp.linux-x86_64-3.4/check_libyaml
    building '_yaml' extension
    creating build/temp.linux-x86_64-3.4/ext
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -I/home/rbradfor/tmp/python-openstackclient/.tox/py34/include/python3.4m -c ext/_yaml.c -o build/temp.linux-x86_64-3.4/ext/_yaml.o
    ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory
     #include "pyconfig.h"
                          ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
    Command "/home/rbradfor/tmp/python-openstackclient/.tox/py34/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-p19auoc2/PyYAML/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xlgu_evx-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/rbradfor/tmp/python-openstackclient/.tox/py34/include/site/python3.4/PyYAML" failed with error code 1 in /tmp/pip-build-p19auoc2/PyYAML

ERROR: could not install deps [-r/home/rbradfor/tmp/python-openstackclient/requirements.txt, -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt]; v = InvocationError('/home/rbradfor/tmp/python-openstackclient/.tox/py34/bin/pip install -U -r/home/rbradfor/tmp/python-openstackclient/requirements.txt -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt (see /home/rbradfor/tmp/python-openstackclient/.tox/py34/log/py34-1.log)', 1)
________________________________________________________________________________________________________________________ summary ________________________________________________________________________________________________________________________
ERROR:   py34: could not install deps [-r/home/rbradfor/tmp/python-openstackclient/requirements.txt, -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt]; v = InvocationError('/home/rbradfor/tmp/python-openstackclient/.tox/py34/bin/pip install -U -r/home/rbradfor/tmp/python-openstackclient/requirements.txt -r/home/rbradfor/tmp/python-openstackclient/test-requirements.txt (see /home/rbradfor/tmp/python-openstackclient/.tox/py34/log/py34-1.log)', 1)

Inconsistent messaging for OpenStackClient

As I mentioned earlier in Moving to OpenStackClient CLI I came across several differences in reconciling the legacy CLI tools.

I have also come across very inconsistent messaging. Here is a simple example.

In nova

$ nova list
ERROR (CommandError): You must provide an auth url via either --os-auth-url or env[OS_AUTH_URL] or specify an auth_system which defines a default url with --os-auth-system or env[OS_AUTH_SYSTEM]

In openstack

$ openstack service list
ERROR: openstack Authorization Failed: Cannot authenticate without an auth_url

$ openstack keypair list
ERROR: openstack Authentication requires 'auth_url', which should be specified in 'HTTPClient'

$ openstack catalog list
ERROR: openstack 'NoneType' object has no attribute 'auth'

$ openstack volume list
ERROR: openstack unsupported operand type(s) for +: 'NoneType' and 'str'

All three errors are effectively the same. That is I have not setup environment variables (OS_PROJECT_NAME, OS_AUTH_URL, OS_USERNAME, OS_PASSWORD) or passed all correctly as arguments.

In delving to the openstackclient source code specifically auth.py#147 I see another message “Set an authentication URL, with –os-auth-url, OS_AUTH_URL or auth.auth_url”. api.auth is also referenced in the Authentication Documentation as the place to start.

Time to delve in the code to see what I can find out.

cd
git clone git://git.openstack.org/openstack/python-openstackclient
cd python-openstackclient
python tools/install_venv.py
source .venv/bin/activate
# Don't need sudo for local environment
which openstack

After setting OS_USERNAME, OS_PASSWORD, and OS_PROJECT NAME (and not setting OS_AUTH_URL in my test) I run the following.

$ openstack image list
WARNING: openstackclient.shell Possible error authenticating: Missing parameter(s):
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url

ERROR: openstack Missing parameter(s):
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url

This matches what I saw in the code, so it’s the installed version that is older code. While user, service and keypair lists returns the same message volume and catalog still do not.

(.venv)rbradfor@octogon:~/python-openstackclient$ openstack volume list
WARNING: openstackclient.shell Possible error authenticating: Missing parameter(s):
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url

ERROR: openstack unsupported operand type(s) for +: 'NoneType' and 'str'

(.venv)rbradfor@octogon:~/python-openstackclient$ openstack catalog list
WARNING: openstackclient.shell Possible error authenticating: Missing parameter(s):
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url

ERROR: openstack 'NoneType' object has no attribute 'auth'

Moving to OpenStackClient CLI

In working with the keynote CLI within the TripleO scripts I came across the following deprecation warning message.

$ keystone token-get
/usr/local/lib/python2.7/dist-packages/keystoneclient/shell.py:65: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)

Time to switch to using the OpenStackClient, historically also called the unified CLI.

Very easy to install.

$ sudo pip install python-openstackclient

$ openstack help
usage: openstack help [-h] [cmd [cmd ...]]

print detailed help for another command

positional arguments:
  cmd         name of the command

optional arguments:
  -h, --help  show this help message and exit

I would also suggest you add the following alias to your startup shell rc.

alias os='openstack'

The --help option also provides a much detailed list of available argument options.

$ os --help
usage: openstack [--version] [-v] [--log-file LOG_FILE] [-q] [--debug]
                 [--os-region-name ]
                 [--os-cacert ] [--verify | --insecure]
                 [--os-default-domain ] [--timing]
                 [--os-compute-api-version ]
                 [--os-network-api-version ]
                 [--os-image-api-version ]
                 [--os-volume-api-version ]
                 [--os-identity-api-version ]
                 [--os-auth-type ] [--os-username ]
                 [--os-identity-provider ]
                 [--os-project-domain-name ]
                 [--os-project-domain-id ]
                 [--os-project-name ]
                 [--os-auth-url ]
                 [--os-trust-id ]
                 [--os-service-provider-endpoint ]
                 [--os-user-domain-id ]
                 [--os-domain-name ]
                 [--os-identity-provider-url ]
                 [--os-token ] [--os-domain-id ]
                 [--os-url ]
                 [--os-user-domain-name ]
                 [--os-user-id ] [--os-password ]
                 [--os-project-id ]
                 [--os-object-api-version ] [-h]

Command-line interface to the OpenStack APIs

optional arguments:
  --version             show program's version number and exit
  -v, --verbose         Increase verbosity of output. Can be repeated.
...

The new CLI provides a number of benefits above the consolidation of syntax into a single client. There is the flexibility in formatting output, both selecting columns and output format.

Working with nova for more simple initial examples.

$ nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID                                   | Name                            | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| a8672506-049f-4fda-bc58-e64a646d587e | cirros-0.3.2-x86_64-uec         | ACTIVE |        |
| 557dec3a-f912-430c-bda1-ace9c669b78d | cirros-0.3.2-x86_64-uec-kernel  | ACTIVE |        |
| 1d9b6c2e-96d9-4426-a98d-7fa378c26189 | cirros-0.3.2-x86_64-uec-ramdisk | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+
$ openstack image list
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| a8672506-049f-4fda-bc58-e64a646d587e | cirros-0.3.2-x86_64-uec         |
| 1d9b6c2e-96d9-4426-a98d-7fa378c26189 | cirros-0.3.2-x86_64-uec-ramdisk |
| 557dec3a-f912-430c-bda1-ace9c669b78d | cirros-0.3.2-x86_64-uec-kernel  |
+--------------------------------------+---------------------------------+

As you can see the Status and Server columns are not in the default format. You can access a list of columns, however both the column order, and even the contents (e.g. ACTIVE v active) means you need to adjust your existing scripts. In this case Status exists, Server does not.

$ openstack image list --long
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+------------+-----------+----------------------------------+-----------------------------------------------------------------------------------------------------+
| ID                                   | Name                            | Disk Format | Container Format |     Size | Status | Visibility | Protected | Owner                            | Properties                                                                                          |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+------------+-----------+----------------------------------+-----------------------------------------------------------------------------------------------------+
| a8672506-049f-4fda-bc58-e64a646d587e | cirros-0.3.2-x86_64-uec         | ami         | ami              | 25165824 | active | public     | False     | 0087dccc995e4ea3aaf209ddc8ad33e2 | kernel_id='557dec3a-f912-430c-bda1-ace9c669b78d', ramdisk_id='1d9b6c2e-96d9-4426-a98d-7fa378c26189' |
| 1d9b6c2e-96d9-4426-a98d-7fa378c26189 | cirros-0.3.2-x86_64-uec-ramdisk | ari         | ari              |  3723817 | active | public     | False     | 0087dccc995e4ea3aaf209ddc8ad33e2 |                                                                                                     |
| 557dec3a-f912-430c-bda1-ace9c669b78d | cirros-0.3.2-x86_64-uec-kernel  | aki         | aki              |  4969360 | active | public     | False     | 0087dccc995e4ea3aaf209ddc8ad33e2 |                                                                                                     |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+------------+-----------+----------------------------------+-----------------------------------------------------------------------------------------------------+

Column Options

openstack provides the ability to pass column names, so we can try to simulate what we seen in the legacy nova client.

$ openstack image list -c ID
+--------------------------------------+
| ID                                   |
+--------------------------------------+
| 1b79bbb2-e8b6-430f-8cb7-ced8f3589807 |
| e52bd241-db30-40e2-9230-40f21ff3e4c7 |
| 59e28cdf-c4af-4b02-bdeb-3779ec4c306d |
+--------------------------------------+

$ openstack image list -c ID -c Name
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| 1b79bbb2-e8b6-430f-8cb7-ced8f3589807 | cirros-0.3.2-x86_64-uec         |
| e52bd241-db30-40e2-9230-40f21ff3e4c7 | cirros-0.3.2-x86_64-uec-ramdisk |
| 59e28cdf-c4af-4b02-bdeb-3779ec4c306d | cirros-0.3.2-x86_64-uec-kernel  |
+--------------------------------------+---------------------------------+

$ openstack image list -c ID -c Name -c Status
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| 1b79bbb2-e8b6-430f-8cb7-ced8f3589807 | cirros-0.3.2-x86_64-uec         |
| e52bd241-db30-40e2-9230-40f21ff3e4c7 | cirros-0.3.2-x86_64-uec-ramdisk |
| 59e28cdf-c4af-4b02-bdeb-3779ec4c306d | cirros-0.3.2-x86_64-uec-kernel  |
+--------------------------------------+---------------------------------+

The Status column which is in both nova and openstack output above seems to provide no output or even an error. However if specified as the only individual column it draws an expected error message.

stack@ubuntu:~$ openstack image list -c Status
ERROR: openstack No recognized column names in ['Status']

Not knowing what the actual list of valid columns are, as it does not match the --long output we move on.

Update 4/23/15
It pays to read the source code, specifically openstackclient/image/v2/image.py. In order to list additional columns you must also specify --long.

$ python openstackclient/shell.py image list --long -c ID -c Name -c Status
+--------------------------------------+---------------------------------+--------+
| ID                                   | Name                            | Status |
+--------------------------------------+---------------------------------+--------+
| c5153c2b-df9c-488c-995e-5cb347c0ee35 | cirros-0.3.2-x86_64-uec         | active |
| ac7e0c04-e47c-43da-ba28-b0d47d293eb7 | cirros-0.3.2-x86_64-uec-ramdisk | active |
| 52cc99f5-a9aa-4f27-98b8-d8ec762a246d | cirros-0.3.2-x86_64-uec-kernel  | active |
+--------------------------------------+---------------------------------+--------+

I should also point out that column names are Case Sensitive. Id is not valid.

$ python openstackclient/shell.py image list --long -c Id -c Name -c Status
+---------------------------------+--------+
| Name                            | Status |
+---------------------------------+--------+
| cirros-0.3.2-x86_64-uec         | active |
| cirros-0.3.2-x86_64-uec-ramdisk | active |
| cirros-0.3.2-x86_64-uec-kernel  | active |
+---------------------------------+--------+

Format Options

One cool feature is the formatting options. There are currently 6 types.

TABLE

$ openstack image list --format table
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| a8672506-049f-4fda-bc58-e64a646d587e | cirros-0.3.2-x86_64-uec         |
| 1d9b6c2e-96d9-4426-a98d-7fa378c26189 | cirros-0.3.2-x86_64-uec-ramdisk |
| 557dec3a-f912-430c-bda1-ace9c669b78d | cirros-0.3.2-x86_64-uec-kernel  |
+--------------------------------------+---------------------------------+

CSV

$ openstack image list --format csv
"ID","Name"
"a8672506-049f-4fda-bc58-e64a646d587e","cirros-0.3.2-x86_64-uec"
"1d9b6c2e-96d9-4426-a98d-7fa378c26189","cirros-0.3.2-x86_64-uec-ramdisk"
"557dec3a-f912-430c-bda1-ace9c669b78d","cirros-0.3.2-x86_64-uec-kernel"

JSON

$ openstack image list --format json | python -m json.tool
[
    {
        "ID": "1b79bbb2-e8b6-430f-8cb7-ced8f3589807",
        "Name": "cirros-0.3.2-x86_64-uec"
    },
    {
        "ID": "e52bd241-db30-40e2-9230-40f21ff3e4c7",
        "Name": "cirros-0.3.2-x86_64-uec-ramdisk"
    },
    {
        "ID": "59e28cdf-c4af-4b02-bdeb-3779ec4c306d",
        "Name": "cirros-0.3.2-x86_64-uec-kernel"
    }
]

HTML

$ openstack image list --format html
<table>
<thead>
<tr><th>ID</th>
<th>Name</th></tr>
</thead>
<tr><td>a8672506-049f-4fda-bc58-e64a646d587e</td>
<td>cirros-0.3.2-x86_64-uec</td></tr>
<tr><td>1d9b6c2e-96d9-4426-a98d-7fa378c26189</td>
<td>cirros-0.3.2-x86_64-uec-ramdisk</td></tr>
<tr><td>557dec3a-f912-430c-bda1-ace9c669b78d</td>
<td>cirros-0.3.2-x86_64-uec-kernel</td></tr>
</table>

YAML

$ openstack image list --format yaml
- {ID: a8672506-049f-4fda-bc58-e64a646d587e, Name: cirros-0.3.2-x86_64-uec}
- {ID: 1d9b6c2e-96d9-4426-a98d-7fa378c26189, Name: cirros-0.3.2-x86_64-uec-ramdisk}
- {ID: 557dec3a-f912-430c-bda1-ace9c669b78d, Name: cirros-0.3.2-x86_64-uec-kernel}

SHELL

$ openstack image list --format shell
usage: openstack image list [-h] [-f {csv,html,json,table,yaml}] [-c COLUMN]
                            [--max-width ]
                            [--quote {all,minimal,none,nonnumeric}]
                            [--public | --private] [--property ]
                            [--long] [--sort [:]]
openstack image list: error: argument -f/--format: invalid choice: 'shell' (choose from 'csv', 'html', 'json', 'table', 'yaml')

As you can see shell is invalid for the list argument, however it is valid for the show argument.

$ openstack image show a8672506-049f-4fda-bc58-e64a646d587e
+------------------+-----------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                           |
+------------------+-----------------------------------------------------------------------------------------------------------------+
| checksum         | 4eada48c2843d2a262c814ddc92ecf2c                                                                                |
| container_format | ami                                                                                                             |
| created_at       | 2015-03-31T19:42:53.000000                                                                                      |
| deleted          | False                                                                                                           |
| disk_format      | ami                                                                                                             |
| id               | a8672506-049f-4fda-bc58-e64a646d587e                                                                            |
| is_public        | True                                                                                                            |
| min_disk         | 0                                                                                                               |
| min_ram          | 0                                                                                                               |
| name             | cirros-0.3.2-x86_64-uec                                                                                         |
| owner            | 0087dccc995e4ea3aaf209ddc8ad33e2                                                                                |
| properties       | {u'kernel_id': u'557dec3a-f912-430c-bda1-ace9c669b78d', u'ramdisk_id': u'1d9b6c2e-96d9-4426-a98d-7fa378c26189'} |
| protected        | False                                                                                                           |
| size             | 25165824                                                                                                        |
| status           | active                                                                                                          |
| updated_at       | 2015-03-31T19:42:54.000000                                                                                      |
+------------------+-----------------------------------------------------------------------------------------------------------------+
$ openstack image show a8672506-049f-4fda-bc58-e64a646d587e --format=shell
checksum="4eada48c2843d2a262c814ddc92ecf2c"
container_format="ami"
created_at="2015-03-31T19:42:53.000000"
deleted="False"
disk_format="ami"
id="a8672506-049f-4fda-bc58-e64a646d587e"
is_public="True"
min_disk="0"
min_ram="0"
name="cirros-0.3.2-x86_64-uec"
owner="0087dccc995e4ea3aaf209ddc8ad33e2"
properties="{u'kernel_id': u'557dec3a-f912-430c-bda1-ace9c669b78d', u'ramdisk_id': u'1d9b6c2e-96d9-4426-a98d-7fa378c26189'}"
protected="False"
size="25165824"
status="active"
updated_at="2015-03-31T19:42:54.000000"

Interactive

The CLI also provides an interactive shell.

$ openstack
(openstack) image list
+--------------------------------------+---------------------------------+
| ID                                   | Name                            |
+--------------------------------------+---------------------------------+
| 1b79bbb2-e8b6-430f-8cb7-ced8f3589807 | cirros-0.3.2-x86_64-uec         |
| e52bd241-db30-40e2-9230-40f21ff3e4c7 | cirros-0.3.2-x86_64-uec-ramdisk |
| 59e28cdf-c4af-4b02-bdeb-3779ec4c306d | cirros-0.3.2-x86_64-uec-kernel  |
+--------------------------------------+---------------------------------+
(openstack) quit

AWS cost saving tips – EBS Volumes

A trivial cost saving tip for checking if you are spending money in your AWS environment on unused resources. This is especially appropriate when using provisioned IOPS EBS volumes.

$ ec2-describe-volumes | grep available

VOLUME	vol-44dff904	8	snap-d86d0884	us-east-1b	available	2014-08-01T14:11:24+0000	standard
VOLUME	vol-62dff922	100		us-east-1b	available	2014-08-01T14:11:24+0000	io1	1000
VOLUME	vol-15dff955	8	snap-d86d0884	us-east-1b	available	2014-08-01T14:11:24+0000	standard
VOLUME	vol-80a88ec0	8	snap-d86d0884	us-east-1b	available	2014-08-01T15:12:54+0000	standard
VOLUME	vol-ca82a48a	100		us-east-1b	available	2014-08-01T16:13:49+0000	standard
VOLUME	vol-5d79581d	8	snap-d86d0884	us-east-1b	available	2014-08-01T18:27:01+0000	standard
VOLUME	vol-baf9dbfa	8	snap-d86d0884	us-east-1b	available	2014-08-03T18:20:59+0000	standard
VOLUME	vol-53ffdd13	8	snap-d86d0884	us-east-1b	available	2014-08-03T18:25:52+0000	standard
VOLUME	vol-ade7daed	8	snap-d86d0884	us-east-1b	available	2014-08-13T20:10:46+0000	standard
VOLUME	vol-34e2df74	8	snap-065a2e52	us-east-1b	available	2014-08-13T20:26:17+0000	standard
VOLUME	vol-cacef38a	100	snap-280ffb7f	us-east-1b	available	2014-08-13T21:19:18+0000	standard
VOLUME	vol-41350a01	8	snap-f23ccba5	us-east-1b	available	2014-08-14T16:54:27+0000	standard
VOLUME	vol-51350a11	100	snap-fc3ccbab	us-east-1b	available	2014-08-14T16:54:27+0000	standard
VOLUME	vol-912f10d1	8	snap-96ee24c1	us-east-1b	available	2014-08-14T17:15:06+0000	standard
VOLUME	vol-a82f10e8	100	snap-9dee24ca	us-east-1b	available	2014-08-14T17:15:06+0000	standard

These are available and unused EBS volumes which you should consider deleting.

Improving performance – A full stack problem

Improving the performance of a web system involves knowledge of how the entire technology stack operates and interacts. There are many simple and common tips that can provide immediate improvements for a website. Some examples include:

  • Using a CDN for assets
  • Compressing content
  • Making fewer requests (web, cache, database)
  • Asynchronous management
  • Optimizing your SQL statements
  • Have more memory
  • Using SSD’s for database servers
  • Updating your software versions
  • Adding more servers
  • Configuring your software correctly
  • … And the general checklist goes on

Understanding where to invest your energy first, knowing what the return on investment can be, and most importantly the measurement and verification of every change made is the difference between blind trial and error and a solid plan and process. Here is a great example for the varied range of outcome to the point about “Updating your software versions”.

On one project the MySQL database was reaching saturation, both the maximum number of database connections and maximum number of concurrent InnoDB transactions. The first is a configurable limit, the second was a hard limit of the very old version of the software. Changing the first configurable limit can have dire consequences, there is a tipping point, however that is a different discussion. A simple software upgrade of MySQL which had many possible improvement benefits, combined with corrected configuration specific for this new version made an immediate improvement. The result moved a production system from crashing consistently under load, to at least barely surviving under load. This is an important first step in improving the customer experience.

In the PHP application stack for the same project the upgrading of several commonly used frameworks including Slim and Twig by the engineering department seemed like a good idea. However applicable load testing and profiling (after it was deployed, yet another discussion point) found the impact was a 30-40% increase in response time for the application layer. This made the system worse, and cancelled out prior work to improve the system.

How to tune a system to support 100x load increase with no impact in performance takes knowledge, experience, planning, testing and verification.

The following summarized graphs; using New Relic monitoring as a means of representative comparison; shows three snapshots of the average response time during various stages of full stack tuning and optimization. This is a very simplified graphical view that is supported by more detailed instrumentation using different products, specifically with much finer granularity of hundreds of metrics.

These graphs represent the work undertaken for a system under peak load showing an average 2,000ms response time, to the same workload under 50ms average response time. That is a 40x improvement!

If your organization can benefit from these types of improvements feel free to Contact Me.

There are numerous steps to achieving this. A few highlights to show the scope of work you need to consider includes:

  • Knowing server CPU saturation verses single core CPU saturation.
  • Network latency detection and mitigation.
  • What are the virtualization mode options of virtual cloud instances?
  • Knowing the network stack benefits of different host operating systems.
  • Simulating production load is much harder than it sounds.
  • Profiling, Profiling, Profiling.
  • Instrumentation can be misleading. Knowing how different monitoring works with sampling and averaging.
  • Tuning the stack is an iterative process.
  • The simple greatest knowledge is to know your code, your libraries, your dependencies and how to optimize each specific area of your technology stack.
  • Not everything works, some expected wins provided no overall or observed benefits.
  • There is always more that can be done. Knowing when to pause and prioritize process optimizations over system optimizations.

These graphs show the improvement work in the application tier (1500ms to 35ms to 25ms) and the database tier (500ms to 125ms to 10ms) at various stages. These graphs do not show for example improvements made in DNS resolution, different CDNs, managing static content, different types and ways of compression, remove unwanted software components and configuration, standardized and consistent stack deployments using chef, and even a reduction in overall servers. All of these successes contributed to a better and more consistent user experience.

40x performance improvements in LAMP stack

Writing re-runable shell script

I recently started playing with devstack again (An all-in-on OpenStack developer setup). Last time was over 3 years ago because I remember a pull request for a missing dependency at the time.

The installation docs provide information to bootstrap your system with a necessary user and privileges, however like many docs for software setup they contain one off instructions.

adduser stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

When you write operations code you need to always be thinking about “testability” and “automation”. It is important to write re-runable code. You should always write parameterized code when possible, which can be refactored into usable functions at any time.

This is a good example to demonstrate a simple test condition for making the initial instructions re-runable.

sudo su -
NEW_USER="stack"
# This creates default group of same username
# This creates user with default HOME in /home/stack
[ `grep ${NEW_USER} /etc/passwd | wc -l` -eq 0 ] && useradd -s /bin/bash -m ${NEW_USER}
NEW_USER_SUDO_FILE="/etc/sudoers.d/${NEW_USER}"
[ ! -s ${NEW_USER_SUDO_FILE} ] && umask 226 && echo "${NEW_USER} ALL=(ALL) NOPASSWD: ALL" > ${NEW_USER_SUDO_FILE}
ls -l ${NEW_USER_SUDO_FILE}

Another reason to avoid RDS

My list of reasons for never using or recommending Amazon’s MySQL RDS service grows every time I experience problems with customers. This was an interesting and still unresolved issue.

ERROR 126 (HY000): Incorrect key file for table '/rdsdbdata/tmp/#sql_5b7_1.MYI'; try to repair it

You may see this is a MyISAM table. The MySQL database is version 5.5, all InnoDB tables and is very small 100MB in total size.
What is happening is that MySQL is generating a temporary table, and this table is being written to disk. I am unable to change the code to improve the query causing this disk I/O.

What I can not understand and have no ability to diagnose is why this error occurs sometimes and generally when the database is under additional system load. With RDS you have no visibility of the server running the production database. While you have SQL access, an API for managing MySQL configuration options (I also add not all MySQL variables), and limited system statistics via a graphical interface, all information about the system performance, disk configuration etc is hidden and not accessible. This is a frustrating limitation of using RDS.

NOTE: While I cannot recommend RDS, I am very happy with AWS EC2 services when correctly configured. For a cloud based MySQL solution I would definitely recommend greater control over your MySQL database using EC2 and EBS.

Basic scalability principles to avert downtime

In the press in the last two days has been the reported outage of Amazon Web Services Elastic Compute Cloud (EC2) in just one North Virginia data center. This has affected many large website includes FourSquare, Hootsuite, Reddit and Quora. A detailed list can be found at ec2disabled.com.

For these popular websites was this avoidable? Absolutely.

Basic scalability principles if deployed in these systems architecture would have averted the significant downtime regardless of your development stack. While I work primarily in MySQL these principles are not new, nor are they complicated, however they are fundamental concepts in scalability that apply to any technology including the popular MongoDB that is being used by a number of affected sites.

Scalability 101 involves some simple basic rules. Here are just two that seem to have been ignored by many affected by this recent AWS EC2 outage.

  1. Never put all your eggs in one basket. If you rely on AWS completely, or you rely on just one availability zone that is putting all your eggs in one basket.
  2. Always keep your important data close to home. When it comes to what is most critical to your business you need access and control to your information. At 5am in the morning when the CEO asks how long will our business be unavailabla and what is needed to resolve the problem, the answer “We have no control over this and have no ETA” is not an acceptable answer.

With a successful implementation and appropriate data redundancy you may not have an environment immediately available however you have access to your important information and the ability to create one quickly. Many large hosting companies can provide additional H/W on near demand, especially if you have an initial minimal footprint. Indeed using Amazon Web Services (AWS) as a means to avert a data center disaster is an ideal implementation of Infrastructure As A Service (IAAS). Even with this issue, organizations that had planned for this type of outage could have easily migrated to another AWS availability zone that was unaffected.

Furthermore, system architecture to support various levels of data availability and scalability ensure you can handle many more various types of unavailability without significant system down time as recently seen. There are many different types of availability and unavailability, know what your definition of downtime is and supporting disasters should be your primary focus of scalability, not an after thought.

As an expert in performance and scalability I can help your organization in the design of a suitable architecture to support successful scalability and disaster. This is not rocket science however many organizations gamble without the expertise of a professional to ensure business viability.

NoSQL options

The NoSQL event in New York had a number of presentations on non relational technologies including of Hadoop, MongoDB and CouchDB.

Coming historically from a relational background of 20 years with Ingres, Oracle and MySQL I have been moving my focus towards non relational data store. The most obvious and well used today is memcached, a non persistent distributed key/value pair store. There are a number of persistent key/value stores in the marketplace, Tokyo Cabinet, Project Voldemort and Redis to name a few.

My list of data store products helps to identify the complex name space of varying products that now exist. A trend is towards schema less solutions, the ability to better manage dynamically typed/formatted information and the Agile Methodology release approach is simply non achievable in a statically type relational database table/column structure. The impact of constant ALTER TABLE commands in a MySQL database makes your production system unusable.

In a highly distribute online and increasing offline operation, fault tolerance and data synchronization and eventual consistency are required features in complex topologies such as multi-master.

I advise and promote a technology agnostic solution when possible. With the use of an API this is actually achievable, however in order to use a variety of backend data store products, one must consider the design patterns for optimal management. Two factors to support a highly distributed data set are no joins and minimal transactional semantics. The Facebook API is a great example, where there are no joins for their MySQL Relational backend. The movement back to a logical and non-normalized schema, or move towards a totally schemaless solution do require great though in the architectural concepts of your application.

Ultimately feature requirements will dictate the relative strengths and weaknesses of products. Full text search is a good example. CouchDB provides native support via Lucene. Another feature I like of couchDB is its append only data mode. This makes durability easy, and auto-recovery after crash a non issue, another feature a transactional relational database can not achieve.

With a 2 day no:sql(east) conference this month, there is definitely greater interest in this space.

Problems compiling MySQL 5.4

Seem’s the year Sun had for improving MySQL, and with an entire new 5.4 branch the development team could not fix the autoconf and compile dependencies that has been in MySQL for all the years I’ve been compiling MySQL. Drizzle has got it right, thanks to the great work of Monty Taylor.

I’m working on the Wafflegrid AWS EC2 AMI’s for Matt Yonkovit and while compiling 5.1 was straight forward under Ubuntu 8.10 Intrepid, compiling 5.4 was more complicated.

For MySQL 5.1 I needed only to do the following:

apt-get install -y build-essential
apt-get install libncurses5-dev
./configure
make
make install

For MySQL 5.4, I elected to use the BUILD scripts (based on Wafflegrid recommendations). That didn’t go far before I needed.

apt-get install -y automake libtool

You then have to go compiling MySQL 5.4 for 10+ minutes to get an abstract error, then you need to consider what dependencies may be missing.
I don’t like to do a blanket apt-get of a long list of proposed packages unless I know they are actually needed.

The error was:

make[1]: Entering directory `/src/mysql-5.4.0-beta/sql'
make[1]: warning: -jN forced in submake: disabling jobserver mode.
/bin/bash ../ylwrap sql_yacc.yy y.tab.c sql_yacc.cc y.tab.h sql_yacc.h y.output sql_yacc.output -- -d --verbose
make -j 6 gen_lex_hash
make[2]: Entering directory `/src/mysql-5.4.0-beta/sql'
rm -f mini_client_errors.c
/bin/ln -s ../libmysql/errmsg.c mini_client_errors.c
make[2]: warning: -jN forced in submake: disabling jobserver mode.
rm -f pack.c
../ylwrap: line 111: -d: command not found
/bin/ln -s ../sql-common/pack.c pack.c
....
make[1]: Leaving directory `/src/mysql-5.4.0-beta/sql'
make: *** [all-recursive] Error 1

What a lovely error ../ylwrap: line 111: -d: command not found

ylwrap is part of yacc, and by default in this instance it’s not even an installed package. I’ve compiled MySQL long enough that it requires yacc, and actually bison but to you think it would hurt if the configure told the user this.

It’s also been some time since I’ve compiled MySQL source, rather focusing on Drizzle. I had forgotten just how many compile warnings MySQL throws. Granted a warning is not an error, but you should not just ignore them in building a quality product.

Drizzle now available on Mosso

Mosso the Rackspace Cloud now has a Drizzle developer image much like the first Drizzle AMI on EC2.

The Mosso interface is definitely different, it’s a GUI, and I definitely prefer CLI, but it’s a simpler navigation for a new user. I suspect an API may be available.

I had an issue with the backup process, more the lack of feedback. The Knowledge Base didn’t help, so both calling and Live Chat directed me ultimately to the same person. I also found a bug in the backup process, that is being able to select an incomplete backup to try and launch a new server. I talked to Support about and apparently already known.

And in true open source form, the Drizzle version is actually one point higher then yesterday’s AWS image.

I don’t know how to *publish* this backup so others can try it. Something on the list of things to do, however I was able to verify my backup with a new instance.

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or g.
Your Drizzle connection id is 2
Server version: 2009.04.998 Source distribution

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

drizzle>
drizzle> select version();
+-------------+
| version()   |
+-------------+
| 2009.04.998 |
+-------------+
1 row in set (0 sec)

drizzle> select count(*) from sakila.film;
+----------+
| count(*) |
+----------+
|     1000 |
+----------+
1 row in set (0.18 sec)

Announcing Drizzle on EC2

I have published the very first sharable Drizzle Amazon Machine Image (AMI) for AWS EC2, based on the good feedback from my discussion at the Drizzle Developer Day on what options we should try.

This first version is a 32bit Developer instance, showcasing Drizzle and all necessary developer tools to build Drizzle from source.

What you will find on drizzle-ami/intrepid-dev32 – ami-b858bfd1

Ubuntu 8.10 Intrepid 32 bit base server installation:

  • build tools
  • drizzle dependencies
  • bzr 1.31.1

From the respective source trees the following software is available:

  • drizzle 2009.04.997
  • libdrizzle 0.0.2
  • gearman 0.0.4
  • memcached 1.2.8
  • libmemcached 0.28

Drizzle has been configured with necessary dependencies for PAM authentication, http_auth, libgearman and MD5 but these don’t seem to be available in the binary distribution.

I will be creating additional AMI’s including 64bit and LAMP ready binary only images.

The following example shows using drizzle on this AMI. Some further work is necessary for full automation, parameters and logging. I’ve raised a number of issues the Drizzle Developers are now hard at work on.

1. Starting Drizzle

ssh ubuntu@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com
sudo /etc/init.d/drizzle-server.init start &

2. Testing Drizzle (the sakila database has been installed)

$ drizzle
Welcome to the Drizzle client..  Commands end with ; or g.
Your Drizzle connection id is 4
Server version: 2009.04.997 Source distribution

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

drizzle> select version();
+-------------+
| version()   |
+-------------+
| 2009.04.997 |
+-------------+
1 row in set (0 sec)

drizzle> select count(*) from sakila.film;
+----------+
| count(*) |
+----------+
|     1000 |
+----------+
1 row in set (0 sec)

3. Compiling Drizzle

sudo su - drizzle
ls
deploy  drizzle  libdrizzle  sakila-drizzle
cd drizzle
./configure --help
Description of plugins:

   === HTTP Authentication Plugin ===
  Plugin Name:      auth_http
  Description:      HTTP based authentications
  Supports build:   static and dynamic

   === PAM Authenication Plugin ===
  Plugin Name:      auth_pam
  Description:      PAM based authenication.
  Supports build:   dynamic

   === compression UDFs ===
  Plugin Name:      compression
  Description:      UDF Plugin for compression
  Supports build:   static and dynamic
  Status:           mandatory

   === crc32 UDF ===
  Plugin Name:      crc32
  Description:      UDF Plugin for crc32
  Supports build:   static and dynamic
  Status:           mandatory

   === Error Message Plugin ===
  Plugin Name:      errmsg_stderr
  Description:      Errmsg Plugin that sends messages to stderr.
  Supports build:   dynamic

   === Daemon Example Plugin ===
  Plugin Name:      hello_world
  Description:      UDF Plugin for Hello World.
  Supports build:   dynamic

   === Gearman Logging Plugin ===
  Plugin Name:      logging_gearman
  Description:      Logging Plugin that logs to Gearman.
  Supports build:   dynamic

   === Query Logging Plugin ===
  Plugin Name:      logging_query
  Description:      Logging Plugin that logs all queries.
  Supports build:   static and dynamic
  Status:           mandatory

   === Syslog Logging Plugin ===
  Plugin Name:      logging_syslog
  Description:      Logging Plugin that writes to syslog.
  Supports build:   static and dynamic
  Status:           mandatory

   === MD5 UDF ===
  Plugin Name:      md5
  Description:      UDF Plugin for MD5
  Supports build:   static and dynamic

   === One Thread Per Connection Scheduler ===
  Plugin Name:      multi_thread
  Description:      plugin for multi_thread
  Supports build:   static
  Status:           mandatory

   === Old libdrizzle Protocol ===
  Plugin Name:      oldlibdrizzle
  Description:      plugin for oldlibdrizzle
  Supports build:   static
  Status:           mandatory

   === Pool of Threads Scheduler ===
  Plugin Name:      pool_of_threads
  Description:      plugin for pool_of_threads
  Supports build:   static
  Status:           mandatory

   === Default Signal Handler ===
  Plugin Name:      signal_handler
  Description:      plugin for signal_handler
  Supports build:   static
  Status:           mandatory

   === Single Thread Scheduler ===
  Plugin Name:      single_thread
  Description:      plugin for single_thread
  Supports build:   static
  Status:           mandatory

   === Archive Storage Engine ===
  Plugin Name:      archive
  Description:      Archive Storage Engine
  Supports build:   static
  Status:           mandatory

   === Blackhole Storage Engine ===
  Plugin Name:      blackhole
  Description:      Basic Write-only Read-never tables
  Supports build:   static and dynamic
  Configurations:   max, max-no-ndb

   === CSV Storage Engine ===
  Plugin Name:      csv
  Description:      Stores tables in text CSV format
  Supports build:   static
  Status:           mandatory

   === Memory Storage Engine ===
  Plugin Name:      heap
  Description:      Volatile memory based tables
  Supports build:   static
  Status:           mandatory

   === InnoDB Storage Engine ===
  Plugin Name:      innobase
  Description:      Transactional Tables using InnoDB
  Supports build:   static and dynamic
  Configurations:   max, max-no-ndb
  Status:           mandatory

   === MyISAM Storage Engine ===
  Plugin Name:      myisam
  Description:      Traditional non-transactional MySQL tables
  Supports build:   static
  Status:           mandatory


Report bugs to <http://bugs.launchpad.net/drizzle>.

Setting up MySQL on Amazon Web Services (AWS) Presentation

On Tuesday at the MySQL Camp 2009 in Santa Clara I presented Setting up MySQL on Amazon Web Services (AWS).

This presentation assumed you know nothing about AWS, and have no account. With Internet access via a Browser and a valid Credit Card, you can have your own running Web Server on the Internet in under 10 minutes, just point and click.

We also step into some more detail online click and point and supplied command line tools to demonstrate some more advanced usage.

Your Code, Your Community, Your Cloud… Project Kenai

Following the opening keynote announcement about Kenai I ventured into a talk on Project Kenai.

With today’s economy, the drive is towards efficiency is certainly a key consideration, it was quoted that dedicated hosting servers only run at 30% efficiency.

An overview again of Cloud Computing

  • Economics – Pay as you go,
  • Developer Centric – rapid self provisioning, api-driven, faster deployment
  • Flexibility – standard services, elastic, on demand, multi-tenant

Types of Clouds

  • Public – pay as you go, multi-tenant application and services
  • Private – Cloud computing model run within a company’s own data center
  • Mixed – Mixed user of public and private clouds according to applications

SmugMug was referenced as a Mixed Cloud example.

Cloud Layers

  • Infrastructure as a Services – Basic storage and computer capabilities offer as a service (eg. AWS)
  • Platform as a Service – Developer platform with build-in services. e.g. Google App Engine
  • Software as Service – applications offered on demand over the network e.g salesforce.com

Some issues raised about this layers included.

  • IaaS issues include Service Level, Privacy, Security, Cost of Exit
  • PaaS interesting point, one that is the bane of MySQL performance tuning, that is instrumentation
  • SaaS nothing you need to download, you take the pieces you need, interact with the cloud. More services simply like doing your Tax online.

Sun offers Project Kenai as well as Zembly.

Project Kenai

  • A platform and ecosystem for developers.
  • Freely host open source projects and code.
  • Connect, community, collaborate and Code with peers
  • Eventually easily deploy application/services to “clouds”

Kenai Features

  • Code Repository with SVN, Mercurial, or an external repository
  • Issue tracking with bugzilla, jira
  • collaboration tools such as wiki, forums, mailing lists
  • document hosting
  • your profile
  • administrative role

Within Kenai you can open up to 5 open source projects and various metrics of the respositories, issue trackers, wiki etc.

The benefits were given as the features are integrated into your project, not distributed across different sites. Agile development within the project sees a release every 2 weeks. Integration with NetBeans and Eclipse is underway.

Kenai is targeted as being the core of the next generation of Sun’s collaboration tools. However when I asked for more details about uptake in Sun, it’s only a request, not a requirement for internal teams.

The API’s for the Sun Cloud are at http://kenai.com/projects/suncloudapis.

Event: CommunityOne East in New York, NY.
Presenter: Tori Wieldt, Sun Microsystems
Article Author: Ronald Bradford

Everybody is talking About Clouds

From the opening keynote at CommunityOne East we begin with Everybody is talking About Clouds.

It’s difficult to get a good definition, the opening cloud definition today was Software/Platform/Storage/Database/Infrastructure as a service. Grid Computing, Visualization, Utility Computing, Application Hosting. Basically all the buzz words we currently know.

Cloud computing has the ideals of truly bringing a freedom of choice. For inside or outside of an enterprise, the lower the barrier, time and cost into freedom of choice give opportunities including:

  • Self-service provisioning
  • Scale up, Scale down.
  • Pay for only what you use.

Sun’s Vision has existed since 1984 with “The NETWORK is the Computer”.

Today, Sun’s View includes Many Clouds, Public and Private, Tuned up for different application needs, geographical, political, with a goal of being Open and Compatible.

How do we think into the future for developing and deploying into the cloud? The answer given today was, The Sun Open Cloud Platform which includes the set of core technologies, API’s and protocols that Sun hopes to see uptake among many different providers.

The Sun Cloud Platform

  • Products and Technologies – VirtualBox, Sun xVM, Q-Laser, MySQL
  • Expertise and Services
  • Partners – Zmanda, Rightscale, Kickapps
  • Open Communities – Glashfish, Java, Open Office, Zfs, Netbeans, Eucalyptus

The Sun Cloud includes:

  • Compute Service
  • Storage Service
  • Virtual Data Center
  • Open API – Public, RESTful, Java, Python, Ruby

The public API has been released today and is available under Kenai. It includes two key points:

  • Everything is a resource http GET, POST, PUT etc
  • A single starting point, other URI’s are discoverable.

What was initially showed was CLI interface exmaples, great to see this still is common, a demonstration using drag and drop via a web interface was also given, showing a load balanced, multi-teired, multi server environment. This was started and tested during the presentation.

Then Using Cyberduck (a WebDAV client on Mac OS/X) and being able to access the storage component at storage.network.com directly, then from Open Office you now get options to Get/Save to Cloud ( using TwoGuys.com, Virtual Data Center example document).

Seamless integration between the tools, and the service. That was impressive.

More information at sun.com/cloud. You can get more details also at the Sun Microsystems Unveils Open Cloud PlatformOfficial Press Release.

Event: CommunityOne East in New York, NY.
Article Author: Ronald Bradford

Extending application data to the cloud

I was one of the invited panel speakers to A panel on Cloud Computing this week in New York. As one of 2 non vendor presenters, it was a great experience to be invited and be involved with vendors.

While I never got to use my slides available here, I did get to both present certain content, and indeed questions and discussions on the night were on other points of my content.

Cloud computing is here, it’s early days and new players will continue to emerge. For example, from the panel there was AppNexus, reviewed favorably at Info World in comparison with EC2 and Google App Engine, 10gen, an open source stack solution and Kaavo which from an initial 60 seconds of playing provide a management service on top of AWS similar to what ElasticFox provides. I need to investigate further how much the feature set extends and would compete with others like RightScale for example.

The greatest mystery came from Hank Williams and his stealth Kloudshare. He did elaborate more on where they aim to provide services. A new term discussed was “Tools as a service”, akin to moving use metaphorically from writing in Assembly language to the advanced frameworks of today’s generation of languages such as Java and Ruby.

Thanks to Murat Aktihanoglu of Unype who chaired the event.

Your data and the cloud

I will be speaking on July 29th in New York at an Entrepreneurs Forum on A Free Panel on Cloud Computing. With a number of experts including Hank Williams of KloudShare, Mike Nolet of AppNexus, and Hans Zaunere of New York PHP fame is should be a great event.

The focus of my presentation will be on “Extending existing applications to leverage the cloud” where I will be discussing both the advantages of the cloud, and the complexities and issues that you will encounter such as data management, data consistency, loss of control, security and latency for example.

Using traditional MySQL based applications I’ll be providing an approach that can lead to your application gaining greater power of cloud computing.


About the Author

Ronald Bradford provides Consulting and Advisory Services in Data Architecture, Performance and Scalability for MySQL Solutions. An IT industry professional for two decades with extensive database experience in MySQL, Oracle and Ingres his expertise covers data architecture, software development, migration, performance analysis and production system implementations. His knowledge from 10 years of consulting across many industry sectors, technologies and countries has provided unique insight into being able to provide solutions to problems. For more information Contact Ronald.

Setting up on EC2

Thanks to my friend Dustin, and his EC2 demo using Elasticfox Firefox Extension for Amazon EC2 I got an EC2 image setup. With other references Link 1,Link 2,Link 3 I was also able to create my own AMI.

Some notes specific for my configuration.

Pre-config ElasticFox key for launching directly from ElasticFox SSH connections.

mkdir ~/ec2-keys
mv ~/Downloads/elasticfox.pem ~/ec2-keys/id_elasticfox
chmod 600 ~/ec2-keys/id_elasticfox
chmod 700 ~/ec2-keys/
ssh -i /Users/rbradfor/ec2-keys/id_elasticfox root@ec2-99-999-999-999.compute-1.amazonaws.com

Installed Software.

apt-get update
apt-get -y autoremove
apt-get -y install apache2
apt-get -y install mysql-server
# Prompts for password (very annoying)
apt-get -y install php5
apache2ctl graceful
echo "Hello World" > /var/www/index.html
echo "< ? phpinfo() ?>" > /var/www/phpinfo.php

Configuration to save AMI.

scp -i ~/ec2-keys/id_elasticfox ~/ec2-keys/id_elasticfox pk-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem cert-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem root@ec2-99-999-999-999.compute-1.amazonaws.com:/mnt
ec2-bundle-vol -d /mnt -c cert-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem -k pk-CHK7DP4475BWUKIUF4WFDIW3VMYDYOHQ.pem -u AccountNumber -r i386 -p ubuntu804_lamp
ec2-upload-bundle -b rbradford_804_lamp_ami -m /mnt/ubuntu804_lamp.manifest.xml -a AccessID -s SecretKey

Working with Google App Engine

Yesterday I took a more serious look at Google App Engine, I got a developer account some weeks ago.

After going though the getting started demo some time ago, I chose an idea for a FaceBook Application and started in true eXtreme Programming (XP) style (i.e. What’s the bare minimum required for first iteration). I taught myself some Python and within just a few minutes had some working data being randomly generated totally within the development SDK environment On my MacBook. I was not able to deploy initially via the big blue deploy button, the catch is you have to register the application manually online.

Then it all worked, and hey presto I’ve got my application up at provided domain hosting at appspot.com

Having coming from a truly relational environment, most notably MySQL of recent years I found the Datastore API different in a number of ways.

  • There is no means of Sequences/Auto Increment. There is an internal Unique Key, but it’s a String, not an integer, not enabling me to re-use it.
  • The ListProperty enables the use of Lists in Python (like Arrays) to be easily stored.
  • The ReferenceProperty is used as a foreign key relationship, and then can be more reference within an object hierarchy
  • I really missed an interactive interface. You have no abililty to look at your data, specifically for me I wanted to seek some data, then I wanted to delete some data, but I had to do all this via code.

Having developed a skelaton FaceBook application before in PHP, I figured a Python version would not be that much more work, but here is where I good stumped Information at Hosting a Facebook Application on Google AppEngine leveraging the PyFacebook project didn’t enable me to integrate Google App Engine with FaceBook just yet.

This had me thinking I need to resort to a standalone simply Python Facebook application to confirm the PyFacebook usage. Now my problems started. Under Mac it’s a lot more complex to install and configure Python/Django etc then under Linux. I tried to do it on my dedicated server, but drat Python is at 2.3.4, and it seems 2.5.x is needed.

Still it was a valuable exercise, I dropped the FaceBook goal and just worked on more Google App Engine stuff. Still early days, but it was productive to try out this new technology.

What I need to work on now is how to hold state within Python infrastructure so I can manage a user login and storing and retrieving user data for my sample app.