Using your devstack cloud

You have setup and installed devstack. Now what!

The Horizon UI will allow you to administer your running cloud from a web interface. We are not going to discuss the web UI in this post.

Using the command line will provide you access to the following initial developer/operator capabilities.

  • Duplicating the features of the UI with the client tools
  • Observing the running services
  • Understanding the logging of OpenStack services
  • Understanding the configuration of OpenStack services
  • Understanding the source code of OpenStack services

This is not an exhaustive list or explanation of each point but an intro into navigating around the running OpenStack services.

Duplicating UI features

OpenStack has a number of individual command line clients for many services, and a common client openstack.

To get started:

$ openstack user list
Missing parameter(s): 
Set a username with --os-username, OS_USERNAME, or auth.username
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url
Set a scope, such as a project or domain, set a project scope with --os-project-name, OS_PROJECT_NAME or auth.project_name, set a domain scope with --os-domain-name, OS_DOMAIN_NAME or auth.domain_name

By default you will need to provide applicable authentication details via arguments or environment variables.
Using the output of the devstack setup, we can obtain applicable details needed for most parameters.

$ ./
This is your host IP address:
This is your host IPv6 address: ::1
Horizon is now available at
Keystone is serving at
The default users are: admin and demo
The password: passwd

We can now retrieve a summary list of users defined in your project with:

$ openstack --os-username=admin --os-password=passwd --os-auth-url= --os-project-name=demo user list
| ID                               | Name     |
| a531ea1011af43bb8277f3e5edfea34b | admin    |
| d6ce303e83b64a2998228c55ebd274c3 | demo     |
| fe7301aa4d2b44b482cd6ba19c24f6b8 | alt_demo |
| e18ae48148df4593b4067785c5e72820 | nova     |
| 9a49deabb7b64454abf411de87c2862c | glance   |
| 1315257f265740f8a32988b014c9e693 | cinder   |

One parameter that is required but no information was available in the devstack installation output was project. There are a number of projects defined in the installation which you can obtain with:

$ openstack --os-username=admin --os-password=passwd --os-auth-url= --os-project-name=admin project list
| ID                               | Name               |
| 3b9f48af38ac40a495ca7b22d4d5c036 | demo               |
| 42c574962a114974bfe35e4a3467df60 | service            |
| 7af69c571e764d5f99688ed2e59930d5 | alt_demo           |
| 893b8954952c4319abd6596b587bba5f | admin              |
| da71fdc9c88f4eddac38937dfef542a2 | invisible_to_admin |

By defining authentication with environment variables you can easily simply CLI command usage. For example:

$ export OS_USERNAME=admin
$ export OS_PASSWORD=passwd
$ export OS_AUTH_URL=
$ export OS_PROJECT_NAME=demo
$ openstack user list

devstack pre-packages a few source files that enable you to avoid specifying these arguments or environment variables manually. For example to duplicate this example:

$ source accrc/admin/demo
$ openstack user list

The openstack command provides a --help option to list the available options. You can also inquire as to commands with the command list option.

$ openstack --help
$ openstack command list

With the openstack command line interface you can perform all the operations needed to configure, administer and run your cloud services.

Observing the running services

OpenStack is made up of a number of services, those key services in devstack start with nova, keystone, glance, cinder and horizon. devstack conveniently packages the individual running services into separate screen processes, leveraging a cursors based view of services via the output of log files.

You can view the running screen sessions by reattaching with.

$ screen -r

If you get the following error when attempting to reattach “Cannot open your terminal ‘/dev/pts/0′ – please check.”, you have likely tried reconnecting in a different shell session. You can address this with:

$ script /dev/null
$ screen -r

Commands in screen are driven by a key combination starting with ^a (ctrl-A). ^a d will detach from your screen session you just reattached to. This is what gets you out of screen. See the later section for the full list screen help commands.

On the command line you can run the following command to list the available images via the glance service.

$ openstack image list
| ID                                   | Name                            | Status |
| 864bad45-d0de-4031-aea6-80b6af72cf2a | cirros-0.3.4-x86_64-uec         | active |
| 75e8b1ef-ae84-41aa-b0a0-7ea785771f14 | cirros-0.3.4-x86_64-uec-ramdisk | active |
| f694bdb1-4bb0-4f18-a7c9-290ad26b1fc8 | cirros-0.3.4-x86_64-uec-kernel  | active |

Within screen you can look at the glance api screen log (^a 5) and can observe the logging that occurs in relation to this command. For example we can see an INFO message to get the images (GET /v2/images), and we can see several DEBUG messages. We will use these DEBUG messages in a later post to describe handling logging output.

The INFO message will look like:

2016-04-04 16:24:00.139 INFO eventlet.wsgi.server [req-acf98429-60de-4d18-a69c-36a7d80bed7c a531ea1011af43bb8277f3e5edfea34b 3b9f48af38ac40a495ca7b22d4d5c036] - - [04/Apr/2016 16:24:00] "GET /v2/images HTTP/1.1" 200 2202 0.116774

While we will discuss logging formats in another post, the standard format (in devstack) includes:

  • Date/Time
  • Logging Level
  • Package
  • Request context. this is made up of
    • req-acf98429-60de-4d18-a69c-36a7d80bed7c a request-id, useful for grouping logging records
    • a531ea1011af43bb8277f3e5edfea34b refers to the user id (as seen in user list above, i.e. admin)
    • 3b9f48af38ac40a495ca7b22d4d5c036 refers to the project id (as seen in the project list above, i.e. demo)
  • The actual log message
In order to page back in screen output, you enter copy mode “^a [” and then you can use ^b (page back) and ^f (page forward) keys.

Understanding the logging of OpenStack services

What is actually observed in the screen output is what is being logged for the Glance API service. We can verify this with the log file logged in /opt/stack/logs.

$ tail -f /opt/stack/logs/g-api.log

NOTE: You may see that there are colors within both the screen and log output. This is an optional configuration setup used by devstack (not an OpenStack default for logging). We will use this later to show a change in the logging of the service.

We can verify the details of the command used within the screen session (^a 5) by killing the running process with ^c.

Using the bash history, you can up arrow to observe the last running command, and restart this.

/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf & echo $! >/opt/stack/status/stack/; fg || echo "g-api failed to start" | tee "/opt/stack/status/stack/g-api.failure"

The actual log file is produced by the screen configuration defined in devstack/stack-screenrc.

screen -t g-api bash
"tuff "/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf
logfile /opt/stack/logs/g-api.log.2016-04-04-110956
log on

In a running OpenStack environment you would configure logging output to file as per the log_file option.

Understanding the configuration of OpenStack services

This command indicated a configuration file /etc/glance/glance-api.conf. Glance like other services may contain several configuration files. These are by default defined in the individual projects namespace under /etc.

$ ls -l /etc/glance/
total 152
-rw-r--r-- 1 stack stack 65106 Apr  4 11:12 glance-api.conf
-rw-r--r-- 1 stack stack  3266 Mar 11 12:22 glance-api-paste.ini
-rw-r--r-- 1 stack stack 13665 Apr  4 11:12 glance-cache.conf
-rw-r--r-- 1 stack stack 51098 Apr  4 11:12 glance-registry.conf
-rw-r--r-- 1 stack stack  1233 Mar 11 12:22 glance-registry-paste.ini
drwxr-xr-x 2 stack root   4096 Apr  4 11:12 metadefs
-rw-r--r-- 1 stack stack  1351 Mar 11 12:22 policy.json
-rw-r--r-- 1 stack stack  1380 Mar 11 12:22 schema-image.json

This is an appropriate time to point to several documentation sources including the Glance Developer Documentation – Configuration Options and the Configuration Guide Image Service options which describe in more detail these listed configuration files and the possible options available. You can find similar documentation for other services.

To demonstrate just how the configuration and logging work with a running service the following will modify the logging of the Glance API service by commenting out the logging configuration lines, and then reverting to the oslo.log configuration defaults.

$ sudo vi /etc/glance/glance-api.conf

Comment out the four logging_ options in the [DEFAULT] section.

#logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s ^[[01;35m%(instance)s^[[00m
#logging_debug_format_suffix = ^[[00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d^[[00m
#logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[00;36m-%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
#logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[01;36m%(request_id)s ^[[00;36m%(user)s %(tenant)s%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m

Now, repeating the earlier steps within the g-api screen window, kill and restart the service.
The first thing you will observe is that the logging no longer contains color (this helps greatly for log file analysis). Repeat the CLI option to list the images, and you will notice a slightly modified logging message occur.

2016-04-05 11:38:57.312 17696 INFO eventlet.wsgi.server [req-1e66b7e5-3429-452e-a9b7-e28ee498f772 a531ea1011af43bb8277f3e5edfea34b 3b9f48af38ac40a495ca7b22d4d5c036 - - -] - - [05/Apr/2016 11:38:57] "GET /v2/images HTTP/1.1" 200 2202 11.551233

The request context now is a modified format (containing additional - - - values) as a result of using the default value of logging_context_format_string. We will discuss the specifics of logging options in a later post.

There are a reasonable number of log files for a minimal devstack installation, some services have multiple log files.

$ cd /opt/stack/logs; ls -l *.log
lrwxrwxrwx 1 stack stack       27 Apr  5 12:49 c-api.log -> c-api.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:49 c-sch.log -> c-sch.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:49 c-vol.log -> c-vol.log.2016-04-05-124004
-rw-r--r-- 1 stack stack 16672591 Apr  5 14:01 dstat-csv.log
lrwxrwxrwx 1 stack stack       27 Apr  5 12:42 dstat.log -> dstat.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:48 g-api.log -> g-api.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:48 g-reg.log -> g-reg.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       29 Apr  5 12:50 horizon.log -> horizon.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       32 Apr  5 12:42 key-access.log -> key-access.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       25 Apr  5 12:42 key.log -> key.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:48 n-api.log -> n-api.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       29 Apr  5 12:49 n-cauth.log -> n-cauth.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       28 Apr  5 12:48 n-cond.log -> n-cond.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:49 n-cpu.log -> n-cpu.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:48 n-crt.log -> n-crt.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       28 Apr  5 12:42 n-dhcp.log -> n-dhcp.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:48 n-net.log -> n-net.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       29 Apr  5 12:49 n-novnc.log -> n-novnc.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       27 Apr  5 12:49 n-sch.log -> n-sch.log.2016-04-05-124004
lrwxrwxrwx 1 stack stack       46 Apr  5 12:40 -> /opt/stack/logs/

To turn off color in logging across service, you can configure this in the devstack local.conf file before starting the stack.

# local.conf

Understanding the source code of OpenStack services

devstack installs the OpenStack code in two ways, via packaging and via source.

Generally all libraries are installed via packaging. You can discern these via looking at the python packages via pip with:

$ pip freeze

This is a list of all Python packages so it’s not possible to determine which are OpenStack specific, and which are dependencies. These installed packages are actually Python source that you can view and even modify.

$ ls -l /usr/local/lib/python2.7/dist-packages/

You can approximate the installed OpenStack packages via source by looking at the base source directory:

$ ls -l /opt/stack
total 92
drwxr-xr-x 10 stack stack 4096 Mar 11 12:23 cinder
drwxr-xr-x  6 stack root  4096 Apr  5 12:42 data
-rw-r--r--  1 stack stack  440 Apr  5 12:52 devstack.subunit
drwxr-xr-x  4 stack stack 4096 Mar 11 12:27 dib-utils
drwxr-xr-x 10 stack stack 4096 Mar 11 12:22 glance
drwxr-xr-x 15 stack stack 4096 Mar 11 12:26 heat
drwxr-xr-x  7 stack stack 4096 Mar 11 12:27 heat-cfntools
drwxr-xr-x 10 stack stack 4096 Mar 11 12:27 heat-templates
drwxr-xr-x 11 stack stack 4096 Mar 11 14:13 horizon
drwxr-xr-x 13 stack stack 4096 Mar 11 11:57 keystone
drwxr-xr-x  2 stack stack 4096 Apr  5 12:50 logs
drwxr-xr-x 12 stack stack 4096 Mar 11 15:45 neutron
drwxr-xr-x 13 stack stack 4096 Mar 11 12:25 nova
drwxr-xr-x  8 stack stack 4096 Mar 11 12:24 noVNC
drwxr-xr-x  4 stack stack 4096 Mar 11 12:27 os-apply-config
drwxr-xr-x  4 stack stack 4096 Mar 11 12:27 os-collect-config
drwxr-xr-x  5 stack stack 4096 Mar 11 12:27 os-refresh-config
drwxr-xr-x  7 stack stack 4096 Apr  5 12:51 requirements
drwxr-xr-x 13 stack stack 4096 Mar 11 15:47 solum
drwxr-xr-x  3 stack stack 4096 Apr  4 11:13 status
drwxr-xr-x 10 stack stack 4096 Mar 11 12:22 swift

devstack enables you to configure which packages you want to install via source. Checkout plugins for more information. For example, the following added to the local.conf would install solum.

# local.conf
enable_plugin solum git://

You have complete flexibility of which branch and version of each package using devstack. This enables you to use devstack as a testing tool for code changes.

At this time to understand more about how software is installed check out devstack documentation and review the script.

What’s next

This is only a cursory introduction into what devstack sets up during the installation process. Subsequent posts will talk more on topics including the configuration options, the different logging capabilities and how to test code changes.

screen help

^a ? will provide the following help output.

                                                                                     Screen key bindings, page 1 of 2.

                                                                                     Command key:  ^A   Literal ^A:  a

  break       ^B b         dumptermcap .            info        i            meta        a            pow_detach  D            reset       Z            title       A            xoff        ^S s      
  clear       C            fit         F            kill        K k          monitor     M            prev        ^H ^P p ^?   screen      ^C c         vbell       ^G           xon         ^Q q      
  colon       :            flow        ^F f         lastmsg     ^M m         next        ^@ ^N sp n   quit        \            select      '            version     v         
  copy        ^[ [         focus       ^I           license     ,            number      N            readbuf     <            silence     _            width       W         
  detach      ^D d         hardcopy    h            lockscreen  ^X x         only        Q            redisplay   ^L l         split       S            windows     ^W w      
  digraph     ^V           help        ?            log         H            other       ^A           remove      X            suspend     ^Z z         wrap        ^R r      
  displays    *            history     { }          login       L            pow_break   B            removebuf   =            time        ^T t         writebuf    >         

^]   paste .
"    windowlist -b
-    select -
0    select 0
1    select 1
2    select 2
3    select 3
4    select 4
5    select 5
6    select 6
7    select 7
8    select 8
9    select 9
I    login on
O    login off
]    paste .
|    split -v
:kB: focus prev

Running a devstack virtual machine with limited memory

If you have a system with only 4GB of RAM, you need to assign at least 2.5GB (2560M) to a virtual machine to install devstack. Even with this limited RAM there are times the devstack installation will fail.

One way to give the installation process an opportunity to complete is to configure your virtual machine to have swap space. The amount of swap space you can configure may be limited to the size of your initial disk partition configuration (which is 8GB). The following steps add a 2GB swap file to your virtual machine.

sudo swapon -s
free -m
sudo fallocate -l 2G /swapfile
ls -lh /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon -s
free -m
echo "/swapfile   none    swap    sw    0   0" | sudo tee -a /etc/fstab
cat /etc/fstab
The use of swap space by your virtual machine instead of available RAM will cause a significant slowdown of any software. For the purposes of a minimal installation this option provides a means to observe a running minimal OpenStack cloud.

Downloading and installing devstack

The following instructions assume you have a running Linux virtual machine that can support the installation of devstack to demonstrate a simple working OpenStack cloud.

For more information about the preparation needed for this step, see these pre-requisite instructions:


You will need to login to your Linux virtual machine as a normal user (e.g. stack if you followed these instructions).

To verify the IP address of your machine you can run:

$ ifconfig eth1

NOTE: This assumes you configured a second network adapter as detailed.

You need to determine the IP address assigned. If this is your first-time using VirtualBox and this was configured with default settings, the value will be

eth1      Link encap:Ethernet  HWaddr 08:00:27:db:42:6e  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::a00:27ff:fedb:426e/64 Scope:Link
          RX packets:398500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:282829 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:35975184 (35.9 MB)  TX bytes:59304714 (59.3 MB)

Verify that you have applicable sudo privileges.

$ sudo id

If you are prompted for a password, then your privileges are not configured correctly. See here.

Download devstack

After connecting to the virtual machine the following commands will download the devstack source code:

$ sudo apt-get install -y git-core
# NOTE: You will not be prompted for a password
#       This is important for the following installation steps
$ git clone

Configure devstack

The following will create an example configuration file suitable for a default devstack installation.

$ cd devstack
# Use the sample default configuration file
$ cp samples/local.conf .
$ HOST_IP=""
$ echo "HOST_IP=${HOST_IP}" >> local.conf

NOTE: If your machine has different IP address you should specify this alternative value.

Install devstack

$ ./

Depending on your physical hardware and network connection, this takes approximately 20 minutes.

When completed you will see the following:

This is your host IP address:
This is your host IPv6 address: ::1
Horizon is now available at
Keystone is serving at
The default users are: admin and demo
The password: nomoresecrete
While the installation of devstack is happening, you should read Configuration section, and look at the devstack/samples/local.conf sample configuration file being used.

Accessing devstack

You now have a running OpenStack cloud. There are two easy ways to access the running services to verify.

  • Connect the Horizon dashboard in your browser with the URL (e.g., and use the user and password described (e.g. admin and nomoresecrete).
  • Use the OpenStack client that is installed with devstack, for example:
$ source accrc/admin/admin
$ openstack image list

See Using your devstack cloud for more information about analyzing your running cloud, restarting services, configuration files and how to demonstrate a code change.

Other devstack commands

There are some useful commands to know about with your devstack setup.

If you restart your virtual machine, you reconnect to devstack by re-running the installation (there is no longer a

$ ./

To shutdown a running devstack.

$ ./

To cleanup your VM of devstack installed software.

$ ./

Setting up Ubuntu using vagrant

As discussed in Setting up an Ubuntu virtual machine using VirtualBox there are several other alternatives to defining an Ubuntu virtual machine. One of these alternatives is using Vagrant.


Vagrant requires the installation of VirtualBox.

Install Vagrant

See Vagrant Downloads for the correct file for your platform.

For Ubuntu, the following commands will download a recent copy and install on your computer.

$ wget
$ sudo dpkg -i vagrant_1.8.1_x86_64.deb

Launching an Ubuntu image

The following commands will initialize an start an Ubuntu 14.04 vagrant instance.

$ vagrant init ubuntu/trusty64
$ vagrant up --provider virtualbox
$ vagrant ssh

You should now be connected to the new virtual machine.

Vagrant creates a port forwarding configuration from your local machine automatically. You can connect via ssh directly with:

ssh vagrant@localhost -p 2222 -i .vagrant/machines/default/virtualbox/private_key

NOTE: Port 2222 may be different if this is already in use. You can verify this via the output of the vagrant up command, for example:

==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)

Post configuration

In order to access your vagrant instance with a specific IP address and leverage the recommended devstack instructions you need to add the line to the Vagrantfile in the directory used on your host computer. You also need to set the virtual machine memory to at least 2.5GB to get a minimal devstack operational.

Vagrant.configure(2) do |config| = "ubuntu/trusty64" "private_network", type: "dhcp"
  config.vm.provider "virtualbox" do |v|
    v.memory = 2560

You will then need to restart the vagrant image in order to have a host-only IP assigned to the virtual machine and applicable memory.

$ vagrant reload
$ vagrant ssh
$ ifconfig eth1
$ free -m

This has created a suitable virtual machine ready for Downloading and installing devstack.

Setting up CentOS on VirtualBox for RDO

Create a CentOS Virtual Machine (VM)

NOTE: There are several different ways in creating a base VM CentOS image. These steps are the more manual approach, however they are provided for completeness in understanding varying options.

To create a virtual machine in VirtualBox select the New icon. This will prompt you for some initial configuration. Use these recommendations:

  • Name and operating System
    • Name: RDO
    • Type: Linux
    • Version: Red Hat (64-bit)
  • Memory Size
    • Use at minimum 4GB.
  • Hard Disk
    • Use the default settings including 8.0GB, VDI type, dynamically allocated, File location and size.

By default your virtual machine is ready to install however by making the following network recommendation it will be easier to access your running virtual machine via SSH and the RDO web interface and APIs from your host computer.

  • Click Settings
  • Select Network
  • Enable Adapter 2 and attach to a Host-only Adapter and select vboxnet0
  • Ok

Install CentOS Operating System

You are now ready to install the Operating System on the virtual machine with the following instructions.

  • Click Start
  • Open the CentOS .iso file you just downloaded.
  • You will be prompted for a number of options, select the default provided and use the following values when prompted.
  • Install CentOS 7
  • Select English and English (United States) (or your choice of language)
  • Select System to configure your installation destination
    • Click Done to use the default VM disk and automatically configure partitioning
  • Select Network & hostname
    • Enable both of the listed Ethernet connections
    • Enter rdo for the Host Name
    • Click Done
  • Click Begin Installation
  • Click Root Password
    • Enter password of your choosing
  • Click User Creation
    • Enter rdo for user name (or any value of your choice)
    • Enter Openstack for password (or any password of your choice)
    • Click Done

When the installation is complete, click Reboot.

You will now be able to login with username: rdo and password: Openstack (or the values you chose).

Post Installation

While the second ethernet adapter for your VM is configured it is not enabled.

$ su -
# Enter root password
$ sed -ie "s/ONBOOT=no/ONBOOT=yes/" /etc/sysconfig/network-scripts/ifcfg-enp0s8
$ ifup enp0s8
$ ip addr
# RDO does not operate with NetworkManager
$ sudo systemctl stop NetworkManager.service
$ sudo systemctl disable NetworkManager.service

The ip output will verify the IP address that was assigned. If you configured the VirtualBox host-only adapter with defaults, the address will be

To verify access to your virtual machine from your host computer, you should SSH with:

$ ssh [email protected]