You have setup and installed devstack. Now what!
The Horizon UI will allow you to administer your running cloud from a web interface. We are not going to discuss the web UI in this post.
Using the command line will provide you access to the following initial developer/operator capabilities.
- Duplicating the features of the UI with the client tools
- Observing the running services
- Understanding the logging of OpenStack services
- Understanding the configuration of OpenStack services
- Understanding the source code of OpenStack services
This is not an exhaustive list or explanation of each point but an intro into navigating around the running OpenStack services.
Duplicating UI features
OpenStack has a number of individual command line clients for many services, and a common client openstack
.
To get started:
$ openstack user list Missing parameter(s): Set a username with --os-username, OS_USERNAME, or auth.username Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url Set a scope, such as a project or domain, set a project scope with --os-project-name, OS_PROJECT_NAME or auth.project_name, set a domain scope with --os-domain-name, OS_DOMAIN_NAME or auth.domain_name
By default you will need to provide applicable authentication details via arguments or environment variables.
Using the output of the devstack setup, we can obtain applicable details needed for most parameters.
$ ./stack.sh ... ... ... This is your host IP address: 192.168.56.101 This is your host IPv6 address: ::1 Horizon is now available at http://192.168.56.101/dashboard Keystone is serving at http://192.168.56.101:5000/ The default users are: admin and demo The password: passwd
We can now retrieve a summary list of users defined in your project with:
$ openstack --os-username=admin --os-password=passwd --os-auth-url=http://192.168.56.101:5000/ --os-project-name=demo user list +----------------------------------+----------+ | ID | Name | +----------------------------------+----------+ | a531ea1011af43bb8277f3e5edfea34b | admin | | d6ce303e83b64a2998228c55ebd274c3 | demo | | fe7301aa4d2b44b482cd6ba19c24f6b8 | alt_demo | | e18ae48148df4593b4067785c5e72820 | nova | | 9a49deabb7b64454abf411de87c2862c | glance | | 1315257f265740f8a32988b014c9e693 | cinder | +----------------------------------+----------+
One parameter that is required but no information was available in the devstack installation output was project. There are a number of projects defined in the installation which you can obtain with:
$ openstack --os-username=admin --os-password=passwd --os-auth-url=http://192.168.56.101:5000/ --os-project-name=admin project list +----------------------------------+--------------------+ | ID | Name | +----------------------------------+--------------------+ | 3b9f48af38ac40a495ca7b22d4d5c036 | demo | | 42c574962a114974bfe35e4a3467df60 | service | | 7af69c571e764d5f99688ed2e59930d5 | alt_demo | | 893b8954952c4319abd6596b587bba5f | admin | | da71fdc9c88f4eddac38937dfef542a2 | invisible_to_admin | +----------------------------------+--------------------+
By defining authentication with environment variables you can easily simply CLI command usage. For example:
$ export OS_USERNAME=admin $ export OS_PASSWORD=passwd $ export OS_AUTH_URL=http://192.168.56.101:5000/ $ export OS_PROJECT_NAME=demo $ openstack user list ...
devstack pre-packages a few source files that enable you to avoid specifying these arguments or environment variables manually. For example to duplicate this example:
$ source accrc/admin/demo $ openstack user list
The openstack
command provides a --help
option to list the available options. You can also inquire as to commands with the command list
option.
$ openstack --help $ openstack command list
With the openstack
command line interface you can perform all the operations needed to configure, administer and run your cloud services.
Observing the running services
OpenStack is made up of a number of services, those key services in devstack start with nova, keystone, glance, cinder and horizon. devstack conveniently packages the individual running services into separate screen processes, leveraging a cursors based view of services via the output of log files.
You can view the running screen sessions by reattaching with.
$ screen -r
If you get the following error when attempting to reattach “Cannot open your terminal ‘/dev/pts/0′ – please check.”, you have likely tried reconnecting in a different shell session. You can address this with:
$ script /dev/null $ screen -r
Commands in screen are driven by a key combination starting with ^a (ctrl-A). ^a d
will detach from your screen session you just reattached to. This is what gets you out of screen. See the later section for the full list screen help commands.
On the command line you can run the following command to list the available images via the glance service.
$ openstack image list +--------------------------------------+---------------------------------+--------+ | ID | Name | Status | +--------------------------------------+---------------------------------+--------+ | 864bad45-d0de-4031-aea6-80b6af72cf2a | cirros-0.3.4-x86_64-uec | active | | 75e8b1ef-ae84-41aa-b0a0-7ea785771f14 | cirros-0.3.4-x86_64-uec-ramdisk | active | | f694bdb1-4bb0-4f18-a7c9-290ad26b1fc8 | cirros-0.3.4-x86_64-uec-kernel | active | +--------------------------------------+---------------------------------+--------+
Within screen you can look at the glance api screen log (^a 5) and can observe the logging that occurs in relation to this command. For example we can see an INFO message to get the images (GET /v2/images), and we can see several DEBUG messages. We will use these DEBUG messages in a later post to describe handling logging output.
The INFO message will look like:
2016-04-04 16:24:00.139 INFO eventlet.wsgi.server [req-acf98429-60de-4d18-a69c-36a7d80bed7c a531ea1011af43bb8277f3e5edfea34b 3b9f48af38ac40a495ca7b22d4d5c036] 192.168.1.60 - - [04/Apr/2016 16:24:00] "GET /v2/images HTTP/1.1" 200 2202 0.116774
While we will discuss logging formats in another post, the standard format (in devstack) includes:
- Date/Time
- Logging Level
- Package
- Request context. this is made up of
- req-acf98429-60de-4d18-a69c-36a7d80bed7c a request-id, useful for grouping logging records
- a531ea1011af43bb8277f3e5edfea34b refers to the user id (as seen in user list above, i.e. admin)
- 3b9f48af38ac40a495ca7b22d4d5c036 refers to the project id (as seen in the project list above, i.e. demo)
- The actual log message
Understanding the logging of OpenStack services
What is actually observed in the screen output is what is being logged for the Glance API service. We can verify this with the log file logged in /opt/stack/logs
.
$ tail -f /opt/stack/logs/g-api.log
NOTE: You may see that there are colors within both the screen and log output. This is an optional configuration setup used by devstack (not an OpenStack default for logging). We will use this later to show a change in the logging of the service.
We can verify the details of the command used within the screen session (^a 5) by killing the running process with ^c.
Using the bash history, you can up arrow to observe the last running command, and restart this.
/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf & echo $! >/opt/stack/status/stack/g-api.pid; fg || echo "g-api failed to start" | tee "/opt/stack/status/stack/g-api.failure"
The actual log file is produced by the screen configuration defined in devstack/stack-screenrc
.
screen -t g-api bash "tuff "/usr/local/bin/glance-api --config-file=/etc/glance/glance-api.conf logfile /opt/stack/logs/g-api.log.2016-04-04-110956 log on
In a running OpenStack environment you would configure logging output to file as per the log_file option.
Understanding the configuration of OpenStack services
This command indicated a configuration file /etc/glance/glance-api.conf
. Glance like other services may contain several configuration files. These are by default defined in the individual projects namespace under /etc
.
$ ls -l /etc/glance/ total 152 -rw-r--r-- 1 stack stack 65106 Apr 4 11:12 glance-api.conf -rw-r--r-- 1 stack stack 3266 Mar 11 12:22 glance-api-paste.ini -rw-r--r-- 1 stack stack 13665 Apr 4 11:12 glance-cache.conf -rw-r--r-- 1 stack stack 51098 Apr 4 11:12 glance-registry.conf -rw-r--r-- 1 stack stack 1233 Mar 11 12:22 glance-registry-paste.ini drwxr-xr-x 2 stack root 4096 Apr 4 11:12 metadefs -rw-r--r-- 1 stack stack 1351 Mar 11 12:22 policy.json -rw-r--r-- 1 stack stack 1380 Mar 11 12:22 schema-image.json
This is an appropriate time to point to several documentation sources including the Glance Developer Documentation – Configuration Options and the Configuration Guide Image Service options which describe in more detail these listed configuration files and the possible options available. You can find similar documentation for other services.
To demonstrate just how the configuration and logging work with a running service the following will modify the logging of the Glance API service by commenting out the logging configuration lines, and then reverting to the oslo.log configuration defaults.
$ sudo vi /etc/glance/glance-api.conf
Comment out the four logging_ options in the [DEFAULT] section.
[DEFAULT] #logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s ^[[01;35m%(instance)s^[[00m #logging_debug_format_suffix = ^[[00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d^[[00m #logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[00;36m-%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m #logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[01;36m%(request_id)s ^[[00;36m%(user)s %(tenant)s%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
Now, repeating the earlier steps within the g-api screen window, kill and restart the service.
The first thing you will observe is that the logging no longer contains color (this helps greatly for log file analysis). Repeat the CLI option to list the images, and you will notice a slightly modified logging message occur.
2016-04-05 11:38:57.312 17696 INFO eventlet.wsgi.server [req-1e66b7e5-3429-452e-a9b7-e28ee498f772 a531ea1011af43bb8277f3e5edfea34b 3b9f48af38ac40a495ca7b22d4d5c036 - - -] 192.168.1.60 - - [05/Apr/2016 11:38:57] "GET /v2/images HTTP/1.1" 200 2202 11.551233
The request context now is a modified format (containing additional - - -
values) as a result of using the default value of logging_context_format_string. We will discuss the specifics of logging options in a later post.
There are a reasonable number of log files for a minimal devstack installation, some services have multiple log files.
$ cd /opt/stack/logs; ls -l *.log lrwxrwxrwx 1 stack stack 27 Apr 5 12:49 c-api.log -> c-api.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:49 c-sch.log -> c-sch.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:49 c-vol.log -> c-vol.log.2016-04-05-124004 -rw-r--r-- 1 stack stack 16672591 Apr 5 14:01 dstat-csv.log lrwxrwxrwx 1 stack stack 27 Apr 5 12:42 dstat.log -> dstat.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:48 g-api.log -> g-api.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:48 g-reg.log -> g-reg.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 29 Apr 5 12:50 horizon.log -> horizon.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 32 Apr 5 12:42 key-access.log -> key-access.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 25 Apr 5 12:42 key.log -> key.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:48 n-api.log -> n-api.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 29 Apr 5 12:49 n-cauth.log -> n-cauth.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 28 Apr 5 12:48 n-cond.log -> n-cond.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:49 n-cpu.log -> n-cpu.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:48 n-crt.log -> n-crt.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 28 Apr 5 12:42 n-dhcp.log -> n-dhcp.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:48 n-net.log -> n-net.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 29 Apr 5 12:49 n-novnc.log -> n-novnc.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 27 Apr 5 12:49 n-sch.log -> n-sch.log.2016-04-05-124004 lrwxrwxrwx 1 stack stack 46 Apr 5 12:40 stack.sh.log -> /opt/stack/logs/stack.sh.log.2016-04-05-124004
To turn off color in logging across service, you can configure this in the devstack local.conf
file before starting the stack.
# local.conf LOG_COLOR=False
Understanding the source code of OpenStack services
devstack installs the OpenStack code in two ways, via packaging and via source.
Generally all libraries are installed via packaging. You can discern these via looking at the python packages via pip with:
$ pip freeze ... oslo.cache==1.5.0 oslo.concurrency==3.6.0 oslo.config==3.9.0 oslo.context==2.2.0 oslo.db==4.6.0 oslo.i18n==3.4.0 oslo.log==3.2.0 oslo.messaging==4.5.0 oslo.middleware==3.7.0 oslo.policy==1.5.0 oslo.reports==1.6.0 oslo.rootwrap==4.1.0 oslo.serialization==2.4.0 oslo.service==1.7.0 oslo.utils==3.7.0 oslo.versionedobjects==1.7.0 oslo.vmware==2.5.0 ... python-barbicanclient==4.0.0 python-ceilometerclient==2.3.0 python-cinderclient==1.6.0 python-designateclient==2.0.0 python-glanceclient==2.0.0 python-heatclient==1.0.0 python-ironicclient==1.2.0 python-keystoneclient==2.3.1 python-magnumclient==1.1.0 python-manilaclient==1.8.0 python-memcached==1.57 python-mimeparse==1.5.1 python-mistralclient==2.0.0 python-neutronclient==4.1.1 python-novaclient==3.3.0 python-openstackclient==2.2.0 python-saharaclient==0.13.0 python-senlinclient==0.4.0 python-subunit==1.2.0 python-swiftclient==3.0.0 python-troveclient==2.1.1 python-zaqarclient==1.0.0 ...
This is a list of all Python packages so it’s not possible to determine which are OpenStack specific, and which are dependencies. These installed packages are actually Python source that you can view and even modify.
$ ls -l /usr/local/lib/python2.7/dist-packages/
You can approximate the installed OpenStack packages via source by looking at the base source directory:
$ ls -l /opt/stack total 92 drwxr-xr-x 10 stack stack 4096 Mar 11 12:23 cinder drwxr-xr-x 6 stack root 4096 Apr 5 12:42 data -rw-r--r-- 1 stack stack 440 Apr 5 12:52 devstack.subunit drwxr-xr-x 4 stack stack 4096 Mar 11 12:27 dib-utils drwxr-xr-x 10 stack stack 4096 Mar 11 12:22 glance drwxr-xr-x 15 stack stack 4096 Mar 11 12:26 heat drwxr-xr-x 7 stack stack 4096 Mar 11 12:27 heat-cfntools drwxr-xr-x 10 stack stack 4096 Mar 11 12:27 heat-templates drwxr-xr-x 11 stack stack 4096 Mar 11 14:13 horizon drwxr-xr-x 13 stack stack 4096 Mar 11 11:57 keystone drwxr-xr-x 2 stack stack 4096 Apr 5 12:50 logs drwxr-xr-x 12 stack stack 4096 Mar 11 15:45 neutron drwxr-xr-x 13 stack stack 4096 Mar 11 12:25 nova drwxr-xr-x 8 stack stack 4096 Mar 11 12:24 noVNC drwxr-xr-x 4 stack stack 4096 Mar 11 12:27 os-apply-config drwxr-xr-x 4 stack stack 4096 Mar 11 12:27 os-collect-config drwxr-xr-x 5 stack stack 4096 Mar 11 12:27 os-refresh-config drwxr-xr-x 7 stack stack 4096 Apr 5 12:51 requirements drwxr-xr-x 13 stack stack 4096 Mar 11 15:47 solum drwxr-xr-x 3 stack stack 4096 Apr 4 11:13 status drwxr-xr-x 10 stack stack 4096 Mar 11 12:22 swift
devstack enables you to configure which packages you want to install via source. Checkout plugins for more information. For example, the following added to the local.conf
would install solum.
# local.conf ... enable_plugin solum git://git.openstack.org/openstack/solum
You have complete flexibility of which branch and version of each package using devstack. This enables you to use devstack as a testing tool for code changes.
At this time to understand more about how software is installed check out devstack documentation and review the stack.sh
script.
What’s next
This is only a cursory introduction into what devstack sets up during the installation process. Subsequent posts will talk more on topics including the configuration options, the different logging capabilities and how to test code changes.
screen help
^a ?
will provide the following help output.
Screen key bindings, page 1 of 2. Command key: ^A Literal ^A: a break ^B b dumptermcap . info i meta a pow_detach D reset Z title A xoff ^S s clear C fit F kill K k monitor M prev ^H ^P p ^? screen ^C c vbell ^G xon ^Q q colon : flow ^F f lastmsg ^M m next ^@ ^N sp n quit \ select ' version v copy ^[ [ focus ^I license , number N readbuf < silence _ width W detach ^D d hardcopy h lockscreen ^X x only Q redisplay ^L l split S windows ^W w digraph ^V help ? log H other ^A remove X suspend ^Z z wrap ^R r displays * history { } login L pow_break B removebuf = time ^T t writebuf > ^] paste . " windowlist -b - select - 0 select 0 1 select 1 2 select 2 3 select 3 4 select 4 5 select 5 6 select 6 7 select 7 8 select 8 9 select 9 I login on O login off ] paste . | split -v :kB: focus prev
[…] Using your OpenStack devstack cloud […]