Following on from Installing Ubuntu OpenStack the following steps help you navigate around the single server installation, monitoring and debugging the installation process.
Configuration
The initial execution of the installer will create a default config.yaml file that defines the container and OpenStack services. After a successful installation this looks like:
$ more $HOME/.cloud-install/config.yaml
container_ip: 10.0.3.149
current_state: 2
deploy_complete: true
install_type: Single
openstack_password: openstack
openstack_release: juno
placements:
controller:
assignments:
LXC:
- nova-cloud-controller
- glance
- glance-simplestreams-sync
- openstack-dashboard
- juju-gui
- keystone
- mysql
- neutron-api
- neutron-openvswitch
- rabbitmq-server
constraints:
cpu-cores: 2
mem: 6144
root-disk: 20480
nova-compute-machine-0:
assignments:
BareMetal:
- nova-compute
constraints:
mem: 4096
root-disk: 40960
quantum-gateway-machine-0:
assignments:
BareMetal:
- quantum-gateway
constraints:
mem: 2048
root-disk: 20480
This file changes during the installation process which I described later.
The LXC Container
The single server installation is managed within a single LXC container. You can obtain details of and connect to the container with the following.
$ sudo lxc-ls --fancy ---------------------------------------------------------------------------- uoi-bootstrap RUNNING 10.0.3.149, 10.0.4.1, 192.168.122.1 - YES $ sudo lxc-info --name uoi-bootstrap Name: uoi-bootstrap State: RUNNING PID: 19623 IP: 10.0.3.149 IP: 10.0.4.1 IP: 192.168.122.1 CPU use: 27692.85 seconds BlkIO use: 63.94 GiB Memory use: 24.29 GiB KMem use: 0 bytes Link: vethC0E9US TX bytes: 507.43 MiB RX bytes: 1.43 GiB Total bytes: 1.93 GiB $ sudo lxc-attach --name uoi-bootstrap
You can also connect to the server directly. As I prefer to NEVER configure or connect to a server as root this is how I access the LXC container.
$ ssh ubuntu@10.0.3.149
Juju Status
When connected to the LXC container you can then look at the status of the Juju orchestration with.
$ export JUJU_HOME=~/.cloud-install/juju
$ juju status
environment: local
machines:
"0":
agent-state: started
agent-version: 1.20.11.1
dns-name: localhost
instance-id: localhost
series: trusty
state-server-member-status: has-vote
"1":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.62
instance-id: ubuntu-local-machine-1
series: trusty
hardware: arch=amd64 cpu-cores=1 mem=4096M root-disk=40960M
"2":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.77
instance-id: ubuntu-local-machine-2
series: trusty
containers:
2/lxc/0:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.147
instance-id: ubuntu-local-machine-2-lxc-0
series: trusty
hardware: arch=amd64
2/lxc/1:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.15
instance-id: ubuntu-local-machine-2-lxc-1
series: trusty
hardware: arch=amd64
2/lxc/2:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.135
instance-id: ubuntu-local-machine-2-lxc-2
series: trusty
hardware: arch=amd64
2/lxc/3:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.133
instance-id: ubuntu-local-machine-2-lxc-3
series: trusty
hardware: arch=amd64
2/lxc/4:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.119
instance-id: ubuntu-local-machine-2-lxc-4
series: trusty
hardware: arch=amd64
2/lxc/5:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.88
instance-id: ubuntu-local-machine-2-lxc-5
series: trusty
hardware: arch=amd64
2/lxc/6:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.155
instance-id: ubuntu-local-machine-2-lxc-6
series: trusty
hardware: arch=amd64
2/lxc/7:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.36
instance-id: ubuntu-local-machine-2-lxc-7
series: trusty
hardware: arch=amd64
2/lxc/8:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.11
instance-id: ubuntu-local-machine-2-lxc-8
series: trusty
hardware: arch=amd64
hardware: arch=amd64 cpu-cores=2 mem=6144M root-disk=20480M
"3":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.10
instance-id: ubuntu-local-machine-3
series: trusty
hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=20480M
"4":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.96
instance-id: ubuntu-local-machine-4
series: trusty
hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M
"5":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.140
instance-id: ubuntu-local-machine-5
series: trusty
hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M
"6":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.197
instance-id: ubuntu-local-machine-6
series: trusty
hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M
services:
glance:
charm: cs:trusty/glance-11
exposed: false
relations:
amqp:
- rabbitmq-server
cluster:
- glance
identity-service:
- keystone
image-service:
- nova-cloud-controller
- nova-compute
object-store:
- swift-proxy
shared-db:
- mysql
units:
glance/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/4
open-ports:
- 9292/tcp
public-address: 10.0.4.119
glance-simplestreams-sync:
charm: local:trusty/glance-simplestreams-sync-0
exposed: false
relations:
amqp:
- rabbitmq-server
identity-service:
- keystone
units:
glance-simplestreams-sync/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/5
public-address: 10.0.4.88
juju-gui:
charm: cs:trusty/juju-gui-16
exposed: false
units:
juju-gui/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/1
open-ports:
- 80/tcp
- 443/tcp
public-address: 10.0.4.15
keystone:
charm: cs:trusty/keystone-12
exposed: false
relations:
cluster:
- keystone
identity-service:
- glance
- glance-simplestreams-sync
- neutron-api
- nova-cloud-controller
- openstack-dashboard
- swift-proxy
shared-db:
- mysql
units:
keystone/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/2
public-address: 10.0.4.135
mysql:
charm: cs:trusty/mysql-12
exposed: false
relations:
cluster:
- mysql
shared-db:
- glance
- keystone
- neutron-api
- nova-cloud-controller
- nova-compute
- quantum-gateway
units:
mysql/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/0
public-address: 10.0.4.147
neutron-api:
charm: cs:trusty/neutron-api-6
exposed: false
relations:
amqp:
- rabbitmq-server
cluster:
- neutron-api
identity-service:
- keystone
neutron-api:
- nova-cloud-controller
neutron-plugin-api:
- neutron-openvswitch
shared-db:
- mysql
units:
neutron-api/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/7
open-ports:
- 9696/tcp
public-address: 10.0.4.36
neutron-openvswitch:
charm: cs:trusty/neutron-openvswitch-2
exposed: false
relations:
amqp:
- rabbitmq-server
neutron-plugin:
- nova-compute
neutron-plugin-api:
- neutron-api
subordinate-to:
- nova-compute
nova-cloud-controller:
charm: cs:trusty/nova-cloud-controller-51
exposed: false
relations:
amqp:
- rabbitmq-server
cloud-compute:
- nova-compute
cluster:
- nova-cloud-controller
identity-service:
- keystone
image-service:
- glance
neutron-api:
- neutron-api
quantum-network-service:
- quantum-gateway
shared-db:
- mysql
units:
nova-cloud-controller/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/3
open-ports:
- 3333/tcp
- 8773/tcp
- 8774/tcp
- 9696/tcp
public-address: 10.0.4.133
nova-compute:
charm: cs:trusty/nova-compute-14
exposed: false
relations:
amqp:
- rabbitmq-server
cloud-compute:
- nova-cloud-controller
compute-peer:
- nova-compute
image-service:
- glance
neutron-plugin:
- neutron-openvswitch
shared-db:
- mysql
units:
nova-compute/0:
agent-state: started
agent-version: 1.20.11.1
machine: "1"
public-address: 10.0.4.62
subordinates:
neutron-openvswitch/0:
upgrading-from: cs:trusty/neutron-openvswitch-2
agent-state: started
agent-version: 1.20.11.1
public-address: 10.0.4.62
openstack-dashboard:
charm: cs:trusty/openstack-dashboard-9
exposed: false
relations:
cluster:
- openstack-dashboard
identity-service:
- keystone
units:
openstack-dashboard/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/6
open-ports:
- 80/tcp
- 443/tcp
public-address: 10.0.4.155
quantum-gateway:
charm: cs:trusty/quantum-gateway-10
exposed: false
relations:
amqp:
- rabbitmq-server
cluster:
- quantum-gateway
quantum-network-service:
- nova-cloud-controller
shared-db:
- mysql
units:
quantum-gateway/0:
agent-state: started
agent-version: 1.20.11.1
machine: "3"
public-address: 10.0.4.10
rabbitmq-server:
charm: cs:trusty/rabbitmq-server-26
exposed: false
relations:
amqp:
- glance
- glance-simplestreams-sync
- neutron-api
- neutron-openvswitch
- nova-cloud-controller
- nova-compute
- quantum-gateway
cluster:
- rabbitmq-server
units:
rabbitmq-server/0:
agent-state: started
agent-version: 1.20.11.1
machine: 2/lxc/8
open-ports:
- 5672/tcp
public-address: 10.0.4.11
You can also look at a subset of the status for a particular service, for example keystone with:
$ juju status keystone
environment: local
machines:
"1":
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.128
instance-id: ubuntu-local-machine-1
series: trusty
containers:
1/lxc/2:
agent-state: started
agent-version: 1.20.11.1
dns-name: 10.0.4.142
instance-id: ubuntu-local-machine-1-lxc-2
series: trusty
hardware: arch=amd64
hardware: arch=amd64 cpu-cores=2 mem=6144M root-disk=20480M
services:
keystone:
charm: cs:trusty/keystone-12
exposed: false
relations:
cluster:
- keystone
identity-service:
- glance
- glance-simplestreams-sync
- neutron-api
- nova-cloud-controller
- openstack-dashboard
shared-db:
- mysql
units:
keystone/0:
agent-state: started
agent-version: 1.20.11.1
machine: 1/lxc/2
public-address: 10.0.4.142
Monitoring the Installation
When performing an installation you can monitor the executed commands with:
$ tail -f $HOME/.cloud-install/commands.log ...
This provides a lot of debugging output. A streamlined logging is actually possible with automated installation described later.
Uninstalling
As the single server instance is in a LXC container, as the documentation states uninstalling the environment is a rather trivial process that takes only a few seconds.
This will teardown the cloud but leaving userdata available for a subsequent deployment.
$ sudo openstack-install -k Warning: This will destroy the host Container housing the OpenStack private cloud. This is a permanent operation. Proceed? [y/N] Y Removing static route Removing host container... Container is removed.
You can also do a more permanent uninstall of the cloud and packages.
$ sudo openstack-install -u Warning: This will uninstall OpenStack and make a best effort to return the system back to its original state. Proceed? [y/N] Y Restoring system to last known state. Ubuntu Openstack Installer Uninstalling ...Single install path.
This does not however seem to cleanup $HOME/.cloud-install. You can safely remove this or move it sideways when re-deploying without any issues.
Installation automation
As described in my original post, the openstack-install script is a cursors-based interactive view. You can automate the installation by defining the needed setup inputs in a separate configuration file and running in headless mode.
$ echo "install_type: Single openstack_password: openstack" > install.yaml $ sudo openstack-install --headless --config install.yaml
This has the added benefit providing a more meaningful log of the state of the installation with less verbose information then in the commands.log file.
[INFO • 06-02 12:02:42 • cloudinstall.install] Running in headless mode.
[INFO • 06-02 12:02:42 • cloudinstall.install] Performing a Single Install
[INFO • 06-02 12:02:42 • cloudinstall.task] [TASKLIST] ['Initializing Environment', 'Creating container', 'Bootstrapping Juju']
[INFO • 06-02 12:02:42 • cloudinstall.task] [TASK] Initializing Environment
[INFO • 06-02 12:02:42 • cloudinstall.consoleui] Building environment
[INFO • 06-02 12:02:42 • cloudinstall.single_install] Prepared userdata: {'extra_sshkeys': ['ssh-rsa ...\n'], 'seed_command': ['env', 'pollinate', '-q']}
[INFO • 06-02 12:02:42 • cloudinstall.single_install] Setting permissions for user rbradfor
[INFO • 06-02 12:02:43 • cloudinstall.task] [TASK] Creating container
[INFO • 06-02 12:04:20 • cloudinstall.single_install] Setting DHCP properties for host container.
[INFO • 06-02 12:04:20 • cloudinstall.single_install] Adding static route for 10.0.4.0/24 via 10.0.3.160
...
[INFO • 06-02 12:22:50 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: pending
[INFO • 06-02 12:23:31 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: installed
[INFO • 06-02 12:23:52 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started
[INFO • 06-02 12:24:34 • cloudinstall.consoleui] Checking availability of keystone: started
[INFO • 06-02 12:24:44 • cloudinstall.consoleui] Checking availability of keystone: started
[INFO • 06-02 12:24:44 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started
[INFO • 06-02 12:27:38 • cloudinstall.consoleui] Checking availability of quantum-gateway: started
[INFO • 06-02 12:27:38 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started
[INFO • 06-02 12:27:38 • cloudinstall.consoleui] Validating network parameters for Neutron
[INFO • 06-02 12:27:53 • cloudinstall.consoleui] All systems go!=
And 25 minutes later you have an available cloud.
If you attempt to look at the GUI status page with openstack-status you will be given a text based version of messages like.
$ sudo openstack-status [INFO • 06-02 12:06:21 • cloudinstall.core] Running openstack-status in headless mode. [INFO • 06-02 12:06:21 • cloudinstall.consoleui] Loaded placements from file. [INFO • 06-02 12:06:21 • cloudinstall.consoleui] Waiting for machines to start: 3 unknown [INFO • 06-02 12:08:20 • cloudinstall.consoleui] Waiting for machines to start: 1 pending, 2 unknown [INFO • 06-02 12:08:48 • cloudinstall.consoleui] Waiting for machines to start: 2 pending, 1 unknown [INFO • 06-02 12:09:04 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 1 pending, 1 unknown [INFO • 06-02 12:09:13 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 2 pending [INFO • 06-02 12:09:20 • cloudinstall.consoleui] Waiting for machines to start: 2 down (started), 1 pending [INFO • 06-02 12:09:26 • cloudinstall.consoleui] Waiting for machines to start: 1 pending, 2 started [INFO • 06-02 12:09:51 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 2 started [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Verifying service deployments [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Missing ConsoleUI() attribute: set_pending_deploys [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Checking if MySQL is deployed [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Deploying MySQL to machine lxc:1 [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Deployed MySQL. [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Checking if Juju GUI is deployed [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Deploying Juju GUI to machine lxc:1 [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Deployed Juju GUI. [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Checking if Keystone is deployed [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Deploying Keystone to machine lxc:1 ...
It seems you can trick it into providing both a GUI and text version with the following in another shell session.
$ sed -ie "/headless/d" $HOME/.cloud-install/config.yaml $ sudo openstack-status
NOTE: You will not get any output until the initial container is completed. This also leaves a .pid file that must be manually cleaned up if you run to soon. The next invocation provides the following message.
$ sudo openstack-status Another instance of openstack-status is running. If you're sure there are no other instances, please remove ~/.cloud-install/openstack.pid $ rm $HOME/.cloud-install/openstack.pid
Monitoring the installation progress
The running config.yaml file changes over the duration of the installation.
It’s most basic configuration (when starting with the GUI) is:
$ more $HOME/.cloud-install/config.yaml current_state: 0 openstack_release: juno
The release is also defined in the $HOME/.cloud-install/openstack_release file.
When starting by passing the configuration as previously mentioned it’s initial state is:
$ more $HOME/.cloud-install/config.yaml config_file: install.yaml current_state: 0 headless: true install_type: Single openstack_password: openstack openstack_release: juno
This is updated when the LXC container is installed.
$ more $HOME/.cloud-install/config.yaml config_file: install.yaml container_ip: 10.0.3.77 current_state: 0 headless: true install_type: Single openstack_password: openstack openstack_release: juno
And also updated during installation, such as.
$ more $HOME/.cloud-install/config.yaml
config_file: install.yaml
container_ip: 10.0.3.77
current_state: 0
headless: true
install_type: Single
openstack_password: openstack
openstack_release: juno
placements:
controller:
assignments:
LXC:
- nova-cloud-controller
- glance
- glance-simplestreams-sync
- openstack-dashboard
- juju-gui
- keystone
- mysql
- neutron-api
- neutron-openvswitch
- rabbitmq-server
constraints:
cpu-cores: 2
mem: 6144
root-disk: 20480
nova-compute-machine-0:
assignments:
BareMetal:
- nova-compute
constraints:
mem: 4096
root-disk: 40960
quantum-gateway-machine-0:
assignments:
BareMetal:
- quantum-gateway
constraints:
mem: 2048
root-disk: 20480
When completed the configuration has the following settings.
config_file: install.yaml container_ip: 10.0.3.77 current_state: 2 deploy_complete: true install_type: Single openstack_password: openstack openstack_release: juno placements: ...
Problems
When using the GUI installer the first time you quit (using Q), it seems to leave the terminal state wrong. The following will reset this to normal use.
$ stty sane ^j # (i.e. Ctrl-J together).
Subsequent uses of openstack-status do not have the same problem.
References
In my next post I am going to talk about the analysis taken to debug errors in the installation, starting with Keystone – hook failed: “config-changed” message I got attempting to install kilo, and hence this more detailed analysis of the installation process components.
