Following on from Installing Ubuntu OpenStack the following steps help you navigate around the single server installation, monitoring and debugging the installation process.
Configuration
The initial execution of the installer will create a default config.yaml
file that defines the container and OpenStack services. After a successful installation this looks like:
$ more $HOME/.cloud-install/config.yaml container_ip: 10.0.3.149 current_state: 2 deploy_complete: true install_type: Single openstack_password: openstack openstack_release: juno placements: controller: assignments: LXC: - nova-cloud-controller - glance - glance-simplestreams-sync - openstack-dashboard - juju-gui - keystone - mysql - neutron-api - neutron-openvswitch - rabbitmq-server constraints: cpu-cores: 2 mem: 6144 root-disk: 20480 nova-compute-machine-0: assignments: BareMetal: - nova-compute constraints: mem: 4096 root-disk: 40960 quantum-gateway-machine-0: assignments: BareMetal: - quantum-gateway constraints: mem: 2048 root-disk: 20480
This file changes during the installation process which I described later.
The LXC Container
The single server installation is managed within a single LXC container. You can obtain details of and connect to the container with the following.
$ sudo lxc-ls --fancy ---------------------------------------------------------------------------- uoi-bootstrap RUNNING 10.0.3.149, 10.0.4.1, 192.168.122.1 - YES $ sudo lxc-info --name uoi-bootstrap Name: uoi-bootstrap State: RUNNING PID: 19623 IP: 10.0.3.149 IP: 10.0.4.1 IP: 192.168.122.1 CPU use: 27692.85 seconds BlkIO use: 63.94 GiB Memory use: 24.29 GiB KMem use: 0 bytes Link: vethC0E9US TX bytes: 507.43 MiB RX bytes: 1.43 GiB Total bytes: 1.93 GiB $ sudo lxc-attach --name uoi-bootstrap
You can also connect to the server directly. As I prefer to NEVER configure or connect to a server as root this is how I access the LXC container.
$ ssh ubuntu@10.0.3.149
Juju Status
When connected to the LXC container you can then look at the status of the Juju orchestration with.
$ export JUJU_HOME=~/.cloud-install/juju $ juju status environment: local machines: "0": agent-state: started agent-version: 1.20.11.1 dns-name: localhost instance-id: localhost series: trusty state-server-member-status: has-vote "1": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.62 instance-id: ubuntu-local-machine-1 series: trusty hardware: arch=amd64 cpu-cores=1 mem=4096M root-disk=40960M "2": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.77 instance-id: ubuntu-local-machine-2 series: trusty containers: 2/lxc/0: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.147 instance-id: ubuntu-local-machine-2-lxc-0 series: trusty hardware: arch=amd64 2/lxc/1: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.15 instance-id: ubuntu-local-machine-2-lxc-1 series: trusty hardware: arch=amd64 2/lxc/2: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.135 instance-id: ubuntu-local-machine-2-lxc-2 series: trusty hardware: arch=amd64 2/lxc/3: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.133 instance-id: ubuntu-local-machine-2-lxc-3 series: trusty hardware: arch=amd64 2/lxc/4: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.119 instance-id: ubuntu-local-machine-2-lxc-4 series: trusty hardware: arch=amd64 2/lxc/5: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.88 instance-id: ubuntu-local-machine-2-lxc-5 series: trusty hardware: arch=amd64 2/lxc/6: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.155 instance-id: ubuntu-local-machine-2-lxc-6 series: trusty hardware: arch=amd64 2/lxc/7: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.36 instance-id: ubuntu-local-machine-2-lxc-7 series: trusty hardware: arch=amd64 2/lxc/8: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.11 instance-id: ubuntu-local-machine-2-lxc-8 series: trusty hardware: arch=amd64 hardware: arch=amd64 cpu-cores=2 mem=6144M root-disk=20480M "3": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.10 instance-id: ubuntu-local-machine-3 series: trusty hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=20480M "4": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.96 instance-id: ubuntu-local-machine-4 series: trusty hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M "5": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.140 instance-id: ubuntu-local-machine-5 series: trusty hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M "6": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.197 instance-id: ubuntu-local-machine-6 series: trusty hardware: arch=amd64 cpu-cores=1 mem=512M root-disk=8192M services: glance: charm: cs:trusty/glance-11 exposed: false relations: amqp: - rabbitmq-server cluster: - glance identity-service: - keystone image-service: - nova-cloud-controller - nova-compute object-store: - swift-proxy shared-db: - mysql units: glance/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/4 open-ports: - 9292/tcp public-address: 10.0.4.119 glance-simplestreams-sync: charm: local:trusty/glance-simplestreams-sync-0 exposed: false relations: amqp: - rabbitmq-server identity-service: - keystone units: glance-simplestreams-sync/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/5 public-address: 10.0.4.88 juju-gui: charm: cs:trusty/juju-gui-16 exposed: false units: juju-gui/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/1 open-ports: - 80/tcp - 443/tcp public-address: 10.0.4.15 keystone: charm: cs:trusty/keystone-12 exposed: false relations: cluster: - keystone identity-service: - glance - glance-simplestreams-sync - neutron-api - nova-cloud-controller - openstack-dashboard - swift-proxy shared-db: - mysql units: keystone/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/2 public-address: 10.0.4.135 mysql: charm: cs:trusty/mysql-12 exposed: false relations: cluster: - mysql shared-db: - glance - keystone - neutron-api - nova-cloud-controller - nova-compute - quantum-gateway units: mysql/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/0 public-address: 10.0.4.147 neutron-api: charm: cs:trusty/neutron-api-6 exposed: false relations: amqp: - rabbitmq-server cluster: - neutron-api identity-service: - keystone neutron-api: - nova-cloud-controller neutron-plugin-api: - neutron-openvswitch shared-db: - mysql units: neutron-api/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/7 open-ports: - 9696/tcp public-address: 10.0.4.36 neutron-openvswitch: charm: cs:trusty/neutron-openvswitch-2 exposed: false relations: amqp: - rabbitmq-server neutron-plugin: - nova-compute neutron-plugin-api: - neutron-api subordinate-to: - nova-compute nova-cloud-controller: charm: cs:trusty/nova-cloud-controller-51 exposed: false relations: amqp: - rabbitmq-server cloud-compute: - nova-compute cluster: - nova-cloud-controller identity-service: - keystone image-service: - glance neutron-api: - neutron-api quantum-network-service: - quantum-gateway shared-db: - mysql units: nova-cloud-controller/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/3 open-ports: - 3333/tcp - 8773/tcp - 8774/tcp - 9696/tcp public-address: 10.0.4.133 nova-compute: charm: cs:trusty/nova-compute-14 exposed: false relations: amqp: - rabbitmq-server cloud-compute: - nova-cloud-controller compute-peer: - nova-compute image-service: - glance neutron-plugin: - neutron-openvswitch shared-db: - mysql units: nova-compute/0: agent-state: started agent-version: 1.20.11.1 machine: "1" public-address: 10.0.4.62 subordinates: neutron-openvswitch/0: upgrading-from: cs:trusty/neutron-openvswitch-2 agent-state: started agent-version: 1.20.11.1 public-address: 10.0.4.62 openstack-dashboard: charm: cs:trusty/openstack-dashboard-9 exposed: false relations: cluster: - openstack-dashboard identity-service: - keystone units: openstack-dashboard/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/6 open-ports: - 80/tcp - 443/tcp public-address: 10.0.4.155 quantum-gateway: charm: cs:trusty/quantum-gateway-10 exposed: false relations: amqp: - rabbitmq-server cluster: - quantum-gateway quantum-network-service: - nova-cloud-controller shared-db: - mysql units: quantum-gateway/0: agent-state: started agent-version: 1.20.11.1 machine: "3" public-address: 10.0.4.10 rabbitmq-server: charm: cs:trusty/rabbitmq-server-26 exposed: false relations: amqp: - glance - glance-simplestreams-sync - neutron-api - neutron-openvswitch - nova-cloud-controller - nova-compute - quantum-gateway cluster: - rabbitmq-server units: rabbitmq-server/0: agent-state: started agent-version: 1.20.11.1 machine: 2/lxc/8 open-ports: - 5672/tcp public-address: 10.0.4.11
You can also look at a subset of the status for a particular service, for example keystone with:
$ juju status keystone environment: local machines: "1": agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.128 instance-id: ubuntu-local-machine-1 series: trusty containers: 1/lxc/2: agent-state: started agent-version: 1.20.11.1 dns-name: 10.0.4.142 instance-id: ubuntu-local-machine-1-lxc-2 series: trusty hardware: arch=amd64 hardware: arch=amd64 cpu-cores=2 mem=6144M root-disk=20480M services: keystone: charm: cs:trusty/keystone-12 exposed: false relations: cluster: - keystone identity-service: - glance - glance-simplestreams-sync - neutron-api - nova-cloud-controller - openstack-dashboard shared-db: - mysql units: keystone/0: agent-state: started agent-version: 1.20.11.1 machine: 1/lxc/2 public-address: 10.0.4.142
Monitoring the Installation
When performing an installation you can monitor the executed commands with:
$ tail -f $HOME/.cloud-install/commands.log ...
This provides a lot of debugging output. A streamlined logging is actually possible with automated installation described later.
Uninstalling
As the single server instance is in a LXC container, as the documentation states uninstalling the environment is a rather trivial process that takes only a few seconds.
This will teardown the cloud but leaving userdata available for a subsequent deployment.
$ sudo openstack-install -k Warning: This will destroy the host Container housing the OpenStack private cloud. This is a permanent operation. Proceed? [y/N] Y Removing static route Removing host container... Container is removed.
You can also do a more permanent uninstall of the cloud and packages.
$ sudo openstack-install -u Warning: This will uninstall OpenStack and make a best effort to return the system back to its original state. Proceed? [y/N] Y Restoring system to last known state. Ubuntu Openstack Installer Uninstalling ...Single install path.
This does not however seem to cleanup $HOME/.cloud-install. You can safely remove this or move it sideways when re-deploying without any issues.
Installation automation
As described in my original post, the openstack-install
script is a cursors-based interactive view. You can automate the installation by defining the needed setup inputs in a separate configuration file and running in headless mode.
$ echo "install_type: Single openstack_password: openstack" > install.yaml $ sudo openstack-install --headless --config install.yaml
This has the added benefit providing a more meaningful log of the state of the installation with less verbose information then in the commands.log file.
[INFO • 06-02 12:02:42 • cloudinstall.install] Running in headless mode. [INFO • 06-02 12:02:42 • cloudinstall.install] Performing a Single Install [INFO • 06-02 12:02:42 • cloudinstall.task] [TASKLIST] ['Initializing Environment', 'Creating container', 'Bootstrapping Juju'] [INFO • 06-02 12:02:42 • cloudinstall.task] [TASK] Initializing Environment [INFO • 06-02 12:02:42 • cloudinstall.consoleui] Building environment [INFO • 06-02 12:02:42 • cloudinstall.single_install] Prepared userdata: {'extra_sshkeys': ['ssh-rsa ...\n'], 'seed_command': ['env', 'pollinate', '-q']} [INFO • 06-02 12:02:42 • cloudinstall.single_install] Setting permissions for user rbradfor [INFO • 06-02 12:02:43 • cloudinstall.task] [TASK] Creating container [INFO • 06-02 12:04:20 • cloudinstall.single_install] Setting DHCP properties for host container. [INFO • 06-02 12:04:20 • cloudinstall.single_install] Adding static route for 10.0.4.0/24 via 10.0.3.160 ... [INFO • 06-02 12:22:50 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: pending [INFO • 06-02 12:23:31 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: installed [INFO • 06-02 12:23:52 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started [INFO • 06-02 12:24:34 • cloudinstall.consoleui] Checking availability of keystone: started [INFO • 06-02 12:24:44 • cloudinstall.consoleui] Checking availability of keystone: started [INFO • 06-02 12:24:44 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started [INFO • 06-02 12:27:38 • cloudinstall.consoleui] Checking availability of quantum-gateway: started [INFO • 06-02 12:27:38 • cloudinstall.consoleui] Checking availability of nova-cloud-controller: started [INFO • 06-02 12:27:38 • cloudinstall.consoleui] Validating network parameters for Neutron [INFO • 06-02 12:27:53 • cloudinstall.consoleui] All systems go!=
And 25 minutes later you have an available cloud.
If you attempt to look at the GUI status page with openstack-status
you will be given a text based version of messages like.
$ sudo openstack-status [INFO • 06-02 12:06:21 • cloudinstall.core] Running openstack-status in headless mode. [INFO • 06-02 12:06:21 • cloudinstall.consoleui] Loaded placements from file. [INFO • 06-02 12:06:21 • cloudinstall.consoleui] Waiting for machines to start: 3 unknown [INFO • 06-02 12:08:20 • cloudinstall.consoleui] Waiting for machines to start: 1 pending, 2 unknown [INFO • 06-02 12:08:48 • cloudinstall.consoleui] Waiting for machines to start: 2 pending, 1 unknown [INFO • 06-02 12:09:04 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 1 pending, 1 unknown [INFO • 06-02 12:09:13 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 2 pending [INFO • 06-02 12:09:20 • cloudinstall.consoleui] Waiting for machines to start: 2 down (started), 1 pending [INFO • 06-02 12:09:26 • cloudinstall.consoleui] Waiting for machines to start: 1 pending, 2 started [INFO • 06-02 12:09:51 • cloudinstall.consoleui] Waiting for machines to start: 1 down (started), 2 started [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Verifying service deployments [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Missing ConsoleUI() attribute: set_pending_deploys [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Checking if MySQL is deployed [INFO • 06-02 12:10:44 • cloudinstall.consoleui] Deploying MySQL to machine lxc:1 [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Deployed MySQL. [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Checking if Juju GUI is deployed [INFO • 06-02 12:10:49 • cloudinstall.consoleui] Deploying Juju GUI to machine lxc:1 [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Deployed Juju GUI. [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Checking if Keystone is deployed [INFO • 06-02 12:11:00 • cloudinstall.consoleui] Deploying Keystone to machine lxc:1 ...
It seems you can trick it into providing both a GUI and text version with the following in another shell session.
$ sed -ie "/headless/d" $HOME/.cloud-install/config.yaml $ sudo openstack-status
NOTE: You will not get any output until the initial container is completed. This also leaves a .pid file that must be manually cleaned up if you run to soon. The next invocation provides the following message.
$ sudo openstack-status Another instance of openstack-status is running. If you're sure there are no other instances, please remove ~/.cloud-install/openstack.pid $ rm $HOME/.cloud-install/openstack.pid
Monitoring the installation progress
The running config.yaml
file changes over the duration of the installation.
It’s most basic configuration (when starting with the GUI) is:
$ more $HOME/.cloud-install/config.yaml current_state: 0 openstack_release: juno
The release is also defined in the $HOME/.cloud-install/openstack_release file.
When starting by passing the configuration as previously mentioned it’s initial state is:
$ more $HOME/.cloud-install/config.yaml config_file: install.yaml current_state: 0 headless: true install_type: Single openstack_password: openstack openstack_release: juno
This is updated when the LXC container is installed.
$ more $HOME/.cloud-install/config.yaml config_file: install.yaml container_ip: 10.0.3.77 current_state: 0 headless: true install_type: Single openstack_password: openstack openstack_release: juno
And also updated during installation, such as.
$ more $HOME/.cloud-install/config.yaml config_file: install.yaml container_ip: 10.0.3.77 current_state: 0 headless: true install_type: Single openstack_password: openstack openstack_release: juno placements: controller: assignments: LXC: - nova-cloud-controller - glance - glance-simplestreams-sync - openstack-dashboard - juju-gui - keystone - mysql - neutron-api - neutron-openvswitch - rabbitmq-server constraints: cpu-cores: 2 mem: 6144 root-disk: 20480 nova-compute-machine-0: assignments: BareMetal: - nova-compute constraints: mem: 4096 root-disk: 40960 quantum-gateway-machine-0: assignments: BareMetal: - quantum-gateway constraints: mem: 2048 root-disk: 20480
When completed the configuration has the following settings.
config_file: install.yaml container_ip: 10.0.3.77 current_state: 2 deploy_complete: true install_type: Single openstack_password: openstack openstack_release: juno placements: ...
Problems
When using the GUI installer the first time you quit (using Q), it seems to leave the terminal state wrong. The following will reset this to normal use.
$ stty sane ^j # (i.e. Ctrl-J together).
Subsequent uses of openstack-status
do not have the same problem.
References
In my next post I am going to talk about the analysis taken to debug errors in the installation, starting with Keystone – hook failed: “config-changed” message I got attempting to install kilo, and hence this more detailed analysis of the installation process components.