Digital transformation strategies

“The cosmos is complex, the cloud does not have to be”.

This quote by Ben Amaba, Worldwide Executive at IBM Cloud, early in his presentation at the Performance without Limits 3.0 on IBM Cloud event was his introduction to what I interpreted as stepping back from “what do I do with the cloud?” to consider “what makes my business successful?”. Indeed “the cloud”, i.e., Infrastructure as a Service (IaaS), should not be the complicated component in your business strategy.

The realization is that today, digital information exists and its growth is accelerating exponentially. Any strategy to embrace this need is essential to maintaining business success. This implies that to achieve transformation, your business has to include using available and potentially un-thought-of digital information to innovate and personalize. The present traditional approach towards provisioning resources and services simply cannot meet this need. Hence the adoption of “utilizing the cloud” is becoming the ubiquitous answer.

The business model that one develops for maximizing this infrastructure-on-demand needs to be a provable, reproducible, resilient and a flexible reference architecture. It needs to have set principles to embrace the potential of the cloud. It needs to minimize the potential of failure.

Amaba talked about having three guiding principles in his presentation. These are:

  1. Hybrid
  2. Discipline
  3. Analytical

As we consider digital transformation strategies, just understanding the potential capability of a hybrid structure is required. This will vary from organization and industry and will rely on a balance of private and public cloud services. Locking your organization into an all public cloud solution (e.g. AWS), or an all private cloud solution (e.g. all VMWare) limits your capacity to adapt.

Implementing an on-premise cloud infrastructure that leverages OpenStack to replace existing propriety off-premise cloud providers such as AWS, Azure and Google Cloud is not the ROI you should hope for. Indeed a hybrid private/public strategy with the capacity to enable greater access to applicable and real-time data and tools, providing the ability for your employees, associated researchers, strategic partners and even individuals via a foldit gamification type approach all increase the innovative transformation that can ensure your company is the disruptor, not the disrupted.

Let’s consider a theoretical example when a hybrid cloud strategy enables the capacity for innovation to occur in record time. CERN announces the release of 300TB of LDR data. If this was released into one specific cloud infrastructure, could your organization support that? For example, if this was the SoftLayer cloud infrastructure, leveraging the compute resources of this cloud provider would be beneficial to your internal organization because one feature of this cloud is that it includes free data transfer across the entire worldwide network. This one feature on a specific cloud has an immediate cost-benefit.

Not being capable of expanding your organizations authentication, intellectual property analysis engines, and tools to quickly and seamlessly cater for the data could also be a competitive disadvantage. Amaba noted that 1/3 of top companies will face disruption in the next few years. Disruption is not limited to a competitor, a customer or a supplier. It can include the lack of ability to adapt timely to opportunity. Is being able to utilize any public cloud rather than one specific public cloud included in your business process? Could your internal global infrastructure and network support an additional 300TB of data immediately? While this is a specific use case, the availability of data and the ability for your organization’s employees to consume, digest and analyze is a digital strategy you need to be prepared for to compete in speed and innovation within your industry. Is the source of new data for your organization an opportunity or a problem?

The discussion of whata hybrid cloud is and how does my organization cater for and uses a hybrid cloud is the reason that thought leadership is needed. To understand the enterprise architecture of legacy systems, the capacity of new cloud-native applications and the huge divide in transition between these to enable utilization of existing data-wealth must also be part of your transformation strategy.

Amaba’s presentation also included discussing the discipline of needing to ensure and provide consistency. This ranges from the varying views of information to your consumers to the choices for workload assignment and access. Analytics was the third principle that encompassed the capability to determine insights, from using big data analysis to cognitive computing.

These thoughts are a reflection on the few notes taken at the time. I am really looking forward to seeing the slides and video presentation to fully reflect and comment in more detail.

Understanding the Oslo Libraries

Underpinning all of the OpenStack projects including Nova, Cinder, Keystone, Glance, Horizon, Heat, Trove, Murano and others is a set of core common libraries that provide a consistent, highly tested and compatible feature set. The Oslo project is a collection of over 30 libraries that are designed to reduce the technical debt of code duplication across projects and provide for a greater quality code path due to the frequency of use in OpenStack projects.

These libraries provide a variety of different features from the more commonly used functionality found in projects including configuration, logging, caching, messaging and database management to more specific features like deprecation management, handling plugins as well as frameworks for command line programs and state machines. The Oslo Python libraries are designed to be Python 2.7 and Python 3.4 compatible, leading the way in migration towards Python 3.

The first stable Oslo library oslo.config was included in the Grizzly release. Now over 30 libraries comprise the Oslo project. These libraries fall into a number of broad categories.

1. Stable OpenStack specific libraries

These libraries, using the olso. prefix are generally well described the library name.

  • oslo.cache
  • oslo.concurrency
  • oslo.context
  • oslo.config
  • oslo.db
  • oslo.i18n
  • oslo.log
  • oslo.messaging
  • oslo.middleware
  • oslo.policy
  • oslo.privsep
  • oslo.reports
  • oslo.serialization
  • oslo.service
  • oslo.utils
  • oslo.versionedobjects
  • oslo.vmware

2. Python libraries that can easily operate with other projects

In addition to the oslo namespace libraries, Oslo has a number of generically named libraries that are not OpenStack specific. The goal is that these libraries can be utilized outside of OpenStack by any Python project. These include:

  • automaton – a framework for building state machines.
  • cliff – a framework for building command line programs.
  • debtcollector – a collection of python patterns that help you collect your technical debt in a non-destructive manner (following deprecation patterns and strategies and so-on).
  • futurist – a collection of async functionality and additions from the future.
  • osprofiler – an OpenStack cross-project profiling library.
  • hacking – a library that provides a set of tools for enforcing coding style guidelines.
  • pbr – (or Python Build Reasonableness) is a add-on library that helps provide (and enforce) a set of sensible default setuptools behaviours.
  • pyCADF – a python implementation of the DMTF Cloud Audit (CADF) data model.
  • stevedore – a library for managing plugins for Python applications.
  • taskflow – a library that helps create applications that handle state/failures… in a reasonable manner.
  • tooz – a library that aims at centralizing the most common distributed primitives like group membership protocol, lock service and leader election

3. Convenience libraries

There are also several libraries that are used during the creation of, or support of OpenStack libraries.

The first was oslo-incubator where as the name suggests, initial libraries were incubated. As this code matured it was refactored into standard libraries. Projects have either graduated, been incorporated elsewhere or been deprecated. While the Oslo Incubator has been removed of libraries in Mitaka, one of the goals of the Newton cycle is to see the adoption of Oslo libraries in all projects. We will be providing a series of blogs to detail the walkthrough and reviews of existing projects for reference.

Other libraries include:

  • oslosphinx is a sphinx add-on library that provides theme and extension support for generating documentation with Sphinx. The Developer Documentation, Release Notes, a number of the OpenStack manuals including the Configuration Reference and now the Nova API Reference rely on this library.

  • oslotest is a helper library that provides base classes and fixtures for creating unit and functional tests.
  • oslo-cookiecutter is a project that creates a skeleton Oslo library from a set of templates.

4. Proposed or deprecated libraries

Some libraries fall outside of these categories, such as oslo.rootwrap. This was a mature library for handling fine filtering of shell commands to run as root. This is now deprecated in favor of oslo.privsep which is a mechanism for running selected python code with elevated privileges.

pylockfile is a legacy (and adopted) inter-process lock management library that was never used within OpenStack.

The oslo.version is an example of a proposed library at present to help in using python metadata to determine versioning.

The Oslo team is also evaluating what other common code may be suitable for an Oslo library.

The meaning behind the Oslo Name

Each OpenStack project has some reason behind the name. Oslo is in reference to the Oslo Peace Accords and “bringing peace” to the OpenStack project.

Oslo is also the capital of Norway, and in Norway you can find Moose. The moose is our project mascot.

are you running KVM or QEMU launched instances?

A recent operators mailing list thread asked this question regarding the OpenStack user survey results of April 2016 (See page 39).

As I verified my own local multi-node devstack dedicated H/W environment with varying commands, I initially came across the following error (which later was found to be misleading).

$ virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking for device /dev/kvm                                         : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization)
  QEMU: Checking for device /dev/vhost-net                                   : WARN (Load the 'vhost_net' module to improve performance of virtio networking)
  QEMU: Checking for device /dev/net/tun                                     : PASS
   LXC: Checking for Linux >= 2.6.26                                         : PASS

This is an attempt to collate a list of varying commands collected from various sources, and the output of these in my Ubuntu 14.04 LTS environment.

# Are you running 64-bit architecture (0=bad; >0 is good)
$ egrep -c ' lm ' /proc/cpuinfo

# Does your processor support hardware virtualization (0=bad; >0 is good)
$ egrep -c '^flags.*(vmx|svm)' /proc/cpuinfo

# Are you running a 64-bit OS
$ uname -m

# Have I installed the right Ubuntu packages
$ dpkg -l | egrep '(libvirt-bin|kvm|ubuntu-vm-builder|bridge-utils)'
ii  bridge-utils                        1.5-6ubuntu2                          amd64        Utilities for configuring the Linux Ethernet bridge
ii  libvirt-bin                         1.2.2-0ubuntu13.1.17                  amd64        programs for the libvirt library
ii  qemu-kvm                            2.0.0+dfsg-2ubuntu1.24                amd64        QEMU Full virtualization

# Have packages configured user privileges
$ grep libvirt /etc/passwd /etc/group
/etc/passwd:libvirt-qemu:x:108:115:Libvirt Qemu,,,:/var/lib/libvirt:/bin/false
/etc/passwd:libvirt-dnsmasq:x:109:116:Libvirt Dnsmasq,,,:/var/lib/libvirt/dnsmasq:/bin/false

# Have I configured QEMU to use KVM
$ cat /etc/modprobe.d/qemu-system-x86.conf
options kvm_intel nested=1

# Have I loaded the KVM kernel modules
$ lsmod | grep kvm
kvm_intel             143630  3 
kvm                   456274  1 kvm_intel

# Are there any KVM related system messages
$ dmesg | grep kvm
[ 2030.719215] kvm: zapping shadow pages for mmio generation wraparound
[ 2032.454780] kvm [6817]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xabcd

# Can I use KVM?
$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

# Can I find a KVM device
$ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 May 11 14:15 /dev/kvm

# Have I configured nested KVM 
$ cat /sys/module/kvm_intel/parameters/nested

All of the above is the default output of a stock Ubuntu 14.04 install on my H/W, and with the correctly configured Bios (which requires a hard reboot to verify, and a camera to record the proof).

Some more analysis when changing the Bios.

$ sudo kvm-ok
INFO: /dev/kvm does not exist
HINT:   sudo modprobe kvm_intel
INFO: Your CPU supports KVM extensions
INFO: KVM (vmx) is disabled by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT),
      and then hard poweroff/poweron your system
KVM acceleration can NOT be used

When running a VirtualBox VM, the following is found.

$ sudo kvm-ok
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

Now checking my OpenStack installation for related KVM needs.

# Have I configured Nova to use KVM virtualization
$ grep virt_type /etc/nova/nova.conf
virt_type = kvm

# Checking hypervisor type via API's
$ curl -s -H "X-Auth-Token: ${OS_TOKEN}" ${COMPUTE_API}/os-hypervisors/detail | $FORMAT_JSON | grep hypervisor_type
            "hypervisor_type": "QEMU",
            "hypervisor_type": "QEMU",

# Checking hypervisor type via OpenStack Client
$ openstack hypervisor show -f json 1 | grep hypervisor_type
  "hypervisor_type": "QEMU"

Devstack by default has configured libvirt to use kvm.

Spinning up an instance I ran the following additional checks.

# List running instances
$ virsh -c qemu:///system list
 Id    Name                           State
 2     instance-00000001              running

# Check processlist for KVM usage
$ ps -ef | grep -i qemu | grep accel=kvm
libvirt+ 19093     1 21 16:24 ?        00:00:03 qemu-system-x86_64 -enable-kvm -name instance-00000001 -S -machine pc-i440fx-trusty,accel=kvm,usb=off...

Information from the running VM in my environment.

$ ssh cirros@

$ egrep -c ' lm ' /proc/cpuinfo

$ egrep -c '^flags.*(vmx|svm)' /proc/cpuinfo

$ uname -m

$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 6
model name	: QEMU Virtual CPU version 2.0.0

So, while the topic of the ML thread does indeed cover the confusion over OpenStack reporting the hypervisor type as QEMU when infact it does seem so but is enabling KVM via my analysis. I find the original question as a valid problem to operators.

And finally, this exercise while a lesson in understanding a little more about hypervisor and commands available, the original data was simply an operator error where sudo was needed (and not for other commands).

$ sudo  virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking for device /dev/kvm                                         : PASS
  QEMU: Checking for device /dev/vhost-net                                   : PASS
  QEMU: Checking for device /dev/net/tun                                     : PASS
   LXC: Checking for Linux >= 2.6.26                                         : PASS