10

Getting Started with OpenStack using DevStack


In this post, I will show how to install a DevStack-based demo OpenStack solution on an existing Ubuntu 14.04 server.

DevStack is an open source project with the goal to offer a development, test or demo environment for working with OpenStack. The compute nodes are based on QEMU, a software virtualization, so do not expect any high performance from this installation. However, it helps you to get acquainted with the handling of OpenStack, to get acquainted with OpenStack’s APIs and/or even to contribute to the OpenStack project.

In this post, we will install OpenStack on an existing Ubuntu server 14.04. I have tried to follow the official documentation, but it was easy to get lost there. The official documentation has led me to some challenges related to file permissions, which you can circumvent by following the step by step guide below.

Step 1: Install Ubuntu 14.04 Server

In my case, I have used Hyper-V and the Ubuntu Server 14.04 ISO for installing a high-performance Ubuntu server on my brand new monster Predator notebook with 64GB RAM. Hyper-V requires Windows 10 Pro license as a minimum, if you are not working with Windows Server licenses. However, you can also choose to install Ubuntu 14.04.x on physical hardware, on VMware vSphere, VMware Workstation, the old VMware Server 2.0.2 or on VirtualBox.

Note: there is a nice and simple way to install Ubuntu on VirtualBox using Vagrant, which I have described in step 1 “Install a Docker Host” in this post. The Docker host Vagrant box is an Ubuntu 14.04 machine and the Docker software does not hurt. The biggest challenge there is to successfully install VirtualBox.

The following Software needs to be installed in addition on Ubuntu:

sudo apt-get install openssh-server git

Step 2: Download DevStack from Git

Now we need to download DevStack using git:

git clone https://git.openstack.org/openstack-dev/devstack
cp devstack/samples/local.conf devstack/

Step 3 (optional): take snapshot of your Ubuntu virtual machine (if it is a virtual machine)

At this point, I have created a snapshot (checkpoint) of the VM on Hyper-V, so I can go back any time, the installation fails.

Step 4 (necessary?): copy and edit local.conf

In the moment, it is not clear, whether it is used during installation, but I have copied local.conf from devstack/samples to devstack, as the official documentation always talks about changing local.conf file :

cp devstack/samples/local.conf devstack/

And I have changed the content of devstack/local.conf like follows:

#local.conf
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
#...
# OV: commented out
# LOGFILE=$DEST/logs/stack.sh.log
#...
# OV: commented out
#LOGDAYS=2
# I have kept following lines:
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data

Step 5: create a non-root user called “stack” and log in as this user

Devstack does not run as root, so we need a non-root user. You might want to re-use an existing non-root user instead, but I recommend to create it like follows:

myname@ubuntu:~$ sudo devstack/tools/create-stack-user.sh
Creating a group called stack
Creating a user called stack
Giving stack user passwordless sudo privileges

The user stack has no password, so we only can log in via su as root. In my case, myname is a sudo user, so I have loged in via the command:

myname@ubuntu:~$ sudo su stack

Step 6: copy the git repository to the “stack” user’s home directory

This step is missing in the official documentation and had caused the installation script to fail due to wrong permissions. As user “stack”, navigate to the directory containing devstack and copy the directory to home directory:

cp -R devstack ~/

The devstack repository is not large, but if you do not want to copy it, you also could change the file permissions by using a chown -R stack:stack devstack.

Step 7: run the installation script

Now the devstack folder and its contents are owned by user stack and the installation script should run with no problems:

cd ~/devstack; ./stack.sh

-> here I was asked for the password again, even though it is defined in local.conf already. I have not investigated yet, whether the Step 4 above is necessary at all. Just re-type the password you have defined in Step 4.

After ~50 minutes, we see following last message of the installation:

2016-06-21 09:44:20.771 | stack.sh completed in 2946 seconds.

In this case, the command was running on a Hyper-V Ubuntu 14.04.3 server with only one core of the i7-6700QM CPU and only 6 Mbps Internet download bandwidth.

Step 8: log in to Horizon as admin

Unfortunately, there is no information given on what to do next. Therefore, I have had a look into stack.sh and I have found that Horizon should be available on URL http://hostname/dashboard/auth/login/?next=/dashboard/ where hostname is 192.168.178.111 in my case:

2016-06-21 (1)

Yes, perfect! Horizon is up and running.

Per trial and error, I have logged in as user “admin” with the password I have provided above and I have reached at the Identity -> Projects screen:

2016-06-21 (2)

Bingo! Time to play around!

🙂

Step 9: explore Compute Nodes

On Admin->Hypervisors, we can see that DevStack has added a single QEMU-based Hypervisor with 512MB RAM on the base ubuntu machine:

2016-06-21 (4)

Step 10: create a virtual machine / Instance

On Projects->Compute->Instances, click on “Launch Instance”:

2016-06-21 (5)

2016-06-21 (6)

and then on “Next”. Then we can choose the image, from which the instance will be built:

2016-06-21 (7)

Click on the Plus-sign next to the cirros image and click “Next”.

In the next screen, I have chosen the smallest flavor with 64MB RAM and no disk. Since the QEMU only provides 512MB, some of the other flavors should cause problems later on (can be tested later).

2016-06-21 (9)

Let us skip the other steps for now and immediately create the instance by clicking “Launch Instance” at the bottom of the screen (you might need to scroll down). You immediately should see a green box flashing with a success message: First, there will be a task “Spawning”, which will change to “None” in a minute or so. At the same time, the instance Power state will be “Running”.

2016-06-21 (10)

The instance is assigned a private IP address 10.0.0.2. However, is there a possibility to access the system?

Yes, from the Ubuntu system, you can ping the newly created instance:

stack@ubuntu:~$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=3.38 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.333 ms

However, without SSH key, we will not be able to SSH into the system; similar to AWS instances.

In 25 minutes, the German team is playing soccer in the European Contest, so I will stop now with a final dashboard screen:

2016-06-21 (11)

We have created and started a virtual machine (instance) on OpenStack.

Appendix A: Workaround for Openstack Login Error “An error occurred authenticating Please try again later”

After some days of OpenStack running with no interaction from my side, I could not log in to horizon anymore.

2016-07-03 (1)

I have found this YouTube video with information on how to fix this issue. However, I could not find any shell script named rejoin-stack.sh, as required. I have re-installed devstack instead. For that, I have issued following commands:

sudo su - stack
cd ~/devstack
./unstack.sh
sudo rm -rf /etc/keystone
./stack.sh

I had to remove the keystone folder with the rm -rf command in order to avoid an Operation not permitted failure during the execution of stack.sh. The downside of this solution is that it takes ~20 minutes of re-installation time again. At the end, I got following information:

This is your host IP address: 192.168.178.134
This is your host IPv6 address: ::1
Horizon is now available at http://192.168.178.134/dashboard
Keystone is serving at http://192.168.178.134/identity/
The default users are: admin and demo
The password: secret
2016-07-03 14:37:50.238 | WARNING:
2016-07-03 14:37:50.238 | Using lib/neutron-legacy is deprecated, and it will be removed in the future
2016-07-03 14:37:50.239 | stack.sh completed in 1082 seconds.

And I could log in again:

2016-07-03 (2)

4

IT Automation Part IV: Ansible Tower “Hello World” Example


This is part IV of a little “Hello World” example for Ansible, an IT automation (DevOps) tool. This time, we will get acquainted with Ansible Tower, a web front end for Ansible. The post has following content:

  • Quickest way of “installing” an Ansible Tower trial system (via Vagrant this time)
  • Step by step guide for a minimalistic “Hello World!” example, following closely the official documentation with following differences:
    • more insights on the dependencies of the steps (discussed with a simple dependency diagram)
    • shorter, simple page step by step tutorial with links to the official documentation at each step
  • Ansible Tower Benefits

Posts of this series:

  • Part I: Ansible Hello World with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part II: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

Why not “Installing” Ansible Tower via Docker?

In parts I to III of this blog series, I have made good experience downloading Docker images from Docker Hub instead of installing the software in question. This is, how we had “installed” Ansible and Saltstack. Also this time, I have tried to do so, but the attempts were less successful. I have tested following images:

  1. leowmjw/ubuntu-ansible-tower is a manual build and 8 months old at the time of writing. I had succeeded to accomplish almost all steps, but at the end, the launched jobs were stuck in the “pending” state and I could not find out, what is wrong.
  2. ybalt/ansible-tower is an automated build, but still 6 months old. The services seem to start, but first greeting I got was a “Fatal Error” welcome screen (maybe, since I did not map the certs private folder, but I did not want to mess around with certifications)

At this point, I had given up following the Docker path for now, even though there are many other public Ansible Tower images on Docker Hub I could test. Instead, I have found that installing Ansible Tower on a VirtualBox image by using Vagrant is a very convenient way of tackling the installation part. It is recommended for proof of concept purposes only, but it is an officially offered way of creating an Ansible Tower service; see this official documentation.

Installing Ansible Tower via Vagrant

Prerequisites

  • VirtualBox 5 is installed. You can get the software from here. See the Appendix A of part 1 of this series for some possible workarounds, if you have problems installing VirtualBox on a Windows system.
  • Vagrant is installed. Get the software from here.
  • Enough resources on the host system: additional 2 GB RAM and 2 vCPUs. However, the top command on my Notebook shows that the host system is running with less than 20% CPU load, although the Ansible Tower is running.

Step 1: install Ansible Tower via Vagrant and connect via SSH

Following this instructions, you can issue the following commands in order to download, start and connect to the Ansible Tower image:

$ vagrant init ansible/tower # creates the Vagrantfile
$ vagrant box add ansible/tower # optional; downloads the Vagrant box 

As an addition to the official documentation, I have added the second command. The download might take a long time,  and the next command will require your attention.

give you some time for a long coffee break, while the next command requires your immediate attention: I had been prompted for the Administrator password twice. If you fail to provide the password in time, you might come back from the coffee break and be surprised by an error with a message like: VBoxManage.exe: error: Failed to create the host-only adapter.

$ vagrant up --provider virtualbox # the --provider option is optional, since virtualbox is the default provider
$ vagrant ssh # connects to the machine

The following (or similar) content will be shown:

$ vagrant ssh
Last login: Sat Jun 11 00:16:51 2016 from gateway

  Welcome to Ansible Tower!

  Log into the web interface here:

    https://10.42.0.42/

    Username: admin
    Password: <some_random_password>

  The documentation for Ansible Tower is available here:

    http://www.ansible.com/tower/

  For help, email support@ansible.com
[vagrant@ansible-tower ~]$

Step 2: follow the official quick start guide (with minor changes)

We closely follow the official quick start guide with a minor adaption at step “4. Create the Credential” below. This is, because Vagrant does not allow for password login of the root user.

Step 2.1: Log into Ansible Tower

We now can connect via browser to http://www.ansible.com/tower/ and log in with the credentials:

2016-06-11 (1)

Step 2.2 Import a License

Since we are logged in the first time ever, we will be provided with a pop-up like follows:
ansible_tower_no-license
Click on “Get a Free Tower Trial License” and choose the appropriate license version. I have applied for the permanent free license for up to 10 nodes, and I have received an email from support@ansible.com with the license file. You can either drag&drop the license file to the browser, or you can copy the license file content into the corresponding field,  so the content of the License File field looks similar to:

{
    "company_name": "YourCompanyName", 
    "contact_email": "youremail@company.com", 
    "contact_name": "Your Name", 
    "hostname": "yourhostname", 
    "instance_count": 10, 
    "license_date": 2128169547, 
    "license_key": "your_license_key", 
    "license_type": "basic", 
    "subscription_name": "Basic Tower up to 10 Nodes"
}

After checking the “I agree …” checkbox and submitting the form, you will be rewarded with a License Accepted pop-up window:

ansible_tower_qs-licenseaccepted

After clicking OK, you will reach the Dashboard:

ansible_tower_qs-home-dashboard

Step 2.3: Add all elements needed for a minimum Hello World Example

Now: where to start? For that, let us have a look at the data model and the dependencies. The data model of Ansible Tower, as provided by the Ansible Tower documentation looks like follows:

ansible_tower_TowerHierarchy

Note that (at least for new versions of Ansible), users can be directly attached to an organization with no need to create a team. The same holds for inventory groups. Let us get rid of those and perform a “Hello World” in following simplified model:

2016.06.13-18_07_15-hc_001

Okay, the figure might look slightly more complex than the one in the Ansible Tower documentation. This is, because I have added all mandatory parent to child dependencies, similar to the Unified Modeling Language notation.

Note: In the Ansible Tower model, each child has only a single parent, while each parent can have an arbitrary number of children. The only exception I have found so far are the Projects: each project can be assigned to one or more organizations.

As in the real world, you need to follow the arrows, if you want to reach your goal:

  1. Users, inventories and projects depend on organizations. However an organization named “default” is already pre-installed, so we do not need to do anything here.
  2. we need to create a user before we can create a credential
  3. we need to create an inventory before we can create a host
  4. we need to add a playbook directory, before we can define a project
  5. we need to create a playbook file, before we can define a job template. In addition, a credentials, an inventory and a project needs to be available.
  6. A job can be launched from a job template panel only

The official quick start guide goes through this model like follows:

2016.06.13-18_45_45-hc_001

Let us follow the same order. Nothing needs to be done for the organization  (“Step 0”): a “default” organization exists already. So, let us start with Step 1:

  1. Create a User
    Click on 2016.06.13-18_52_06-hc_001 and then on Users and then on 2016.06.13-18_53_09-hc_001 and add and save following data:
    ansible_tower_qs-organizations-create-user-form
  2. Create an Inventory
    Click on Inventories, then 2016.06.13-18_53_09-hc_001 and and and save following data:
    ansible_tower_qs-inventories-click-to-save-new-inventory
  3. Create a Host
    Click on Inventories, then on the newly created inventory, and then on 2016.06.13-18_53_09-hc_001 on the far right. Add and save following data:
    ansible_tower_qs-inventories-host-properties-form
  4. Create a Credential
    Click on 2016.06.13-18_52_06-hc_001 and then on Credentials and then on 2016.06.13-18_53_09-hc_001 . Here we deviate a little bit from the official documentation. The documentation asks to add and save following data:
    ansible_tower_qs-credentials-check-ssh-password-ask-at-runtimes-for-new-credential
    However, we have installed Ansible Tower using Vagrant, and per default, the root user cannot log in with a password. Therefore we will use the vagrant user and log in with the private SSH key like follows:
    2016.06.13-19_31_35-hc_001
    The private SSH key can be found on the host system, from where you have started the ansible tower. From the directory containing the Vagrantfile, you need to navigate to CODE .vagrant/machines/default/virtualbox CODE and open the file CODE private_key. The content needs to be cut & paste into the Private Key field above.
  5. Create a Playbook Directory
    Start an ssh session to the Ansible Tower host. In our case, this is done by issuing
    CODE vagrant ssh CODE
    as shown above. Issue the command
    CODE sudo mkdir /var/lib/awx/projects/helloworld CODE
  6. Create a Playbook
    In the ssh session, start an editor session, e.g.
    CODE vi /var/lib/awx/projects/helloworld/helloworld.yml CODE:
    cut&paste following content into the file:

    ---
    - name: Hello World!
      hosts: all
    
      tasks:
    
      - name: Hello World!
        shell: echo "Hi! Tower is working!"
  7. Create a Project
    Click on Projects and on 2016.06.13-18_53_09-hc_001 and add and save following data:
    ansible_tower_qs-projects-create-project-form
  8. Create a Job Template
    Click on Job Templates and on 2016.06.13-18_53_09-hc_001 and add and save following data:
    ansible_tower_qs-job-templates-form
    Note that you need to use the other credential named “Vagrant User per SSH Key” we have created in Step 4 above.
  9. Create (Launch) a Job
    Click on Job Templates and on 2016.06.13-19_22_38-hc_001 in the corresponding line of the job template. In case of a password credential, add the password into the prompt (not needed for SSH key authentication):
    ansible_tower_qs-jobs-enter-ssh-password-to-launch-job

This will lead to following result:

ansible_tower_qs-jobs-home-showing-successful-job

The command line output can be checked by clicking the 2016.06.13-19_28_00-hc_001 button:

ansible_tower_qs-job-results-stdout

Why Ansible Tower?

When we compare the many steps needed to run a simple “Hello World” example we could have accomplished on the command-line in less than half of the time (see e.g. part 1 of this blog series), we could ask ourselves, why someone would like to use Ansible tower at all. The main reasons (as I see it) are like follows:

  • Fine-grained role-based access control: in a real-world example, most steps above would be performed by an Ansible Tower administrator. As an example, you could define a team that is allowed to launch certain jobs only. Moreover, Ansible Tower seems to allow a quite fine-grained access control on what a team or user is allowed to do with Inventories and Job Templates (see 2016.06.13-20_33_32-hc_001 -> Teams -> Permissions):2016.06.13-20_27_22-hc_001
  • Unlike StaltStack, Ansible (without Tower) does not have any notion of immediate or scheduled background jobs. In addition, it does not automatically record the execution time and results of any performed jobs, making audit trails a complex task. Ansible Tower is filling this gap.
  • Nice and easy graphical handling including statistics etc.
  • Ansible Tower offers a modern RESTful API, which allows it to integrate with existing tools and processes

Summary

In this blog post we have shown how Vagrant can help to quickly set up a local Ansible Tower trial. We have discussed the Ansible Tower object model in its simplest form, and could explain, why and which minimal steps are needed to perform a simple “Hello World”. At the end, we have compared Ansible Tower with simple command-line based Ansible (without Tower).

0

CoreOS Cluster Discovery & Troubleshooting


This blog post explores CoreOS cluster discovery problems and how to troubleshoot them. This blog post is part 8 of my DDDocker (a dummy’s docker diary) series. Since this post is one of my more popular blog posts on linkedIn, I have decided to move it to my WordPress site.

In the DDDocker (7) linkedIn post I had shown how to easily set up a CoreOS cluster using Vagrant. However, I had experienced problems with the health of the CoreOS cluster, caused by the presence of a HTTP proxy. This post is exploring on CoreOS cluster health and troubleshooting steps.

Based on what we have learned in DDDocker (7)  and the current post, we can install a healthy CoreOS cluster in less than ten or 15 minutes following this post.

Version History

v1 (2015-07-21) : Manual Cluster Discovery & Troubleshooting
v2 (2015-07-27): Added the Appendix “Healthy Docker (CoreOS) Cluster in less than 10 Minutes”
v3 (2015-08-19): Moved the Appendix “Healthy Docker (CoreOS) Cluster in less than 10 Minutes” to an independent blog post on WordPress
v4 (2016-06-03): Article moved to WordPress (here) and re-written the introduction

Cluster Discovery

In the DDDocker (7) linkedIn post I have created a CoreOS cluster with 3 cluster nodes. This is our starting point for the current post:

2015.07.15-17_17_50-hc_001

Before following the instructions in the CoreOS Quick Start Guide, section “Process Management with fleet” let us test, whether the cluster members are aware of each other:

Checking the Cluster Health

We will see below that the cluster is not healthy, since I have deployed

Connect to the first node using “vagrant ssh core-01”, or via putty like described in the DDDocker (7) post. Within the window, type:

fleetctl list-machines

Problems with Cluster Initialization behind a HTTP Proxy

In my case, this does not work:

2015-07-17_164244_capture_053

The reason is that fleet depends on etcd and the nodes cannot connect to the public etcd discovery server on https://discovery.etcd.io, as defined in the user-data file.

This can be seen with

journalctl -b -u etcd | less

where we see a note saying “failed to connect discovery service[https://discovery.etcd.io/e86ff40c2076ccf901fdc0681f11417e]”:

The problem: I have performed the installation behind a HTTP proxy and etcd discovery does not yet support communication via HTTP proxies (see here change request for CoreOS).

This is, why the discovery server has not discovered any nodes on my token, which I can see in a browser onhttps://discovery.etcd.io/e86ff40c2076ccf901fdc0681f11417e:

Workaround: connect to the Internet temporarily

There are two possible workarounds/solutions:

  1. connect to the Internet temporarily (for discovery process only)
  2. add a local etcd discovery agent as suggested on http://stackoverflow.com/questions/25019355/coreos-fleetctl-list-machines-show-error.

In my case, I have chosen 1, using a hotspot function of my mobile phone: only seconds after having connected to the Internet, the cluster nodes are seen on the public discovery agent:

and the cluster turns our to be “healthy”:

and also fleetctl works fine on core-01:

All cluster nodes are visible and healthy. The same picture is seen on all 3 cluster nodes. And it does not change, if I stop the Internet connection and hide behind a HTTP proxy again.

Summary

We have seen that cluster discovery of a CoreOS cluster depends on the etcd discovery service. The easiest way to set up etcd is to use the public discovery service on https://discovery.etcd.io. However, you need to make sure that each cluster node can reach this service without the need to pass a HTTP proxy, since etcd discovery does not support proxies yet. Even if curl or wget works, etcd discover will still fail.

There are two ways to resolve the etcd discovery issue:

  1. –shown in this post–
    temporarily connect to the Internet
  2. –not tested on my side–
    create your own etcd discovery service, as described onhttp://stackoverflow.com/questions/25019355/coreos-fleetctl-list-machines-show-error. For etcd newbies like me, also https://github.com/coreos/etcd/issues/1404 seems to give some insight.

The temporary connection to the Internet worked like a charm, and the cluster is still healthy after disconnecting the cluster from the Internet.

Further Reading