IT Automation Part II: Ansible “Hello World” for Templating


This post is a continuation of my previous  post, where we have gone through a little “Hello World” example using Ansible, an IT automation tool. Last time we had performed SSH remote shell commands. This time, we will go though a little templating use case, where

  1. a shell script and a data file are created from Jinja2 templates,
  2. the files are uploaded,
  3. the shell script is performed on the remote target machine and
  4. the log is retrieved from the target machine.

We will see, how surprisingly easy those tasks are performed using Ansible and how input variables (like host name, order ID) come into play.

Posts of this series:

  • Part I: Ansible Hello World with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part II: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

2015.11.20-17_32_39-hc_001

 

Use Case

Our main goal today is to work with jinja2 templates and variables. For that, we look at following use case:

As an SaaS provider, I want to automatically configure an application via the application’s command-line base import mechanism.

The steps that need to be performed are:

  • create a host- and order-specific import script and an import data file
  • upload the import files
  • remotely run the script, which in turn imports the data file
  • retrieve the result (log)

1. Create import script

Prerequisites

If you have followed the instructions in part 1 of the Ansible Hello World example, then all prerequisites are met and you can skip this section. If not, you need to prepare the system like follows

  1. Install a Docker host. For that, follow the instructions “1. Install a Docker Host” on my previous blog.
  2. Follow the instructions 2. Create an Ansible Docker Image.
    download the Ansible Docker image and
  3. start the Ansible container using “docker run -it williamyeh/ansible:ubuntu14.04-onbuild /bin/bash”
  4. Create an inventory file /etc/ansible/inventory with the content
vi /etc/ansible/hosts

and add the following lines:

[vagranthosts]
192.168.33.10

Here, 192.168.33.10 is the IP address of the used Ubuntu VM, installed by Vagrant and based on a Vagrant box “ubuntu-trusty64-docker” from William-Yeh.

1.1 Create and upload a static import script

1. Create an import script on the Ansible machine like follows:

cd /tmp; vi importscript.sh

Add and save the following content:

#!/bin/sh
echo now simulating the import of the file /tmp/import-data.txt
# here you would perform the actual import...

2. In the same /tmp directory, create a playbook file “copy_file.yml” with following content:

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Copy File
  hosts: vagranthosts
  remote_user: vagranthost

  tasks:
  - name: copy file
    copy: src=/tmp/importscript.sh dest=/tmp/importscript.sh

3. Now we can run the playbook:

ansible-playbook -i /etc/ansible/hosts copy_file.yml

And we get:

root@930360e7db68:/tmp# ansible-playbook -i /etc/ansible/hosts copy_file.yml

PLAY [Copy Files] *************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [copy file] *************************************************************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=2 changed=1 unreachable=0 failed=0

Here, we have replaced the list of hosts by a reference to the inventory file.

4. And we verify that the file has been transferred:

vagrant@localhost ~ $ cat /tmp/importscript.sh
#!/bin/sh
echo now simulating the import of the file /tmp/import-data.txt
# here you would perform the actual import...

All in all, we have successfully uploaded a static file to the target machine.

1.2 Create and upload a templated import script

1. Create an import script template on the Ansible machine like follows:

cd /tmp; vi importscript.sh.jinja2

Add and save the following content:

#!/bin/sh
echo now simulating the import of the file /tmp/import-data-{{orderNumber}}.txt on the host={{ansible_host}}
# here you would perform the actual import...

Note, that we have introduced two variables “orderNumber” and “ansible_host”, which need to be considered at the time we run the ansible playbook.

2. In the same /tmp directory, create a playbook file “template_file.yml”

cp copy_file.yml template_file.yml; vi template_file.yml

with following content:

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Copy File
  hosts: vagranthosts
  remote_user: vagranthost

  tasks:
  - name: create file from template and copy to remote system 
    template: src=/tmp/importscript.shi.jinja2 dest=/tmp/importscript_from_template.sh

We have replaced the copy statement by a template statement.

3. Now we can run the playbook:

ansible-playbook -i /etc/ansible/hosts template_file.yml

but we get an error:

root@930360e7db68:/tmp# ansible-playbook -i /etc/ansible/hosts template_file.yml

PLAY [Copy Files] *************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [create file from template and copy to remote system] *******************
fatal: [192.168.33.10] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'orderNumber' is undefined", 'failed': True}
fatal: [192.168.33.10] => {'msg': "AnsibleUndefinedVariable: One or more undefined variables: 'orderNumber' is undefined", 'failed': True}

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
 to retry, use: --limit @/root/template_file.retry

192.168.33.10 : ok=1 changed=0 unreachable=1 failed=0

It seems like the template module is verifying that all variables in a template are resolved. Good: that feature prevents us from uploading half-resolved template files.

4. Now let us try again, but now we specify the orderNumber variable on the command line:

ansible-playbook -i /etc/ansible/hosts --extra-vars="orderNumber=2015-11-20-0815" template_file.yml

This time we get positive feedback:

root@930360e7db68:/tmp# ansible-playbook -i /etc/ansible/hosts --extra-vars="orderNumber=2015-11-20-0815" template_file.yml
PLAY [Copy Files] *************************************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [create file from template and copy to remote system] *******************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=2 changed=1 unreachable=0 failed=0

Successful, this time…

But wait a moment, wasn’t there a second variable “ansible_host” in the jinja2 template file, we have forgotten to specify at commandline? Let us see, what we get at the target system:

vagrant@localhost ~ $ cat /tmp/importscript_from_template.sh
#!/bin/sh
echo now simulating the import of the file /tmp/import-data-2015-11-20-0815.txt on the host=192.168.33.10
# here you would perform the actual import...

We find that ansible_host is a pre-defined Ansible variable, that automatically is set to the host as defined in the inventory file.

Are there other possibilities to define variables? Yes, many of them: the commandline (as just tested), the playbook, the host section within the playbook, the inventory file (for development versions >2.0 only!, …), playbook include files&roles, and many more; see e.g. a long precedence list of the variable definitions in the official Ansible variables documentation.

1.3 Testing variables in the Playbook

Let us test variables in the playbook’s host section:

1. For that we change the content of the file “importscript.sh.jinja2” like follows, introducing a new “who” variable:

#!/bin/sh
echo import performed by {{who}}
echo now simulating the import of the file /tmp/import-data-{{orderNumber}}.txt on the host={{ansible_host}}
# here you would perform the actual import...

And we add the “who” variable to the host in the playbook like follows:

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Copy Files created from templates
  hosts: vagranthosts
  vars:
    who: me
  remote_user: vagrant

  tasks:
  - name: create file from template and copy to remote system
    template: src=/tmp/importscript.sh.jinja2 dest=/tmp/importscript_from_template.sh

2. and we re-run the command

ansible-playbook -i /etc/ansible/hosts --extra-vars="orderNumber=2015-11-20-0815" template_file.yml

and we get:

PLAY [Copy Files created from templates] **************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [create file from template and copy to remote system] *******************
changed: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=2 changed=1 unreachable=0 failed=0

3. let us validate the content of the transferred file:

vagrant@localhost ~ $ cat /tmp/importscript_from_template.sh
#!/bin/sh
echo import performed by me
echo now simulating the import of the file /tmp/import-data-2015-11-20-0815.txt on the host=192.168.33.10
# here you would perform the actual import...

Perfect; success: all in all, we have created a template file with

  • pre-defined variables,
  • variables that are host-specific and
  • variables that are defined at runtime as an argument of the CLI command.

Note, that a variable cannot be defined under the task section (you will get the error message “ERROR: vars is not a legal parameter in an Ansible task or handler”, if you try to). As a workaround, if you want to use task-specific variables, you can create a playbook per task and define the variable under the host section of the playbook.

Note also, that it is considered as Ansible best practice to define host-specific variables in the inventory file instead of the playbook. Check out the documentation in order to find several ways to define variables in the inventory. However, be careful, since the Docker image is still on version 1.9.4 (at the time of writing this is the latest stable release) and specification of variables in the inventory file requires v2.0.

2. Upload an import data file, perform the shell script and download the result log file

1. In order to come closer to our use case, we still need transfer a data file, and execute the import script on the remote target. For that, we define:

vi /tmp/import-data.txt.jinja2

2. add the content

# this is the import data for order={{orderNumber}} and host={{ansible_host}}
# imported by {{who}}
Some import data

3. create a playbook named import_playbook.yml like follows:

cp template_file.yml import_playbook.yml; vi import_playbook.yml

with the content:

---
# This playbook uses the win_ping module to test connectivity to Windows hosts
- name: Copy Files created from templates
  hosts: vagranthosts
  vars:
    who: me
  remote_user: vagrant

  tasks:
  - name: create import script file from template and copy to remote system
    template: src=/tmp/importscript.sh.jinja2 dest=/tmp/importscript-{{orderNumber}}.sh
  - name: create import data file from template and copy to remote system
    template: src=/tmp/import-data.txt.jinja2 dest=/tmp/import-data-{{orderNumber}}.txt
  - name: perform the import of /tmp/import-data-{{orderNumber}}.txt
    shell: /bin/sh /tmp/importscript-{{orderNumber}}.sh > /tmp/importscript-{{orderNumber}}.log
  - name: fetch the log from the target system
    fetch: src=/tmp/importscript-{{orderNumber}}.log dest=/tmp

In this playbook, we perform all required steps in our use case: upload script and data, perform the script and retrieve the detailed feedback we would have got, if we had performed the script locally on the target machine.

4. run the playbook

ansible-playbook -i /etc/ansible/hosts --extra-vars="orderNumber=2015-11-20-0816" import_playbook.yml

With that we get the output:

root@930360e7db68:/tmp# ansible-playbook -i /etc/ansible/hosts --extra-vars="orderNumber=2015-11-20-0816" import_playbook.yml

PLAY [Copy Files created from templates] **************************************

GATHERING FACTS ***************************************************************
ok: [192.168.33.10]

TASK: [create import script file from template and copy to remote system] *****
changed: [192.168.33.10]

TASK: [create import data file from template and copy to remote system] *******
changed: [192.168.33.10]

TASK: [perform the import of /tmp/import-data-{{orderNumber}}.txt] ************
changed: [192.168.33.10]

TASK: [fetch the log from the target system] **********************************
ok: [192.168.33.10]

PLAY RECAP ********************************************************************
192.168.33.10 : ok=5 changed=3 unreachable=0 failed=0

5. On the remote target, we check the files created:

vagrant@localhost ~ $ cat /tmp/importscript-2015-11-20-0816.sh
#!/bin/sh
echo import performed by me
echo now simulating the import of the file /tmp/import-data-2015-11-20-0816.txt on the host=192.168.33.10
# here you would perform the actual import...
vagrant@localhost ~ $ cat /tmp/import-data-2015-11-20-0816.txt
# this is the import data for order=2015-11-20-0816 and host=192.168.33.10
# imported by me
Some import data
vagrant@localhost ~ $ cat /tmp/importscript-2015-11-20-0816.log
import performed by me
now simulating the import of the file /tmp/import-data-2015-11-20-0816.txt on the host=192.168.33.10

6. And on the Ansible machine, check the retrieved log file:

root@930360e7db68:/tmp# cat /tmp/192.168.33.10/tmp/importscript-2015-11-20-0816.log
import performed by me
now simulating the import of the file /tmp/import-data-2015-11-20-0816.txt on the host=192.168.33.10

Note that the file is automatically copied into a path that consists of the specified /tmp” base path, the ansible-host and the source path. This behavior can be suppressed with the variable flat=yes (see http://docs.ansible.com/ansible/fetch_module.html) for details.

Summary

We have shown, how easy it is to implement an IT automation use case, where a script and a data file are created from template, the files are uploaded to a remote target, the script is run and the command line log is retrieved.

Further Testing

If you want to go through a more sophisticated Jinja2 example, you might want to check out this blog post of Daniel Schneller I have found via google.


3 thoughts on “IT Automation Part II: Ansible “Hello World” for Templating

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s