IT Automation Part III: Saltstack “Hello World” Example


This blog post will explore basic capabilities of Salt by going through a hands-on “hello world” IT automation example. Moreover, those basic capabilities are compared for Salt vs. Ansible.

This post is (almost) self-contained, and there is no need to read part I and part II of this series first.

Salt is a quite recent IT automation tool (project started in 2011), which was built to scale well beyond tens of thousands of servers. In this Hello World example, we will investigate, how to run Salt in a Docker container and how to perform some simple remote shell commands on the target machines.

Posts of this series:

  • Part I: Ansible Hello World with a comparison of Ansible vs. Salt vs. Chef vs. Puppet and a hello world example with focus on Playbooks (i.e. tasks), Inventories (i.e. groups of targets) and remote shell script execution.
  • Part II: Ansible Hello World reloaded with focus on templating: create and upload files based on jinja2 templates.
  • Part III: Salt Hello World example: same content as part I, but with Salt instead of Ansible
  • Part IV: Ansible Tower Hello World: investigates Ansible Tower, a professional Web Portal for Ansible

2015.12.04-13_48_53-hc_001

Why Salt?

Salt is a newcomer in the game of IT automation. Salt is competing against Ansible, Chef, Puppet and maybe CFEngine (in reverse order of begin of project). You will find recent comparisons of Puppet vs. Chef vs. Ansible vs. Salt in the articles here you and here. A more comprehensive, but older article can be found on this InfoWorld article from 2013.

Most of the articles mention that Salt’s strength is its scalability. However, most of the articles do not mention following (I think) important advantages of Salt:

  • Salt is is Apache 2.0 licensed, whereas Ansible is GPL licensed. Therefore, Salt can be used by developers of commercial software without fear that they could be forced to publish the source code of all their work.
  • Salt comes with a REST interface (example) a that is independent of the Web portal, whereas the REST interface is part of Ansible Tower, which in turn is a commercial product.
  • Salt has the better step by step guides. However, for both, Ansible and Salt, the documentation sometimes fails to make clear, which feature is available in the development version only, which is annoying, if you are using the latest stable version, as we will do in this article.

You will find some more practical differences we will have found out during our “hello world” journey on the Summary section.

Goal

Our main goal today is to get familiar with Salt’s way of defining provisioning tasks and target hosts and groups, similar to what we have done in part 1 of this series.

Provisioning tasks are defined as

Targets and groups are defined within

For a in-depth comparison of targets and groups within Ansible vs. Salt, see Summary section. We will see that all in all, the target&group definition and search capabilities are much more elaborated in case of Salt than they are in case of Ansible.

Use Case

The example use case, we will explore, is:

As an IT administrator, I want to manage target machine groups in order to run arbitrary shell commands on a group of target machines with a single CLI command on the management machine.

The steps that need to be performed are:

  • prepare and test that the automation tool can connect to the target machines
  • define target groups and shell commands and centrally save the information for later usage
  • remotely run the shell commands on the specified targets

We will follow closely the Saltstack Fundamentals guide, with the difference that we will use lightweight Docker images as a replacement of the much larger Vagrant Virtualbox machines. For me, this is reducing download times and the needed resources on my notebook. Note, that I did not choose the saltstack official-looking Docker images, since they are lacking any documentation. Instead, we will use this Docker image from jacksoncage that has a good documentation (thanks to Jackson!).

Prerequisites

  • Install a Docker host. On Windows and iMac, you will find a convenient way of doing so described in the Chapter “Install a Docker host” of part I of this blog post series.

Download and Start Salt

Once, you have installed Docker, downloading, installing and running a Salt master is done by a single docker command on the Docker host. However, in order to allow the configuration files to be stored on the Docker hosts, we create and enter a directory called “master” before:

Step 1: download and install a Salt master with a single Docker command:

mkdir ~/master; cd ~/master; 
docker run -i -t --name=master -h master -p 4505 -p 4506 \
   -p 8080 -p 8081 -e SALT_NAME=master -e SALT_USE=master \
   -v `pwd`/srv/salt:/srv/salt:rw jacksoncage/salt

This is done in an interactive session, so you need to open a second SSH session for connection to the Docker host and issue the command

docker exec -it master bash

In order to connect to the command-line of the Salt master.

Step 2: download and install a Salt target (=”minion”) with a single Docker command:

Now, on a third Docker host SSH session, we start a target machine, a so-called minion:

mkdir ~/minion1; cd ~/minion1;
docker run -i -t --name=minion1 -h minion1 --link master:master -p 4505 -p 4506 \
 -p 8080 -p 8081 -e SALT_NAME=minion1 -e SALT_USE=minion \
 -v `pwd`/srv/salt:/srv/salt:rw jacksoncage/salt

The resulting log should look something like:

vagrant@localhost ~/minion1 $ docker run -i -t --name=minion1 -h minion1 --link master:master -p 4505 -p 4506 -p 8080 -p 8081 -e SALT_NAME=minion1 -e SALT_USE=minion -v `pwd`/srv/salt:/srv/salt:rw jacksoncage/salt
INFO: Starting salt-minion with log level info with hostname minion1
[INFO ] Setting up the Salt Minion "minion1"
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO ] Generating keys: /etc/salt/pki/minion
[INFO ] Authentication with master at 172.17.0.5 successful!
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO ] Added mine.update to schedular
[INFO ] Added new job __mine_interval to scheduler
[INFO ] Minion is starting as user 'root'
[INFO ] Starting pub socket on ipc:///var/run/salt/minion/minion_event_c5a7daa544_pub.ipc
[INFO ] Starting pull socket on ipc:///var/run/salt/minion/minion_event_c5a7daa544_pull.ipc
[INFO ] Minion is ready to receive requests!
[INFO ] Running scheduled job: __mine_interval

The log shows, that the minion is automatically generating keys and is authenticating with the master. Nice feature. We note:

For Salt you need to install an agent but for Ansible you need to manually add public SSH keys. What is better?

Note that Salt is rewarding your agent installation effort (see Appendix A) with an automated key creation and distribution system. This is superior to Ansible’s requirement to manually import the public SSH key to all targets.

Step 3 (optional): perform your first remote task issued from command-line of the master

Now we are ready to issue our first remote command. On the docker exec session on the master, just enter:

salt 'minion1' disk.usage

and see, what happens:

root@master:/# salt 'minion1' disk.usage
minion1:
    ----------
    /:
        ----------
        1K-blocks:
            41251136
        available:
            12871844
        capacity:
            68%
        filesystem:
            none
        used:
            26643380
    /dev:
        ----------
        1K-blocks:
            766904
        available:
            766904
        capacity:
            0%
        filesystem:
            tmpfs
        used:
            0

Cool! You have just performed your first remote task via Salt.

Step 4 (optional): perform your first remote shell command issued from command-line of the master

While it is recommended to make use of Salt’s large library of commands, we can always revert back to the execution of shell commands like follows:

salt 'minion1' cmd.run 'df'

In this case, we will get something like:

root@master:/# salt 'minion1' cmd.run 'df'
minion1:
    Filesystem                                             1K-blocks     Used Available Use% Mounted on
    rootfs                                                  41251136 26644516  12870708  68% /
    none                                                    41251136 26644516  12870708  68% /
    tmpfs                                                     766904        0    766904   0% /dev
    shm                                                        65536        0     65536   0% /dev/shm
    tmpfs                                                     766904        0    766904   0% /sys/fs/cgroup
    /dev/disk/by-uuid/3af531bb-7c15-4e60-b23f-4853c47ccc91  41251136 26644516  12870708  68% /srv/salt
    tmpfs                                                     766904        0    766904   0% /proc/kcore
    tmpfs                                                     766904        0    766904   0% /proc/latency_stats
    tmpfs                                                     766904        0    766904   0% /proc/timer_stats

So, we have executed our first arbitrary shell command.

Step 5 (optional): perform our hello world command similar like we did in Part I using Ansible:

Let us now perform the same command we had performed on Ansible with:

ansible all -i '192.168.33.10,' -u vagrant -m shell -a "echo hello world\! > hello_world"

For Salt, this translates to:

salt 'minion1' cmd.run "echo hello world\! > hello_world"

Since the command does not produce any output on STDOUT, we only get:

minion1:

But where will we find the file?

Per default, Salt runs as root in /root. This can also be found out via:

salt 'minion1' cmd.run 'whoami;pwd'

Instead of logging into minion1, we also can check the content of the file remotely:

salt 'minion1' cmd.run 'cat /root/hello_world'

Now we get:

minion1:
    hello world!

Success!

Step 6: perform our hello world command as a non-root user:

In part 1 of this series, we have used Ansible to perform a command as a non-root user like follows:

ansible all -i '192.168.33.10,' -u vagrant -m shell -a "echo hello world\! > hello_world"

Let us try to perform the corresponding command on Salt. For that, we need to create a user on the minion:

Step 6.1: create a user

The Docker images do not come with a salt user; Salt is run as root. We do not want to perform all tests as root. Therefore, let us create a test user:

salt minion1 user.add test

and you can check the list of users with:

salt minion1 user.list_users
Step 6.2: perform a command as a non-root user

Now we try to execute the command as different user:

salt 'minion1' cmd.run runas=test 'whoami'

with this, we get:

minion1:
    test

Side discussion about the documentation issue: Now I understand why people tend to complain about Salt’s documentation: I had quite a long Odyssey to find the runas variable we have used above. The official documentation of the cmd module does not mention the runas variable. I had only found the user variable, but it did not work the desired way (maybe it is meant as the local user, not the remote user?). This has lead me to upgrade the system to the latest stable salt version 2015.5.3 (Lithium), which is a real challenge (see Appendix C below). And this did not help at all. I had considered other workarounds like using sudo -u test <command>, but this workaround has its own challenges, e.g. if you want to redirect the output of a command to a file as a non-root user. Let us forget about upgrades and workarounds for now. We have found the real solution…

Now let us perform the same command as we did in Part 1:

salt 'minion1' cmd.run runas=test "echo hello world\! > hello_world"
salt 'minion1' cmd.run runas=test "cat /home/test/hello_world"

and we get the expected output from the second command:

minion1:
    hello world!

Perfect, that works!

Defining Tasks, so-called States

In Part 1 of the series, we have created an Ansible playbook with following content:

---
# This playbook will write "Hello World!" to the file hello_world
- name: Echo
  hosts: 192.168.33.10
  remote_user: vagrant 

  tasks: 
  - name: echo 
    shell: echo Hello World! > hello_world

How does this translate to Salt? Salt does not define tasks, but it defines desired states. However, this is a matter of semantics only: with Ansible, after you have run a playbook with the task “upgrade”, the state of the system is upgraded. In Salt, you define a state “upgraded” and if you apply this salt state, you also will receive a system that is upgraded.

Step 7: create a state file (corresponds to an Ansible playbook file):

Our Ansible example above will translate to following state file we save on /srv/salt/hello_world.sls:

# This state file will write "Hello World!" to the file hello_world as user test
run_echo_hello_world:
  cmd.run:
    - name: echo Hello World > hello_world
    - user: test

Note: in the current version of Salt (tested 2015.5.3 Lithium) there is an inconsistency between the variables runas and user: for  cmd.run on command-line you need runas (and user is ignored), whereas in the state file, you need to specify the user and (runas is ignored). Try it out. If you want to be sure, you can set both, user and runas to the same value for both, the command line and the state file.

Step 8: apply the state file and check the result:

Now we apply the state file:

salt 'minion*' state.apply hello_world

and we check the file(s) with:

root@master:/# salt 'minion*' cmd.run runas=test 'ls -l hello_world; cat hello_world'  
minion1:
    -rw-r--r-- 1 test test 13 Nov 27 18:06 hello_world
    Hello World!

Perfect, that works now!

Targets and Groups

Salt offers many possibilities to choose your targets. You can explicitly specify the salt minion name like in the command

salt 'minion1' test.ping

Globbing allows us to perform the same command on different minions. E.g. to choose all minions whose name start with ‘minion’ we issue the command:

salt 'minion*' test.ping

Instead we also can use regular expressions:

salt -E 'minion[0-9]' test.ping

list the minion names:

salt -L 'minion1,minion2' test.ping

or even choose per IP address or per subnet:

salt -S '172.17.0.0/24' test.ping

The A very flexible possibility to choose the targets is by using so-called grains. To choose all minions running Ubuntu we issue the command:

salt -G 'os:Ubuntu' test.ping

Farther below, we will define our own grain in order to choose the right targets.

Last but not least, we can combine all target types using and and or operators like follows:

salt -C 'G@os:Ubuntu and minion* or S@172.17.0.0/24' test.ping
or
salt -C '( G@os:Ubuntu and minion* ) or S@172.17.0.0/24' test.ping

where the parenthesis are not mandatory in this case, since the operator ‘and’ takes precedence before ‘or’.

Note that the whitespaces between parenthesis and expressions are required: e.g. '(G@os:Ubuntu and minion*) or S@172.17.0.0/24' would not work the way expected.

Note that the fundamentals get started example is wrong here: it has specified an example “S@192.168.50.*" which does not work and should be replaced by S@192.168.50.0/24. I have sent a question to the salt-users google group to find out how to report this error.

Now we have understood how to choose targets on the command line, let us define something that comes close to the inventory file of Ansible. On part I of this series, we have defined a group called “vagranthosts” in an inventory file /etc/ansible/hosts with the content

[vagranthosts]
192.168.33.10

Step 9: create a node group:

In case of Salt, we can do something similar wit did in case of Ansible by using so-called Node Groups. Node Groups are something like saved node searches and are specified in the /etc/salt/master configuration file:

nodegroups:
  myglobbinggroup: 'minion*'
  myregexgroup:    'E@minion[0-9]'
  mysubnetgroup:   'S@172.17.0.0/24'
  mylistgroup:     'L@minion1,minion2'
  mygrainsgroup:   'G@os:Ubuntu'
  mycompoundgroup1: 'G@os:Ubuntu and ( minion* or S@172.17.0.0/24 )'

Starting from version 2015.8.0, the nodegroup can also be used in compound nodegroup expressions, e.g.

  # requires version >=2015.8.0:
  mycompoundgroup2: 'G@os:Ubuntu and ( N@myglobbinggroup or S@172.17.0.0/24 )'

Note also that there seems to be a problem with the example group4 on the nodegroups documentation:  If you use it, it leads to the error message “‘list’ object has no attribute ‘split'” (tested with version 2015.5.3). I have not tested it with the most recent development version >=2015.8.0, though:

  # does not work in v2015.5.3 (not tested with other versions yet):
  group4:
    - 'G@foo:bar'
    - 'or'
    - 'G@foo:baz'

Step 10 (optional): test node groups:

Now we can specify the group on the command line. All of the following commands apart from the last one should have the same results:

salt -N 'myglobbinggroup' test.ping
salt -N 'myregexgroup' test.ping
salt -N 'mysubnetgroup' test.ping
salt -N 'mylistgroup' test.ping
salt -N 'mygrainsgroup' test.ping
salt -N 'mycompoundgroup1' test.ping
# the next one may fail, as pointed out above:
salt -N 'mycompoundgroup2' test.ping

Step 11: use node group to finish our use case:

Now let us perform our provisioning task to come closer to our use case, namely to perform an echo hello world, redirect it into a file and show that the file content is the one expected.

Step 11.1: cleaning:

First let us make sure the hello_world file is removed on the minion:

salt -N 'myregexgroup' cmd.run runas=test 'ls -l hello_world; rm hello_world; ls -l hello_world'

You should get a one or two messages like:

   ls: cannot access hello_world: No such file or directory

Step 11.2: creating the hello_world file using the pre-defined state:

We already have defined the state and the group, so we can apply the state like follows:

salt -N 'myregexgroup' state.apply hello_world

Note that the state had defined that the file is created as user “test”. We can verify the content of the file by issuing the following command:

salt -N 'myregexgroup' cmd.run runas=test 'ls -l hello_world; cat hello_world'

We will get:

minion1:
    -rw-r--r-- 1 test test 13 Dec  3 12:28 hello_world
    Hello World!

Bingo!

Now we have used Salt to perform the same provisioning task as we had in part I using Ansible.

Summary: Salt vs. Ansible again

So, what did we learn? We have performed the same “hello world” tasks on Salt, as we did for Ansible in part I:  namely performing a remote shell command that has created a file. For that we have defined a Salt state, which corresponds to a playbook in Ansible. For defining the user and target (group), we have configured a node group on the /etc/salt/master configuration file, which corresponds to an Ansible inventory file.

Salt States vs. Ansible Playbooks

Salt states and Ansible playbooks are very much alike: both group a set of smaller task into a named task group. The difference is that the naming of the task group is associated with a summary task name (e.g. “create user” in case of Ansible playbooks, while the naming will be associated with a to be reached state (e.g. “user created”) in case of Salt state files.

Salt Node Groups vs. Ansible Inventory

Targets and groups are implemented quite differently in Ansible and Salt:

  • In Ansible, target groups within an inventory file are listing individual targets (even if name ranges like ‘node[01:10]’ are allowed), while
  • Salt Node Groups in the master configuration file work more like an advanced target search like “look for all connected targets with Operating System Ubuntu that have IP addresses in Subnet 172.0.0.0/24”

So, Salt Node Groups offer many more possibilities to define target groups than Ansible Inventories with the drawbacks that

  • there is no error message in Salt, if the target you want to update is not reachable at time of provisioning attempt. The unreachable target is just ignored, even if we make use of explicit target lists.
  • node groups are defined in the master configuration file, which is less flexible than Ansible Inventory files, whose path can be chosen at runtime using the ansible -i switch.

Salt Documentation vs. Ansible Documentation

Salt has better official step by step guides than Ansible and both, Salt and Ansible have their challenges with documentation. E.g. in case of Salt I have detected errors in the step by step guides and those do not seem to be under version control based on GitHub. And for both, Ansible and Salt, I have spent a lot of time finding out that a described feature I wanted to test is not available in the latest stable release, but requires an upgrade to a development release.

Ease of getting started with Salt vs. Ansible

In had expected that Salt is more complex to get started with, since it requires the installation of a Salt agent on the target machines (minions). However, Ansible had its own challenges, since it largely relies on SSH public keys that need to distributed manually.

A great plus: background operations and RESTful interface of Salt

If I want to get an IT automation tool with

  • background operations (see Appendix B) and a
  • RESTful interface,

without the need to buy commercial software, Salt seems to be a good choice, while Ansible does not offer those freeware possibilities. However, if you are looking for a professional, commercial alternative, you might want to test Ansible Tower, which allows you to create job templates via Web UI, gives RESTful access to many pre-defined objects like inventories, users, teams, projects, etc. See here the data sheet of Ansible Tower.

Note: there is a minor drawback of Ansible Tower: playbooks cannot be manipulated via the Web UI or the REST interface. The recommendation is to manage playbooks via Git.

Coming back to Salt’s background jobs and REST API, it looks like

  • the Salt’s background jobs do not seem to have in-built error reporting and retry mechanisms implemented: it seems to be the task of the administrator to poll and evaluate the result of background jobs or to make sure that a failed job will cause a Administrator defined action.
  • There is no pre-defined REST interface to perform tasks. It is the task of the Administrator to create the web hooks. See this example on how you can use Salt’s rest_cherrypy module to create web hooks with the goal to perform pre-defined tasks. And it does not look very RESTful, what we get at the end: a web hook that allows to POST variables to, which will be used to perform a set of tasks. There are no objects and there are no CRUD methods (create/read/update/delete) to manipulate those objects.

The license Topic…

Last but not least, Salt is Apache 2.0 licensed and you potentially can integrate Salt tightly into your (commercial) software without being forced to publish the source of your software.

Appendix A: Installing Salt on Target Machines (Minions)

Salt requires Salt agent to be installed on the target machines (a.k.a. minions). For most types *nix of targets, the installation can be bootstrapped with shell scripts found on git. I have not tested it yet, but in theory, installing the latest salt-master development build on Ubuntu should be as easy as typing following commands on the Linux shell:

# the next 2 commands are only needed in case you are behind a HTTP proxy:
# (adapt the http proxy name/IP address and port to fit to your environment)
export http_proxy=http://proxy.company.com:8080
export https_proxy=http://proxy.company.com:8080
# install curl, if not already installed (test with "which curl"):
apt-get update
apt-get install curl

# download the salt bootstrap shell script:
curl -o install_salt.sh -L https://bootstrap.saltstack.com
# install salt-master development version:
sudo sh install_salt.sh -M -N git develop
# or install salt-minion development version:
sudo sh install_salt.sh git develop

Appendix B: Running Jobs in the Background

Ansible offers background jobs, but only as part of the commercial web portal product “Ansible Tower”. For Salt, background jobs are integral part of the core and can be initiated and managed from command line (and REST??).

salt 'minion1' --async cmd.run 'df'

result:

root@master:/# salt 'minion1' --async cmd.run 'df' 
Executed command with job ID: 20151126092909642966

Retrieve the result of the job:

root@master:/# salt 'minion1' --async cmd.run 'df'
Executed command with job ID: 20151126092909642966
root@master:/# salt-run jobs.list_job 20151126092909642966
[WARNING ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
Arguments:
    - df
Function:
    cmd.run
Minions:
    - minion1
Result:
    ----------
    minion1:
        ----------
        return:
            Filesystem                                             1K-blocks     Used Available Use% Mounted on
            rootfs                                                  41251136 26576440  12938784  68% /
            none                                                    41251136 26576440  12938784  68% /
            tmpfs                                                     766904        0    766904   0% /dev
            shm                                                        65536        0     65536   0% /dev/shm
            tmpfs                                                     766904        0    766904   0% /sys/fs/cgroup
            /dev/disk/by-uuid/3af531bb-7c15-4e60-b23f-4853c47ccc91  41251136 26576440  12938784  68% /srv/salt
            tmpfs                                                     766904        0    766904   0% /proc/kcore
            tmpfs                                                     766904        0    766904   0% /proc/latency_stats
            tmpfs                                                     766904        0    766904   0% /proc/timer_stats
StartTime:
    2015, Nov 26 09:29:09.642966
Target:
    minion1
Target-type:
    glob
User:
    root
jid:
    20151126092909642966

offers the possibilities to run jobs asynchronously on the command line.

Appendix C: Upgrade Salt via Salt

Don’t do it!

If you insist to do it, please consider the following topics:

  • Upgrading Salt is not needed for accomplishing the tasks shown in this blog post
  • Upgrading Salt via Salt is a real challenge, because
    • upgrading Salt requires a restart of the salt-minion service, which will cause the remote salt upgrade process to hang
    • upgrading Salt did not work using the pkg.install module in my case, and remote upgrade using apt-get install cannot work, since the administrator is asked some questions, even if the -y flag is set.
  • If you Upgrade the Master of the used Docker master image, then the Minions are not compatible anymore. Old Minions might still work (after a restart of the salt-minion process), but if you spin up a new minion from the docker image (which has the old version), it fails to connect to the master.

So, fpr performing this hello world and beyond, upgrade Salt only, if absolutely necessary and better perform the upgrade locally on the system (or use another IT automation tool like Ansible to do so).

For the brave among you, here is a log of all my pitfalls. I have tried to follow the instructions in the article Upgrade salt-master and minions on Ubuntu servers:

Step 1 – Update your apt repositories

First you need to make sure your apt repositories are up to date, so you get the latest stable versions. Easiest way to do this is via salt itself:

sudo salt '*' cmd.run 'apt-get update'

or in case you are behind a proxy http://proxy.company.com:8080:

sudo salt '*' cmd.run 'export http_proxy=http://proxy.company.com:8080; apt-get update'

A problem I see with this command is that the command might take quite long without any feedback for a long time. Instead of starting a new, separate SSH session to the master, we also can perform the salt command in the background with

sudo salt '*' cmd.run --async 'export http_proxy=http://proxy.company.com:8080; apt-get update'

In this case, we will get a feedback like

root@master:/# sudo salt 'minion1' cmd.run --async 'export http_proxy=http://proxy.company.com:8080; apt-get update'
 Executed command with job ID: 20151127133545387540

Find the status of the job by typing something like:

salt-run jobs.list_job 20151127133545387540

If the command is not yet finished, we will get a Result: “———-“. If it has finished, and was successful, we will get something like:

Arguments:
    - export http_proxy=http://172.28.12.5:8080; apt-get update
Function:
    cmd.run
Minions:
    - minion1
Result:
    ----------
    minion1:
        ----------
        return:
            Ign http://ppa.launchpad.net trusty InRelease
            Get:1 http://ppa.launchpad.net trusty Release.gpg [316 B]
            Get:2 http://ppa.launchpad.net trusty Release [15.1 kB]
            Ign http://archive.ubuntu.com trusty InRelease
            Get:3 http://archive.ubuntu.com trusty-updates InRelease [64.4 kB]
            Get:4 http://ppa.launchpad.net trusty/main amd64 Packages [2138 B]
            Get:5 http://archive.ubuntu.com trusty-security InRelease [64.4 kB]
            Get:6 http://archive.ubuntu.com trusty Release.gpg [933 B]
            Get:7 http://archive.ubuntu.com trusty-updates/main Sources [309 kB]
            Get:8 http://archive.ubuntu.com trusty-updates/restricted Sources [5219 B]
            Get:9 http://archive.ubuntu.com trusty-updates/universe Sources [181 kB]
            Get:10 http://archive.ubuntu.com trusty-updates/main amd64 Packages [824 kB]
            Get:11 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [23.4 kB]
            Get:12 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [426 kB]
            Get:13 http://archive.ubuntu.com trusty Release [58.5 kB]
            Get:14 http://archive.ubuntu.com trusty-security/main Sources [126 kB]
            Get:15 http://archive.ubuntu.com trusty-security/restricted Sources [3920 B]
            Get:16 http://archive.ubuntu.com trusty-security/universe Sources [36.0 kB]
            Get:17 http://archive.ubuntu.com trusty-security/main amd64 Packages [465 kB]
            Get:18 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [20.2 kB]
            Get:19 http://archive.ubuntu.com trusty-security/universe amd64 Packages [156 kB]
            Get:20 http://archive.ubuntu.com trusty/main Sources [1335 kB]
            Get:21 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
            Get:22 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
            Get:23 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
            Get:24 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
            Get:25 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
            Fetched 21.4 MB in 32s (657 kB/s)
            Reading package lists...
StartTime:
    2015, Nov 27 13:35:45.387540
Target:
    minion1
Target-type:
    glob
User:
    sudo_root
jid:
    20151127133545387540

Step 2 – Upgrade your master

Upgrading the master first ensures you don’t run into any version compatibility issues between your master and minions. So ssh into your master and run:

Better create a backup of /etc/salt/master and /etc/salt/minion, since the files will be overwritten during the upgrade:

sudo cp -p /etc/salt/master ~/master.bak
sudo cp -p /etc/salt/minion ~/minion.bak
sudo apt-get -y upgrade salt-master

or in case you are behind a proxy http://proxy.company.com:8080:

sudo http_proxy=http://proxy.company.com:8080 apt-get -y upgrade salt-master

Note that the process will ask you, whether /etc/salt/master and /etc/salt/minion should be overwritten. Since I have believed that I have not changed those configurations, I have answered with “Y” for both, but we can review the changes, since we have created backups above. However, I did not know that the docker start.sh command had performed changes in those files, so I better should have chosen “N”.

Since I have chosen “Y”, I had got:

root@master:/# salt '*' test.version
[CRITICAL] Could not deserialize msgpack message: This often happens when trying to read a file not in binary mode.Please open an issue and include the following error:
...(traceback information)...

Not good. Try again:

root@master:/# sudo http_proxy=http://172.28.12.5:8080 apt-get -y upgrade salt-master
 Reading package lists... Done
 Building dependency tree
 Reading state information... Done
 Calculating upgrade... Done
 salt-master is already the newest version.
 The following packages have been kept back:
 python-pip
 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Okay, is already upgraded, even though the upgrade process was stuck. Then, I have tried to upgrade python-pip:

root@master:/# sudo http_proxy=http://172.28.12.5:8080 apt-get -y upgrade python-pip

But I was still getting the critical error. After stopping and restarting the docker master container, I get:

root@master:/# salt '*' test.version
minion1:
    Minion did not return. [No response]
master:
    Minion did not return. [No response]

Might this might be, because the salt-minion version is not compatible with the salt-master version? No. I have restarted the minion service by issuing

service salt-minion restart

on minion1 (which will stop the docker container; so you need to perform docker start minion1 again). After that, we get:

root@master:/# salt '*' test.version
minion1:
    2014.7.2
master:
    Minion did not return. [No response]

So, at least the old minion is working again. However, doing the same on the master does not help: it does not even try to authenticate with the salt master (on localhost). Maybe, since we have changed /etc/salt/minion?

Yes, that was it. The default master name seems to be “salt” instead of “master” and I had to add the line

master: master

to /etc/salt/minion and perform a ‘service salt-minion restart’, then the local minion is finding its master again.

root@master:/etc/salt# salt '*' test.version
master:
    2015.5.3
minion1:
    2014.7.2

Now let us upgrade the minion as well.

Step 3 – Upgrade your minions

Before we attempt to upgrade, let’s take a quick look at the existing versions we have running. This might surprise you, I definitely found a couple of cloud instances that were running older version of salt-minion that I somehow had not upgraded in the past. So get a list of what version of salt your minions are running issue this salt command:

sudo salt '*' test.version

And you’ll get a nice display of every version currently in use. Another useful option here is the command ‘manage.versions’ which shows you a list of up to date minions vs those that need updating. Here is how you run it:

salt-run manage.versions

As I see it today, it is not possible to upgrade salt minion versions using apt-get install via salt. The reasons are:

  1. Upgrade of salt-minion requires a reboot of salt-minion. If this is done via salt, the process is stuck. However, there is a workaround using the “at” module described in this salt FAQ.
  2. Even if the -y switch is given to apt-get install salt-minion, there are two interactive steps during the installation process: you are asked, whether you want to rewrite the files /etc/salt/minion and /etc/salt/master. Possibly this is fixed by using the pkg.install module of salt, but pkg.install did not work in my case.

Now I tried to follow the instructions like:

sudo salt '*' pkg.install salt-minion refresh=True

This is the correct way to do it with Salt, but it was stuck in my case. Therefore I have tried:

sudo salt '*' cmd.run 'apt-get -y install salt-minion'

or in case you are behind a proxy http://172.28.12.5:8080:

sudo salt '*' cmd.run 'http_proxy=http://172.28.12.5:8080 apt-get -y install salt-minion'

This is not accepted by cmd.run (although the command works locally on the minion. Is this a salt bug?). Instead we get:

master:
    TypeError encountered executing cmd.run: run() takes at least 1 argument (0 given). See debug log for more info.

Therefore, I have exchanged this by:

sudo salt '*' cmd.run 'echo http_proxy=http://172.28.12.5:8080 apt-get -y install salt-minion > salt-minion_install.sh; sh salt-minion_install.sh; rm salt-minion_install.sh'

However, this was stuck infinitely and I had to kill apt-get locally on the minion.

At the end, I have restarted the minion, and performed the commands

http_proxy=http://172.28.12.5:8080 apt-get -y install salt-minion
service salt-minion restart

locally on the minion. This has worked.

Step 4 – Verify everything worked

Everything should be upgraded now and running the latest version of salt-minion. You can verify this by running the test.version command again:

sudo salt '*' test.version

If you see some minions aren’t using the latest version you may need to manually intervene to see what is stopping apt from upgrading things for you.

Appendix D: Details on Targets, Groups and Variables for Ansible and Salt

In case of Ansible, targets and target groups are defined in so-called inventory files.

  • Inventory files are kept separate from other Ansible configuration files. Inventory-files can be selected on the command-line (default: /etc/ansible/hosts).
  • Groups can be defined as a
    • List of targets (IP addresses or FQDNs) with the possibility to define ranges like www[01:50].example.com or db-[a:f].example.com
  • Host variables and group variables can be defined in the inventory files (required: version >2.0) or in separate files in host_vars or group_vars folders that have the same as the host or the group.
  • Other variables can be assigned on the fly on command line using the -e switch.

In case of Salt, targets and target groups are defined as Node Groups in the /etc/salt/master configuration file.

  • Node Group definitions are not kept separate from the Salt configuration file. The configuration file can be selected from command line (default /etc/salt/master),
  • Node Groups can be defined as
    • Individual Salt Names with globbing supported (e.g. ‘minion*’)
    • List of targets (IP addresses or Salt Names)
    • IP addresses or Subnets
    • List of other groups (version > 2015.8.0)
    • Regular Expressions
    • Grains (e.g. matching against operating system types, roles, user-defined grains, etc.)
      • user-defined grains can defined centrally or on the target machine
    • combination of all above as compound search allowing and and or operators.
  • Host variables can be defined as grain withing the master and/or minion configuration files. Those can be used in state files using Jinja2 semantics, e.g. {% if grains['os'] == 'RedHat' %}
  • Other variables can be defined using jinja symantics in state files and so-called pillar sls files (best practice). See here for some examples. We will need this in part IV, where we will work with Jinja2 templates, similar to what we have done in part II of the series.

All in all, the target&group definition and search capabilities are much more elaborated in case of Salt than they are in case of Ansible.

Goto summary section


3 thoughts on “IT Automation Part III: Saltstack “Hello World” Example

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s