vRA SaltStack Config – Event-Driven Automation

In the previous post we introduced State Files and explored a bit how they allow a declarative approach to configuration management. We also covered how States Files can be applied to Targets through Commands and Jobs and we will explore in the future the concept of Highstate. All of this is human initiated: an operator user choose to apply a State File to a Target. However, Salt also supports an event-driven approach: Salt executes automation tasks (not necessarily applying State Files) as a reaction to specific event(s).

Event-driven is a distinctive capability of Salt and it is one of its foundation elements. Indeed, Salt is built around an event infrastructure that can be leveraged and extended to drive reactive provisioning, configuration, and management across all devices in your infrastructure. Hereafter there are some examples of things that people do with Salt’s event driven automation:

• Instantly restore a configuration file when an unauthorized change is made;
• Notify a Slack channel when a build fails;
• Send an email when a user logs on to a production server;
• Automatically scale compute resources;

The schema below shows the main components of the Salt’s event driven systems. Wow, honestly it looks more complex than actually it is! I made it a bit more complex than actually needed because I wanted to stress the point that there are different type of events and commands (actions). Let’s start with events (red arrows), please note the following is not orthodox (aka Salt fully compliant), but it helped me to understand it and I hope this works for you as well: 1) internal events in my schema are those generated by the Master, 2) of course events are generated from Minions and 3) Beacon events are fired from a Beacon configurations (still generated from Minions) and are the output of a monitoring activity. On the other side, commands (in green): 1) commands executed on the Master(s) such as Salt Runners (i.e. tasks you would start using salt-run) and Wheels (commands to manage your Salt environment such as tasks you would start using salt-key utility); 2) remote execution, these run an execution module on the targeted minions (tasks you would start using salt).

Events

Salt’s internal components communicate with each other by sending and listening to events. In Salt, events are sent for about everything you could imagine: Salt Minion connects, Key is accepted or rejected, Job is sent, Job results are returned from a Minion, etc. Events are made up of two main parts:

• Tag: which identifies the event that was fired. All Salt events are prefixed with salt/, with additional levels based on the type of event. For example, Job events are prefixed with salt/job/. Each event tag part is separated using / to provide simple namespacing and can include specific details such as a Job ID or Minion ID;
• Data: that contains details about the event. Each event contains a timestamp and additional keys and values that are unique to that specific event.

Salt comes with tons of built-in events, in addition to that you can enable some additional events in the Salt system, as well as generate your own custom events (that’s super easy!). In addition to these, you can also use Beacons to monitor and raise events for things that are not Salt-related (e.g. systems logins, disk usage, service status, etc.)

Beacons

The Beacon system allows the minion to hook into a variety of system processes and continually monitor these processes. When monitored activity occurs in a system process, an event is sent on the Salt event bus. Salt Beacons can currently monitor and send Salt events for many system activities, including:

• file system changes;
• system load;
• service status;
• shell activity, such as user login;
• network and disk usage;

Reactors

Salt’s Reactor mechanism gives you the ability to trigger actions in response to any event. Not only can you trigger actions when Jobs and tasks complete, you can trigger actions when services go down, users log in, files are changed, and from anywhere that you can send a custom event (end of a build script, cron job, etc.). A Salt Reactor is a configuration that defines how to react (what action execute) to specific events that are fired within the event system.

Reactors States are State Files that are executed to react against specific events. There are three types of reactions:

• Remote execution: run an execution module on the targeted Minions. This is anything you would do by calling the salt command including applying a State or Highstate;
• Salt Runners: these are tasks you would start using salt-run. For example, the HTTP runner can trigger a webhook;
• Wheel: these commands manage your Salt environment, performing tasks such as accepting keys and updating configuration settings. Some of these tasks are exposed by the salt-key utility.

Lab Time!

I am going to close this post with a super basic example of event-driven automation. We will use Salt to ensure a specific web page is served from a web server :

• we will use a Salt Beacon to monitor the status of a file (our web page) and this will notify Salt Master if the file is changed
• we will configure a Salt Reactor to react upon Beacon event in order to restore the original file

As a prerequisite we just need a CentOS machine with Salt Minion installed (see this post for instructions). Here after there is the list of macro steps:

1. Apache Web Server installed and serving the following static web page /var/www/html/index.php
2. Python package named pyinotify installed
3. A Beacons on the Minion that notify whenever a change is applied to the file /var/www/html/index.php
4. A Reactor that triggers a State File designed to restore the original file when a change is notified from the Beacon

We will implement steps from 1 to 3 with Salt (of course) using the following single State File.

# Install and configure Apache HTTPD on CentOS/RHEL
# Deploy Beacon with prerequisites to monitor file /var/www/html/index.php

# Prerequisites for Beacons

install_epel_pip:
  pkg.installed:
    - pkgs: 
      - epel-release
      - python2-pip
    - reload_modules: True

install_inotify:
  pip.installed:
    - name: pyinotify
    - bin_env: /usr/bin/python
    - require:
      - pkg: install_epel_pip

# Install and configure Apache HTTPD

install_apache:
  pkg.installed:
    - pkgs:
      - httpd
      - php

configure_apache:
  file.managed:
    - name: /etc/httpd/conf/httpd.conf
    - source: salt://applications/apache/files/httpd.conf
    - template: jinja
    - require:
      - pkg: install_apache
    - defaults:
        admin_email: user@host.org
        group: apache
        user: apache
        http_port: 80

# Deploy sample web page

configure_web_page:
  file.managed:
    - name: /var/www/html/index.php
    - source: salt://applications/apache/files/index.php
    - require:
      - pkg: install_apache

# Deploy Beacons

deploy_beacons_file:
  file.managed:
    - name: /etc/salt/minion.d/beacons.conf
    - source: salt://applications/apache/files/beacons.conf
    - makedirs: True

# Restart Apache HTTPD and Salt Minion processes upon respective config files are updated

restart_salt-minion:
  service.running:
    - name: salt-minion
    - enable: True
    - watch:
      - deploy_beacons_file

start_apache:
  service.running:
    - name: httpd
    - enable: True
    - watch:
      - file: configure_apache

The State File above is actually the same used in my previous post with the additions of index.php and beacons.conf deployments, install of Beacon prerequisites and php. You can proceed it by updating it. Here is detail about changes:

# Prerequisites for Beacons

install_epel_pip:
  pkg.installed:
    - pkgs: 
      - epel-release
      - python2-pip
    - reload_modules: True

install_inotify:
  pip.installed:
    - name: pyinotify
    - bin_env: /usr/bin/python
    - require:
      - pkg: install_epel_pip

Prerequisite for Beacons is the Python package named pyinotify, in order to install it we need to have pip installed from epel-release repo. Please, note the followings in the snippet above:

• option reload_module is set to True. This allows you to force Salt to reload all modules. In many cases Salt is clever enough to transparently reload the modules, but here we need it to make sure we can actually use the pip state module in the install_inotify section of the state file
• in the install_inotify section we use the pip state module to install the pyinotify Python package

# Install and configure Apache HTTPD

install_apache:
  pkg.installed:
    - pkgs:
      - httpd
      - php

In the install_apache section we also install php that is needed to run the sample web page.

# Deploy sample web page

configure_web_page:
  file.managed:
    - name: /var/www/html/index.php
    - source: salt://applications/apache/files/index.php
    - require:
      - pkg: install_apache

# Deploy Beacons

deploy_beacons_file:
  file.managed:
    - name: /etc/salt/minion.d/beacons.conf
    - source: salt://applications/apache/files/beacons.conf
    - makedirs: True

With the snippet above we deploy on the targeted machine the index.php and the beacon.conf files, later in this post we will make sure these files are available in the Salt Master File Server. The only thing worth to mention is that in the deploy_bacons_file section we use the option makedirs set to True which functionality is pretty obvious.

# Restart Apache HTTPD and Salt Minion processes upon respective config files are updated

restart_salt-minion:
  service.running:
    - name: salt-minion
    - enable: True
    - watch:
      - deploy_beacons_file

The new Beacon configuration require a Salt Minion restart.

As mentioned above we need to make sure both index.php and the beacon.conf files are available in the Salt Master File Server. Through the vRA SaltStack Config user interface add the following sample web page file under /applications/apache/files/index.php

<html>
<head>
    <title>Event Driven Automation Test Page</title>
</head>
<body>
    <h1>Web Server Test Page</h1>
    <?php
    echo 'Serving this page from Host IP Address : '. $_SERVER['HTTP_HOST'];
    ?> 
</body>
</html>

Than add also the following beacons.conf under the same path. As you can see the Beacon configuration is pretty simple it just states the absolute path of the file to be monitored and the mask option specify what action on the file should trigger the Beacon Event. It should be wise to add t the end of the beacons.conf file the disable_during_state_run: True parameter prevents the inotify beacon from generating Reactor events due to Salt itself modifying the file. In this case it is not strictly required as we opted for the close_write action to fire the Beacon Event.

# Beacons file that watched /var/www/html/index.php

beacons:
  inotify:
    - files:
      /var/www/html/index.php
        mask:
          - close_write

Ok now it is time to configure the Reactor mechanism, in our case this is comprised by 3 elements:

1. A reacotr.conf file on the Master: this file specify what events we want Salt to react and what action(s) Salt need to execute. As we said earlier in this post Salt actions within a Reactor can be: Remote executions, Salt Runners and Wheels
2. An orchestration state file: this allows to the Reactor to apply a specific state file as a reaction to a specific event. Orchestration is a broad concept that we are not touching here
3. State File: this is the declarative automation that make sure the required index.php is in place in the right path

Let’s start from the reactor.conf configuration file. Make sure the reactor.conf below is placed on the Salt Master in the following path /etc/salt/master.d or to append the two final lines to the existing reactor.conf if you have one in place already. You can do this manually or through Salt (at this point you should know how to do that), just make sure to restart the Salt Master process after updating the Reactor file. This file is simply instructing Salt Master to execute the fix-index-file.orch state file when it receives an event with the following tag salt/beacon/*/inotify//var/www/html/index.php

# Reactor configuration to be located /etc/salt/master.d/reactor.conf on the Salt Master

reactor:
  - 'salt/beacon/*/inotify//var/www/html/index.php':
    - 'salt://reactor/orchestrate/fix-index-file.orch'

In the Reactor file we are referring to a state file named fix-index-file.orch, we need to make sure it exist in the File Server. Through the vRA SaltStack Config user interface add the State file in the snippet below under /reactor/orchestrate/fix-index-file.orch. This State file is an Orchestration that applies a State file located in /applications/apache/index-file.sls (arg parameter) to the target specified in the tgt parameters. You surely noted that the tgt has a parametric value, the {{ data['id'] }} syntax is instructing the state file to use as Target the event payload value identified by the key 'id'. In our case this is the Minion ID that fired the Beacon Event.

# Orchestrator to react on index.php file changes

fix_index_file:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg: 
      - applications.apache.index-file

In the above Orchestration we refer to another State file, we need to make sure this exists in our File Server. Through the vRA SaltStack Config user interface add the State file in the snippet below under this path /applications/apache/index-file.sls. This State file simply make sure the original index.php is present in the right location on the targeted machine. Nothing special here, we just need to note that I had to add a 20 second delay in order to allow me to test/demonstrate this use case because Salt is so fast that fix the issue in a couple of seconds.

# State file used to restore the original index.php file

# Need to sleep 20 sec because salt is too fast!!!
sleep_20:
  cmd.run:
    - name: sleep 20

deploy_index_file:
  file.managed:
    - name: /var/www/html/index.php
    - source: salt://applications/apache/files/index.php
    - require:
      - sleep 20

restart_apache:
  service.running:
    - name: httpd
    - enable: True
    - watch:
      - deploy_index_file

Test

As first thing we need to apply the state files that install Apache Web Server along with our sample web page, Beacon and its prerequisites. Select your Minion, click on run Command button, fill the form with state.apply function and the path to our apache State file.

Once the Job is successfully completed access the sample web page through http://<machine-ip-or-fqdn> in your browser.

Now ssh to your machine and edit /var/www/html/index.php with your favourite editor, apply some changes than save it. Here the 20 seconds delay in our Reactor mechanism allows us to refresh the web page to actually see the change. Without this delay it would be almost impossible for us to refresh the page, of course the delay is just for test/demonstration purposes.

The Reactor triggers the State file to be applied against our Minion and once it is completed the original web page is restored.

This is just a trivial example to get you introduced to the powerful and funny world of Event Driven Automation. Now it is your time to experiment!