The coolest features in Salt 2019.2 Fluorine

26 minute read Updated:

Salt Fluorine

It started as a series of tweets about a few exciting features I’ve found in Salt Fluorine git branch. Then the release was postponed from 2018.11 to 2019.2, and that gave me enough time to consolidate these tweets (and lots of other features) into a full-blown blog post. It is not a replacement for the official release notes, but I hope you will enjoy it!

UPDATE (2020.05.20): the 2019.2.5 bugfix release is available and is strongly recommended.

A list of the most exciting features in Salt 2019.2 Fluorine

Network automation

My network management skills are rusty, but I used to work in this field around 2004-2009. I had to deal with Cisco and Huawei gear, and the CLI experience was awful. However, I still remember my excitement when I got the chance to work with Juniper hardware and Junos in particular. It felt like proper Unix, had version control (with diffs!), atomic commit/rollback support and tons of other well-thought-out features.

Many configuration management tools are evolving in this direction, unifying change management processes across different networking equipment from different vendors. With Salt, it is possible to have version control, code reviews, automatic rollouts, and other nice features, even for network devices that do not support this out of the box. Things have improved a lot, so I ALMOST want to work in this field again 😀

There are lots of network-related features in Salt Fluorine. Below is a brief list, please refer to the docs for a complete description.

Details

Multiple SaltSSH versions

This is a clever hack to allow SaltSSH work across different major Python versions and use multiple versions of Salt. The only necessary condition is that your states should be compatible with all the versions you want to use.

To use SaltSSH across Python 2.7 - Python 3.x, you need to install Salt packages for both versions of the interpreter on a master machine. Using multiple Salt versions is trickier - to do that you need to install them into isolated folders, including all necessary dependencies:

ssh_ext_alternatives:
  2016.3:                 # Namespace, can be actually anything.
  py-version: [2, 6]      # Constraint to specific interpreter version
  path: /opt/2016.3/salt  # Main Salt installation
  dependencies:           # List of dependencies and their installation paths
    jinja2: /opt/jinja2
    yaml: /opt/yaml
    tornado: /opt/tornado
    msgpack: /opt/msgpack
    certifi: /opt/certifi
    singledispatch: /opt/singledispatch.py
    singledispatch_helpers: /opt/singledispatch_helpers.py
    markupsafe: /opt/markupsafe
    backports_abc: /opt/backports_abc.py

Details

Terraform roster for SaltSSH

It looks like Salt has lost the battle for multi-cloud infrastructure provisioning (especially for non-VM resource types). It is hard to beat the vast amount of different providers in Terraform, its speed, excellent vendor support program and even things like fast GCP feature development using Magic Modules. So, the only sensible strategy left is to provide good interoperability with Terraform, in order to use both tools in tandem.

With the terraform-provider-salt you can keep your infrastructure-as-code self-contained in a single directory and use Terraform + salt-ssh combo to provision and configure your nodes:

$ terraform apply
$ salt-ssh '*' state.highstate

This is done by creating a mapping between the Terraform resources and salt-ssh roster entries:

resource "libvirt_domain" "domain" {
  name = "domain-${count.index}"
  memory = 1024
  disk {
       volume_id = "${element(libvirt_volume.volume.*.id, count.index)}"
  }

  network_interface {
    network_name = "default"
    hostname = "minion${count.index}"
    wait_for_lease = 1
  }
  cloudinit = "${libvirt_cloudinit.init.id}"
  count = 2
}

resource "salt_host" "example" {
    host = "${libvirt_domain.domain.network_interface.0.addresses.0}
}

Also, you can use local_file Terraform resource to render simple pillar files:

resource "local_file" "pillar_database_cluster" {
  filename = "${path.module}/srv/pillar/terraform_database_cluster.sls"
  content = <<EOF
terraform:
  database_master_ip: ${salt_host.master.host}
EOF
}

NOTE: please do not confuse the terraform-provider-salt with the built-in salt-masterless Terraform provisioner. The latter one connects to a node via ssh, syncs pillar/state trees, runs the salt-bootstrap.sh script and then uses the salt-call --local command to apply a state.

Details

Ansible playbooks and inventory

Ansible vs. Salt

Ansible is definitely more popular, but people still switch to Salt. The reasons can be very different, such as infrastructure growth, unacceptable playbook running time, a need to react to infrastructure events automatically, and even the existential fear of IBM. As with Terraform, the right strategy is to simplify interoperability and increase the ease of switching.

Salt.modules.ansiblegate was first introduced in Salt 2018.3. In Salt Fluorine it gained the ability to run Ansible playbooks (both from states and orchestration jobs).

Also from now on Salt uses ansible-inventory command internally to retrieve a roster.

Details

CLI exit codes

This feature is not very visible from a user perspective but is quite essential. It streamlines many edge cases in CLI exit codes and can simplify your scripts or CI workflows.

Salt CLI exit codes

In addition to the table above:

% salt-call --local cmd.run cmd='return 42' ; echo $?
[ERROR   ] Command 'return 42' failed with return code: 42
[ERROR   ] retcode: 42
[ERROR   ] Command 'return 42' failed with return code: 42
[ERROR   ] output:
local:
1


% salt-call --local cmd.run cmd='return 42' success_retcodes='[42]'; echo $?
local:
0

Details

Loadable matchers

When you read the Salt targeting documentation, it is not immediately obvious that minions are responsible for matching themselves against a target specification. Every minion looks at the targeting data broadcasted by the salt-master via ZeroMQ pub/sub messaging channel and decides if they match it (NOTE: this process is more nuanced if you have zmq_filtering enabled).

In Salt Fluorine all matchers were moved to the separate module namespace and can be dynamically loaded. It is not possible to create a new matcher, because the required CLI plumbing does not exist yet. For now, you can only override any existing matcher by placing a file into the _matchers directory in your file_roots and propagating it with saltutil.sync_matchers:

# list_matcher.py
from __future__ import absolute_import, print_function, unicode_literals
from salt.ext import six

def match(self, tgt):
    '''
    Determines if this host is on the list
    '''
    if isinstance(tgt, six.string_types):
        # The stock matcher splits on `,`.  Change to `/` below.
        tgt = tgt.split('/')
    return bool(self.opts['id'] in tgt)

Details

Loadable serializers

Add your own custom serializers (XML, anyone?) via Salt dynamic module distribution mechanism. You can use them to manage domain-specific file formats, such as configuration files.

Example serializer (put this into salt://_serializers/hcl.py):

"""HCL deserializer."""

try:
    import hcl  # https://github.com/virtuald/pyhcl
    available = True
except ImportError:
    available = False

from salt.serializers import DeserializationError
from salt.ext import six

__all__ = ['deserialize', 'available']


def deserialize(stream_or_string, **options):
    try:
        if not isinstance(stream_or_string, (bytes, six.string_types)):
            return hcl.load(stream_or_string)
        if isinstance(stream_or_string, bytes):
            stream_or_string = stream_or_string.decode('utf-8')
        return hcl.loads(stream_or_string)
    except Exception as e:
        raise DeserializationError(e)


def serialize(obj, **options):
    raise NotImplementedError("pyhcl can only serialize to json")
# salt/hcl.sls

{% load_text as hcl_example %}
variable "ami" {
  description = "the AMI to use"
}

resource "aws_instance" "web" {
  ami               = "${var.ami}"
  count             = 2
  source_dest_check = false

  connection {
    user = "root"
  }
}
{% endload %}

{% do salt.log.warning(salt['slsutil.deserialize']('hcl', hcl_example)) %}

To sync serializers from salt://_serializers run the salt '*' saltutil.sync_serializers command:

% salt-call --local saltutil.sync_serializers
local:
    - serializers.hcl

% salt-call --local state.apply hcl
[WARNING ] {"resource": {"aws_instance": {"web": {"ami": "${var.ami}", "connection": {"user": "root"},
"count": 2, "source_dest_check": false}}}, "variable": {"ami": {"description": "the AMI to use"}}}

Details

Docker proxy minion

This adds the new minion type called docker, which uses docker.call to run Salt modules in Docker containers without installing Salt in the container. Also, it can run state.sls, state.apply and state.highstate in the Docker container.

Details

Other Docker improvements

Using proxy minion from the CLI

This is a huge time saver for the real men who write their own device drivers people who write and troubleshoot their own proxy minions. It allows you to use proxy minions from the minion box without a master:

% salt-call --local --proxyid PROXY_ID test.ping

This salt-call console command does basically the same thing as the separate salt-proxy --proxyid PROXY_ID process, but just for a single call. It also nicely complements the above-mentioned Docker proxy minion feature.

Details

Grains in grains

When writing custom grains, sometimes it is useful to reuse the data from already existing grain. It is hard to do because the grain evaluation order is nondeterministic, but this restriction has been partially lifted. For custom grains, if the function accepts an argument named grains, then the previously rendered core grains will be passed in as a dictionary.

import re

def if_count(grains):
    """Return number of interfaces excluding the loopback."""
    return {
        'if_count': len([i for i in grains['ip_interfaces'] if i != 'lo'])
    }

def role(grains):
    """Extract a server role from FQDN like env-role-NN.domain.tld."""
    match = re.match(
        '^[a-z0-9]+-([a-z0-9]+)-[0-9]+\.[a-z0-9-]+\.[a-z]+$',
        grains['fqdn']
    )
    return {'role': match.group(1)} if match else {}

Details

SaltSSH key password

Previously, using ssh-agent was the only way to use password-protected SSH keys with salt-ssh. Now, the private key passphrase can be set in 3 different places:

Details

Configurable module environment

On the surface, this feature allows you to specify arbitrary environment variables for aptpkg, yumpkg and zypper execution modules. Under the hood, it is much more interesting, because it allows you to specify custom environment variables for shell commands executed by Salt modules (states, execution modules, returners, etc.). To do that, you need to add something like this to either grains or pillars:

system-environment:
  <type>:               # Type of the module (states, modules, etc.).
    <module>:           # Module name
      _:                # Namespace for all functions in the module
        <VAR>: "value"
      <function>:       # Namespace only for particular function in the module
        <VAR>: "value"

The configuration is quite granular: it allows you to define variables for any module type, module name and even for a specific function. The only caveat is that the environment won’t be applied to all existing modules automatically. It requires explicit module-level support. To do that, you need to call salt.utils.environment.get_module_environment(globals(), function=None) in your module and pass the resulting environment to __salt__['cmd.run'] or __salt__['cmd.run_all'].

Details

Wtmp/btmp beacon improvements

Details

Cross platform file monitoring

It is similar to the inotify beacon, with the main difference that it uses the cross-platform watchdog module and works on Linux 2.6+, Mac OS X, BSD Unix variants, Windows Vista and later and also has an OS-independent polling mechanism.

beacons:
  watchdog:
    - directories:
        /path/to/dir:
          mask:
            - create
            - modify
            - delete
            - move

Details

Redis SDB module

Initially, SDB modules in Salt were designed to store secrets, as an alternative to pillar files. Redis does not look like a good option to store sensitive data (as compared to Vault and some of the other SDB modules). However, if you want to store more dynamic data across different masters and minions and then use it in your states, then Redis looks like a good solution. Also, there is the Redis cache backend and also Redis returner, which you can query using the SDB module.

Details

Google Chat module

Salt @sadserver bot for Google Chat 
{% if pillar.get('sadserver_scheduled_run', False) %}

sadserver_quote:
  # NOTE: Put use_superseded: ['module.run'] into the minion config
  module.run:
    - google_chat.send_message:
      # https://developers.google.com/hangouts/chat/how-tos/webhooks
      - url: "https://chat.googleapis.com/v1/spaces/XXX/messages?key=YYY&token=ZZZ"
      - message: {{ salt['cmd.run']('python3 -c \'import json, random, sys, urllib.request; sys.stdout.buffer.write(("```{}```".format(random.choice(json.loads(urllib.request.urlopen("https://brokenco.de/files/sadserver.json").read().decode("utf-8")))["full_text"].strip())).encode("utf-8"))\'') | yaml_encode }}
      # Also try https://brokenco.de/files/sadoperator.json

{% else %}

sadserver_morning_schedule:
  schedule.present:
    - function: state.sls_id
    - job_args:
        - sadserver_quote
        - {{ sls }}
    - job_kwargs:
        pillar:
          sadserver_scheduled_run: True
    - when:
        - Monday 9:00am
        - Tuesday 9:00am
        - Wednesday 9:00am
        - Thursday 9:00am
        - Friday 9:00am

{% endif %}

Details

SMTP attachments

With the addition of the new parameter named attachments, you can send local files over email by using smtp state/module. One obvious example is sending out generated client certificates:

{% set email='user@example.com' %}
{% set filename = email.replace('@', '-').replace('.', '-') %}

email-openvpn-config-for-{{ filename }}:
  smtp.send_msg:
    - name: |
        Your personal OpenVPN configuration file is attached below:
    - subject: 'Your OpenVPN profile'
    - recipient: {{ email }}
    - sender: admin@example.com
    - profile: smtp-credentials
    - attachments:
        - /etc/pki/openvpn/{{ filename }}.ovpn
    - onchanges:
        - file: /etc/pki/openvpn/{{ filename }}.ovpn

/etc/pki/openvpn/{{ filename }}.ovpn:
  file.managed:
    - mode: 0600
    - template: jinja
    - defaults:
      ca: __slot__:salt:file.read(/etc/pki/ca.crt)
      tls_crypt: __slot__:salt:file.read(/etc/pki/static.key)
      cert: __slot__:salt:file.read(/etc/pki/issued/{{ filename }}.crt)
      key: __slot__:salt:file.read(/etc/pki/private/{{ filename }}.key)
    - source: salt://openvpn/files/config.ovpn.j2
# salt/openvpn/files/config.ovpn.j2

client
dev tun
remote openvpn.example.com 1194 udp
connect-retry-max 3
resolv-retry 5
tls-timeout 10
nobind
persist-key
persist-tun
explicit-exit-notify
verb 3
remote-cert-tls server

<ca>
{{ ca }}
</ca>

<cert>
{{ cert }}
</cert>

<key>
{{ key }}
</key>

<tls-crypt>
{{ tls_crypt }}
</tls-crypt>

Details

Jira integration

This is an execution module to manipulate Jira tickets using the Jira python client library. It could be used to automatically raise Jira tickets when something happens and supports the following operations:

There is no corresponding state module, so if you want to use it in a state, you need to use the module.run function.

Details

KMS decryption renderer

The new renderer that utilizes AWS Key Management Service to decrypt pillar values using Fernet symmetric encryption. To use it you need to specify KMS credentials and then include your ciphertexts in a pillar file:

#!yaml|aws_kms

a-secret: gAAAAABaj5uzShPI3PEz6nL5Vhk2eEHxGXSZj8g71B84CZsVjAAt

Please note, that the aws_kms renderer will attempt to decrypt each pillar value recursively (only string types). You probably want to use it sparingly and for small pillar files.

Details

Enhancements in package modules

Various enhancements for the FreeBSD pkgng execution module

Details

Other enhancements

New Azure ARM modules

Huge improvements in Azure ARM execution and state modules:

Azure GovCloud support

General improvements

Compute module/state

Network module/state

Resource module/state

Details

Cloud map improvements

The first feature allows you to override providers in map nodes (previously it was only possible to specify a provider in the cloud profile):

webapp:
  - node1:
    provider: ny:openstack
  - node2:
    provider: nj:openstack
  - node3:
    provider: toronto:openstack

The second feature enables use of cloud map data sourced from the pillar (salt-call cloud.map_run map_pillar=my-prd-map):

cloud:
  maps:
    my-prd-map:
      named-profile-01:
        - named-instance-01
        - named-instance-02

Details

Other cloud improvements

Windows runas improvements

Details

Windows Advanced Audit policies

This module allows you to view and modify the audit settings as they are applied on the machine (either set directly or via local or domain group policy). The audit settings are broken down into nine categories (and the default one named All):

You can use the auditpol.get_settings, auditpol.get_setting and auditpol.set_setting functions in the following way:

# Get current state of all audit settings
salt * auditpol.get_settings

# Get the current state of all audit settings in the "Account Logon"
# category

salt * auditpol.get_settings category="Account Logon"
# Get current state of the "Credential Validation" setting

salt * auditpol.get_setting name="Credential Validation"
# Set the state of the "Credential Validation" setting to Success and

# Failure
salt * auditpol.set_setting name="Credential Validation" value="Success and Failure"

# Set the state of the "Credential Validation" setting to No Auditing
salt * auditpol.set_setting name="Credential Validation" value="No Auditing"

Details

LXD module/state

This feature was extracted from the lxd-formula and requires the pylxd module to work.

Details

Libvirt improvements

NI Linux RT improvements

SmartOS improvements

Details

GlusterFS improvements

Vault improvements

Runner function to unseal Vault server

salt-run vault.unseal uses the keys from the fault configuration to unseal Vault server.

PR #46996 by Daniel Wallace

wrapped_token authentication method

A new wrapped_token authentication method that allows automatic unwrapping of vault wrapped tokens in the local vault configuration.

PR #47072 by Beorn Facchini

vault.write_raw function

This function allows you to write raw data to a vault:

salt '*' vault.write_raw "secret/my/secret" '{"user":"foo","password": "bar"}'

PR #47712 by @slaws

Named roles

An option to define named role for creation of vault tokens. User can define specific token named role for minion created tokens and explicitly define its behavior and access policies. Example: https://www.nomadproject.io/docs/vault-integration/index.html#vault-token-role-configuration.

PR #48586 by @astorath

Zabbix improvements

Nifty tricks

JID in the logs

This feature is super useful for debugging purposes because it allows you to grep the logs for a specific job ID. It is off by default, but I think you should enable it right away! Add the following lines to your master/minion config file (the relevant part is %(jid)s):

log_fmt_logfile: '%(asctime)s,%(msecs)03d [%(name)-17s][%(levelname)-8s] %(jid)s %(message)s'
log_level: info

Then you can use the JID to filter the log:

% sudo salt --show-jid minion1 state.apply teststate

jid: 20181128221721109828
minion1:
----------
          ID: test_notification
    Function: test.show_notification
      Result: True
     Comment: Hello there!
     Started: 22:17:21.283097
    Duration: 2.02 ms
     Changes:

Summary for minion1
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   2.020 ms

% sudo salt minion1 cmd.run 'grep "\[JID: 20181128221721109828\]" /var/log/salt/minion'

minion1:
    2018-11-28 22:17:21,133 [salt.minion      ][INFO    ] [JID: 20181128221721109828] Starting a new job 20181128221721109828 with PID 1955
    2018-11-28 22:17:21,237 [salt.state       ][INFO    ] [JID: 20181128221721109828] Loading fresh modules for state activity
    2018-11-28 22:17:21,283 [salt.state       ][INFO    ] [JID: 20181128221721109828] Running state [test_notification] at time 22:17:21.283098
    2018-11-28 22:17:21,283 [salt.state       ][INFO    ] [JID: 20181128221721109828] Executing state test.show_notification for [test_notification]
    2018-11-28 22:17:21,284 [salt.state       ][INFO    ] [JID: 20181128221721109828] Hello there!
    2018-11-28 22:17:21,285 [salt.state       ][INFO    ] [JID: 20181128221721109828] Completed state [test_notification] at time 22:17:21.285117 (duration_in_ms=2.02)
    2018-11-28 22:17:21,287 [salt.minion      ][INFO    ] [JID: 20181128221721109828] Returning information for job: 20181128221721109828

Docs: Logging changes. PR #48660 by Gareth J. Greenaway

Per-state failhard

If you want to abort the state execution process when any single state fails, you can use global or per-state failhard option. Starting with Salt Fluorine, you can override the global failhard: True option by adding failhard: False to any state. This is also helpful if you want to use onfail, onfail_in or onfail_any state requisites because otherwise they are ignored.

PR #46448 by Herbert

Wildcard pillar includes

Previously it was only possible to use wildcards to include states, and now you can do the same for pillar (both in top.sls and individual pillar files via include):

base:
  '*':
    - users.*

PR #45269 by Richard W .

Nodegroup support for compound matcher

Use salt -C 'N@nodegroup' on the command line and compound nodegroup matchers in your top.sls (apparently, this requires putting nodegroups into both master and minion configuration files). PR #47421 by Matt Phillips

List the states that will be applied on highstate

% salt '*' state.show_states

minion1:
    - base
    - timezone
    - ntp

Unlike the state.show_top, this function also shows included states.

PR #44475 by Petr Michalec

slsutil.merge_all function

The same as slsutil.merge, but can merge more than two dictionaries at once:

# pillar file
{% set layer1 = {'a': 'a', 'b': 'b', 'c': 'c'} %}
{% set layer2 = {'a': 'z'} %}
{% set layer3 = {'c': 'x'} %}
value: {{ salt['slsutil.merge_all']([layer1, layer2, layer3]) }}
% salt-call --local pillar.get value
local:
    ----------
    a:
        z
    b:
        b
    c:
        x

PR #47679 by Tyler Couto

New functions in the defaults module

PRs #44850, #44851 and #45051 by Ahmed M. AbouZaid

state.sls_exists function

The main use-case for this function is to conditionally include a state only if the corresponding file exists (for example, to apply per-role or per-host states automatically). Unlike the file.exists, it should work with gitfs too.

base:
  '*':
    - base
    {% set minion_state = 'minion.' ~ (grains.id | replace('.', '_')) -%}
    {% if salt['state.sls_exists'](minion_state) -%}
    - {{ minion_state }}
    {%- endif %}

PR #45730 by Christian McHugh

file.tidied state

This is a port of Puppet tidy resource. It allows you to recursively remove unwanted files based on specific criteria. For example:

prepared-for-distribution:
  file.tidied:
    - name: /tmp/source
    - matches:
      - .*\.pyc
      - .*\.orig
      - .*\.rej
      - .DS_Store

PR #47718 by Dirk Heinrichs

Use unless and onlyif together

Allow the unless and onlyif requisites to both work when specified in a state file. Previously, if you tried to specify them simultaneously, onlyif would take precedence and the unless statement was ignored.

PR #45229 by Gareth J. Greenaway

Relative Jinja imports

Allow you to include Jinja templates using relative paths: {% from './foo' import bar %}. Quite useful for writing formulas or if you have a Jinja macro that is used across different states.

PR #47490 by @plastikos

config.items helper

This helper returns a complete config from the currently running minion process, including the default values.

# salt minion1 config.items

minion1:
    ----------
    __cli:
        salt-minion
    __role:
        minion
    acceptance_wait_time:
        10
    acceptance_wait_time_max:
        0
    always_verify_signature:
        False
    append_minionid_config_dirs:
    auth_safemode:
        False
    auth_timeout:
        5
    auth_tries:
        7
    auto_accept:
        True
    autoload_dynamic_modules:
        True
    autosign_timeout:
        120
    ...

You can use it to find the difference between two minions (with the help of yamldiff):

# yamldiff --file1 <(salt minion1 config.items --out yaml) \
  --file2 <(salt minion2 config.items --out yaml)

- minion1: {
+ minion2: {
-   uuid: "c6c51464-65cc-b044-94dd-927488c90cfc",
+   uuid: "538dce5d-228c-3d42-83ca-98ab6b999f83",
-  id: "minion1",
+  id: "minion2",
-  ipv6: true,
+  ipv6: false,

PR #48681 by @Arabus

Other notable features

See also:

Also, you can follow me on Twitter where I periodically post things like this:

SaltTips tweet