What's new in Salt 3000 Neon

34 minute read Updated:

Salt Neon

This is an unofficial summary of what’s new in the Salt Neon release. As with Salt Fluorine, it also started as a series of tweets and mostly mentions new features. If you want to read about other changes and deprecations (for example, RAET is gone), then go read the official release notes and the new handcrafted changelog.

UPDATE (2020.05.20): the 3000.3 bugfix release is available and is strongly recommended instead of 3000.

What's new in Salt 3000 Neon: Chroot, Saltcheck, Unless/Onlyif, Slots, Multiple instances, Cloud, Mine, Slack Webhooks, Nifty tricks

New release strategy

Salt Release Strategy

Starting with the Neon release, Salt adopted a new, single branch release strategy and a non-date based version schema beginning at 3000. The version format is MAJOR.PATCH. For a planned release containing features and bug fixes, the MAJOR version will be incremented. The expected release cadence is 3 - 4 months. Since all future development is happening in the master branch (except for critical bugfixes), this means that most of the PRs from the old neon and develop branches are not included in this release and need to be backported. Read the SEP 14, FAQ, and watch the Salt Office Hours for more details.

Security note

For historical reasons, Salt requires PyCrypto as a “lowest common denominator”. However, PyCrypto is unmaintained and best practice is to manually upgrade to use a more maintained library such as PyCryptodome.

There is a picking order as to which package is used:

Due to issues found during package testing for the several supported Linux distributions, Salt temporarily switched back the crypto depencency to PyCrypto.

See the relevant issues: #52674, #56039, #56095

Chroot support

The first execution module (inspired by dockermod) can execute Salt modules and states in a chroot environment. It provides the following functions:

There are two requirements for this to work:

  1. chroot.create creates an empty chroot directory (with dev and proc dirs) and doesn’t install any binaries there (you have to do so yourself)
  2. chroot.call, chroot.apply, chroot.sls and chroot.highstate unpack a Salt thin archive and run it inside a chroot, so having Python installed is also necessary

Freezer

The second execution module can freeze and restore package sets. It provides the following functions:

The produced freeze files (lists of repos and packages) are in yaml format and stored in the /var/cache/salt/minion/freezer folder.

The chroot feature also brings a few related changes:

Details

Saltcheck updates

Saltcheck was introduced by William Cannon in Salt 2013.3.0 and is probably the easiest (built-in) way to test your Salt states automatically, and also validate any existing infrastructure that is not managed with Salt. Saltcheck uses Salt execution modules and a bit of YAML/Jinja (or any other renderer syntax) to make assertions against the desired system state. It may look like someone who saves himself from being drowned by pulling on his own hair, but actually, it is quite effective. Here is an example:

{# baron/munchausen.sls #}
/tmp/swamp.txt:
  file.managed:
    - contents: |
        Münchhausen
{# baron/saltcheck-tests/muchausen.tst #}
ensure_the_swamp_contains_munchausen:
  module_and_function: file.search
  args:
    - /tmp/swamp.txt
    - Münchhausen
  assertion: assertTrue
% sudo salt minion1 state.apply baron.munchausen

minion1:
----------
          ID: /tmp/swamp.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/swamp.txt updated
     Started: 11:03:04.452589
    Duration: 10.000000009313226 ms
     Changes:
              ----------
              diff:
                  New file

Summary for minion1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  10.000 ms
% sudo salt minion1 saltcheck.run_state_tests baron.munchausen

minion1:
    |_
      ----------
      baron.munchausen:
          ----------
          ensure_the_swamp_contains_munchausen:
              ----------
              duration:
                  3.9526
              status:
                  Pass
    |_
      ----------
      TEST RESULTS:
          ----------
          Execution Time:
              3.9526
          Failed:
              0
          Missing Tests:
              0
          Passed:
              1
          Skipped:
              0

In Salt Neon it gained several improvements:

If you haven’t written any tests for your Salt states yet, give it a try. You’ll have an automated safety net and less need to pull out your hair 🙂

Details

Also, Christian wrote an intro titled Validation with SaltStack’s Saltcheck that contains examples of how to:

Cloud modules

Tencent Cloud

A module for Tencent Cloud (the 2nd largest cloud provider in China). It requires the tencentcloud-sdk-python package and provides the following functions:

PR #54526 by likexian (docs: salt.cloud.tencentcloud)

Azure ARM DNS modules

Provide CRUD operations on zones and record sets:

And a couple of states:

PR #55424 by Nicholas Hughes

AWS SSM

New boto_ssm module to manage AWS Systems Manager parameters:

PR #54982 by Christian McHugh

AWS Elasticsearch

Functions:

States:

PR #55768 by Herbert (docs: salt.modules.boto3_elasticsearch, salt.states.boto3_elasticsearch)

vSphere tagging

Add tagging ability to vCenter proxymodule through the following functions:

PR #54058 by @xeacott (docs: salt.modules.vsphere)

Venafi Cloud update

Update to venafi pillar and runner to restore interoperability with Venafi Cloud (automated certificate issuance service). Also adds support for Venafi Trust Protection Platform. Requires the vcert-python library.

PR #55858 by Ryan Treat and Aleksander Rykalin

Other cloud features

Module calls in unless/onlyif

This is probably THE coolest feature in Salt Neon. It allows using any execution module in onlyif/unless requisites (at runtime, unlike Jinja):

{%- from 'postgresql/map.jinja' import pg with context -%}

Enable idle_in_transaction_session_timeout for Postgres 9.6 and later:
  file.managed:
    - name: {{ pg.conf_d }}/session_timeout.conf
    - contents: 'idle_in_transaction_session_timeout = 60000'
    - watch_in:
        service: postgresql
    - onlyif:
      - fun: postgres.psql_query
        query: "SELECT setting FROM pg_catalog.pg_settings WHERE name = 'server_version_num' AND setting::int >= 90600"
        runas: postgres

postgresql:
  service.running:
    - enable: True
    - reload: True

CAVEAT: Unfortunately, this feature doesn’t work everywhere. The following state modules have their own mod_run_check implementations that do not support module calls:

This is going to be fixed in Salt Sodium, see the open PR #55974 by Christian McHugh

PRs #51846 and #52969 by Daniel Wallace and Christian McHugh (docs: Unless and onlyif Enhancements)

Slots

Slots were introduced in Salt 2018.3.0 with the following disclaimer: This functionality is under development and could be changed in future releases. Since then, several issues were filed but haven’t received enough attention from SaltStack employees in the previous major release (2019.2.0, which is almost a year). Another year later, in Salt Neon, some of the issues were fixed with help from the community:

{% set user = 'joeblade' %}

zshrc for {{ user }}:
  file.managed:
    # Dictionary lookup and appended text
    - name: __slot__:salt:user.info({{ user }}).home ~ /.zshrc
    - user: {{ user }}
    - group: {{ user }}
    - template: jinja
    - contents: |
        # ALMIGHTY SYSADMIN, PLEASE DO [NOT] MANAGE THIS FILE FOR ME
        autoload -Uz compinit
        compinit
        alias l='ls -a'
        alias ll='ls -lF'
        alias la='ls -laF'
        alias ..='cd ..'
        alias ...='cd ../..'
        alias ....='cd ../../..'
        alias .....='cd ../../../..'
        alias cd=' builtin cd'
    - context:
        # Context is not used in this example and is here only
        # to demonstrate that slot parsing works inside dicts
        user_name: __slot__:salt:user.info({{ user }}).fullname
    - unless:
      - fun: file.search
        args:
          # Slot as unless argument
          - __slot__:salt:user.info({{ user }}).home ~ /.zshrc
          - "ALMIGHTY SYSADMIN, PLEASE DO NOT MANAGE THIS FILE FOR ME"
        ignore_if_missing: True

Multiple instances

Engines

Existing Salt engines are singletons, i.e. can interact with only one thing. In Salt Neon, it is possible to run multiple instances of any particular engine. Just specify a different engine alias and use the same engine_module parameter:

engines:
  - production_logstash:
      engine_module: logstash
      host: production_log.my_network.com
      port: 5959
      proto: tcp
  - develop_logstash:
      engine_module: logstash
      host: develop_log.my_network.com
      port: 5959
      proto: tcp

The PR is titled Initial work to allow running multiple instances of a Salt engine. The word “initial” could mean that some bits are not implemented yet. During my tests, it worked fine; however, I haven’t tried running anything complex like the reactor engine. Some engines like stalekey are inherently singular and can’t work in parallel. Also, I think it would be helpful if engine name was appended to the process name (when the setproctitle module is installed), because right now it is not apparent what engine a MinionProcessManager runs:

% ps ax | grep '[s]alt'

15988 ?        Ss     0:00 /usr/bin/python3 /usr/bin/salt-minion
16014 ?        Sl     0:03 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager
16017 ?        S      0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiprocessingLoggingQueue
16044 ?        S      0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager
16045 ?        S      0:00 /usr/bin/python3 /usr/bin/salt-minion KeepAlive MultiMinionProcessManager MinionProcessManager

PR #50059 by Gareth J. Greenaway (docs: Engines)

Beacons

Unlike engines, some Salt beacons can monitor multiple things. For example, inotify beacon can watch multiple files. Other beacons (e.g., log) can monitor only one thing. In Salt Neon, it is possible to run multiple instances of any Salt beacon. Just specify a different beacon alias and use the same beacon_module parameter:

beacons:
  watch_importand_file:
    - beacon_module: inotify
    - files:
        /etc/important_file: {}
  watch_another_file:
    - beacon_module: inotify
    - files:
        /etc/another_file: {}

Again, the word “initial” in the commit message could potentially mean that some bits are not implemented yet.

PR #55794 by Gareth J. Greenaway (docs: Configuring beacons)

MetaProxy

The description is quite vague: This PR creates a new loadable module type called a metaproxy and abstracts the existing proxy minion to become a type of metaproxy. This enables us to build different types of proxy minions that can still load existing proxymodules. Below is my personal guess on what this feature could actually mean.

A few facts about proxy minions:

A few details about the MetaProxy PR:

Given all of the above, I think the MetaProxy system is designed to control multiple devices from a single proxy process (probably, one per device vendor). It could save a lot of memory/CPU resources and avoid the need to manage countless proxy minion processes.

And since the PR contains no alternative proxy implementations (and no metaproxy-capable matcher modules), I think the actual solution will be offered only in Salt Enterprise.

After this feature was submitted, a few things happened:

  1. In June 2019, an open-source salt-sproxy (Salt Super-Proxy) package was released by Mircea Ulinic. It uses agentless architecture (similar to Salt SSH or Ansible), and I think it pretty much solves the same problem as the Enterprise MetaProxy - manage more devices with fewer resources. As of Apr 30, 2020 it was downloaded more than 20k times.
  2. In July 2019, MetaProxy was backported to 2019.2.1. It was done against the official development policy, which says that new features should go into the develop branch. As a result, it completely broke the salt-proxy process.
  3. In November 2019, it was announced that the missing Enterprise MetaProxy piece is called Delta Proxy. SaltStack did consider open-sourcing it, but I was informed that the decision hadn’t been made.
  4. And finally, a couple of years later it was open-sourced in Salt 3004 Silicon.

<RANT>

I honestly believe that this is a tough position to be in (i.e., decide which feature should be open and which one shouldn’t). It creates an inherent tension between the open-source community and the company, plus wastes a lot of efforts.

For example, there are multiple attempts to build an open-source GUI for Salt: Alcali, Silica, Molten, SaltGUI, Foreman Salt, Salt Dash, SaltPad, Obdi, Saltshaker, and also Uyuni, the upstream project for SUSE Manager. Instead, those efforts could be spent on improving the official one if it was open.

While it is hard to make money on Open Source projects, I believe that a better way is to have a single open codebase (without artificial Elasticsearch-like license restrictions) and sell high-order (or orthogonal) products and services.

</RANT>

A type of metaproxy is controlled by the following proxy config option (currently, custom metaproxy modules can’t be loaded/synchronized via the fileserver’s _metaproxy folder):

metaproxy: proxy

Since the DeltaProxy isn’t open-source (hopefully yet), the MetaProxy feature has no practical use. If you can provide more insights about MetaProxy or DeltaProxy, I’ll be happy to update this section of the post.

PRs #50183 and #53616 by C. R. Oldham .

Salt Mine

Previously, all mine functions (and their data) were retrievable by all minions. In Salt Neon, it is possible to define a minion-side ACL. When a minion requests a function from the Salt Mine that is not allowed to be requested by that minion, it will get no data, just as if the requested function is not present in the Salt Mine.

mine_functions:
  network.ip_addrs:
    - interface: eth0
    - cidr: 10.0.0.0/8
    - allow_tgt: 'I@role:master'
    - allow_tgt_type: 'compound'

When Salt master receives mine data from a minion, it also receives the defined ACL and stores it together with the data. When another minion requests the mine data, Salt master checks the minion ID against the stored ACL. Obviously, the ACL will be secure only if the targeting criteria can’t be spoofed by a rogue minion (e.g., grains are insecure).

In addition to that, the format to define mine_functions has been extended to allow lists (like in module.run), to support both positional and keyword arguments:

# Old format
mine_functions:
  test.arg:
    foo: foo
    bar: bar

# New format
mine_functions:
  test.arg:
    - foo
    - bar
    - kwarg1: foo
    - kwarg2: bar

The format to define mine aliases also has been extended:

# Old format
mine_functions:
  test_alias:
    - mine_function: test.arg
    - foo
    - bar

# New format
mine_functions:
  test_alias:
    - mine_function: test.arg
    - foo
    - bar
    - kwarg1: foo
    - kwarg2: bar

Both formats are supported simultaneously without any config options or deprecations!

CAVEAT: Unfortunately, it looks like this change broke the mine data format if you use an older master with the new minion: #56118.

PR #55760 by Herbert (docs: Mine functions)

Event relay engines

Fluent engine

The fluent engine reads messages from the Salt event bus and pushes them onto a fluentd endpoint.

Engine configuration (needs the fluent-logger-python module installed):

engines:
  - fluent:
      host: localhost
      port: 24224
      app: engine

Example fluentd configuration:

<source>
    @type forward
    port 24224
</source>
<match saltstack.**>
    @type file
    path /var/log/td-agent/saltstack
</match>

PR #55711 by Christian McHugh (docs: salt.engines.fluent)

Script engine

Sometimes it is necessary to consume a structured event log (or an external data source) and inject the result into Salt’s event bus. The script engine creates events based on a command’s output and can run on a Salt master or minion. It reads the output line by line and deserializes each line into a data structure using any serializer. If the resulting data structure contains the tag key, then an event will be fired on the event bus (with an optional data payload).

#/etc/salt/minion.d/engines.conf
engines:
  - script:
      cmd: /etc/salt/script.sh
      output: json
#!/bin/bash
# /etc/salt/script.sh

while true; do
    echo -e '{"tag": "thermal_zone0", "data": {"temp": "'$(cat /sys/class/thermal/thermal_zone0/temp)'"}}'
    sleep 5
done
% sudo salt-run state.event pretty=True

thermal_zone0	{
    "_stamp": "2019-09-04T15:08:46.220902",
    "cmd": "_minion_event",
    "data": {
        "id": "minion1",
        "temp": "50000"
    },
    "id": "minion1",
    "pretag": null,
    "tag": "thermal_zone0"
}
thermal_zone0	{
    "_stamp": "2019-09-04T15:08:51.224414",
    "cmd": "_minion_event",
    "data": {
        "id": "minion1",
        "temp": "50000"
    },
    "id": "minion1",
    "pretag": null,
    "tag": "thermal_zone0"
}

If a script doesn’t run indefinitely, you can run it periodically by adding interval: N to the engine configuration.

Two caveats:

PR #50005 by austin (docs: salt.engines.script)

Certificate expiration beacon

SSL/TLS certificates tend to expire at inappropriate times. If you do not have any certificate renewal automation (e.g., Certbot), it is worth setting up expiry reminders. You can use the new cert_info beacon to do so:

# This can go either to a minion config or to a pillar
beacons:
  cert_info:
    - files:
        - /tmp/puppet.pem
    - notify_days: 45
    - interval: 86400

When a certificate is about to expire, you will get a Salt event that looks like this:

% sudo salt-run state.event 'salt/beacon/*/cert_info/' pretty=True

salt/beacon/minion1/cert_info/	{
    "_stamp": "2020-01-09T09:21:43.184517",
    "certificates": [
        {
            "cert_path": "/tmp/puppet.pem",
            "extensions": [
                {
                    "ext_data": "DNS:puppet.com, DNS:docs.puppet.com, DNS:www.puppet.com",
                    "ext_name": "subjectAltName"
                }
            ],
            "has_expired": true,
            "issuer": "C=\"FR\",ST=\"Paris\",L=\"Paris\",O=\"Gandi\",CN=\"Gandi Standard SSL CA 2\"",
            "issuer_dict": {
                "C": "FR",
                "CN": "Gandi Standard SSL CA 2",
                "L": "Paris",
                "O": "Gandi",
                "ST": "Paris"
            },
            "notAfter": "2019-11-05 23:59:59Z",
            "notAfter_raw": "20191105235959Z",
            "notBefore": "2018-11-05 00:00:00Z",
            "notBefore_raw": "20181105000000Z",
            "serial_number": "333090271171444178457395344333836335102",
            "signature_algorithm": "sha256WithRSAEncryption",
            "subject": "OU=\"Domain Control Validated\",OU=\"PositiveSSL Multi-Domain\",CN=\"puppet.com\"",
            "subject_dict": {
                "CN": "puppet.com",
                "OU": "PositiveSSL Multi-Domain"
            },
            "version": 2
        }
    ],
    "id": "minion1"
}

Now let’s do something more useful - forward these events to a Slack channel using the new webhook-based state module.

1. Place the following snippet into /etc/salt/master.d/reactor.conf

reactor:
  - salt/beacon/*/cert_info/:
    - /srv/salt/reactor/cert_info.sls

slack:
  # Create an incoming webhook here: https://api.slack.com/messaging/webhooks
  hook: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

2. Define the reactor state using the new json_query Jinja filter

# /srv/salt/reactor/cert_info.sls
Send expired certs to Slack:
  runner.state.orchestrate:
    - args:
      - mods: slack
      - pillar:
          expired_cert_minion: {{ data['id'] }}
          expired_cert_message: |
            {{ data['certificates'] | json_query("[].[cert_path, extensions[?ext_name=='subjectAltName'].ext_data | [0], notAfter] | map(&[] | join(' :: ', @), @) | join('\n', @)") }}

Do not forget to run sudo apt-get install jmespath on the Salt master to enable the filter.

3. Define the Slack notification state

# /srv/salt/slack.sls
slack-message:
  slack.post_message:
    - message: |
        The following certificate(s) are about to expire on "{{ pillar["expired_cert_minion"] }}":
        {{ pillar["expired_cert_message"] }}
    - webhook: {{ salt['config.get']('slack:hook') }}

And then restart the Salt master. You should see something like this when a certificate is about to expire:

Expired Certificate Slack Notification

PR #54902 by Nicholas Hughes

Support for Slack webhooks

Slack state

The slack.post_message state has been updated to support webhooks (in addition to API keys). If a webhook argument is defined, the state will use it to post the message:

slack-message:
  slack.post_message:
    - message: Hello there!
    - username: SaltStack
    - icon_emoji: ":glitch_crab:"
    - channel: "#random"
    - webhook: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

For security reasons, it is better to store the webhook URL in a master or minion config:

slack:
  hook: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

And then use - webhook: {{ salt['config.get']('slack:hook') }} in your state instead of the URL.

The old way of using API keys is still supported:

slack-message:
  slack.post_message:
    - message: Hello there!
    - from_name: SaltStack
    - icon: https://cdn.mirantis.com/wp-content/uploads/2017/02/image01.png
    - channel: "#random"
    - api_key: AAABBBCCC

PR #52715 by Gareth J. Greenaway (docs: salt.states.slack.post_message)

Slack webhook returner

The slack_webhook is a new webhook returner based on the existing slack returner that uses API keys.

Place the following snippet into /etc/salt/minion.d/slack.conf and restart the minion:

slack_webhook:
  webhook: T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  show_tasks: true

Then you’ll be able to use the returner as follows:

% sudo salt minion1 state.sls neon.warning --return slack_webhook

Slack Webhook Returner

CAVEAT: It only works with state.apply, state.sls, and state.highstate functions. This limitation will be fixed in #55968.

PR #55342 by Carlos D. Álvaro

Misc modules

XML module and state

This is a simple XML execution module to get and set XML attributes and values. The corresponding state module has just one function: xml.value_present.

# rss.sls
HackerNews:
  file.managed:
    - name: /tmp/hn.rss
    # Also try https://hnrss.org/newest
    - source: https://hnrss.org/frontpage?count=30
    - skip_verify: true

{% set idx = salt['random.rand_int'](start=1, end=30) %}

Random HN item:
  module.run:
    - name: test.arg
    - kwargs:
        title: __slot__:salt:xml.get_value(/tmp/hn.rss, channel/item[{{ idx }}]/title)
        url: __slot__:salt:xml.get_value(/tmp/hn.rss, channel/item[{{ idx }}]/link)
% sudo salt minion1 state.apply rss --out json | \
  jq '.[][] | select(.__id__ | contains("Random HN item")) | .changes.ret.kwargs'

{
  "title": "The Terrors and Joys of Terraform",
  "url": "https://medium.com/driven-by-code/the-terrors-and-joys-of-terraform-88bbd1aa4359"
}

PR: #54977 by Christian McHugh (docs: XML module)

ssh_auth.manage state

This state makes unnecessary having both ssh_auth.present and ssh_auth.absent to control authorized ssh keys. Now you can specify a single list of authorized keys to ensure that there are no stale ones:

all_ssh_keys:
  ssh_auth.manage:
    - user: joeblade
    - enc: ssh-dss
    - options:
      - option1="value1"
    - ssh_keys:
      - AAAABBBBCCCCDDD
      - ssh-rsa DDDDEEEEFFFFGGGG== colin@swinbourne.local
      - GGGGHHHHIIIIJJJJ== interceptor@micros.local

PR #54981 by Christian McHugh (docs: ssh_auth.manage)

Keystore module

New keystore execution and state modules to manage Java Keystore files are shipped in Salt Neon. To enable them, install the pyjks Python library.

The following execution functions are provided:

# salt-call keystore.list /path/to/keystore.jks changeit

local:
  |_
    ----------
    alias:
        hostname1
    expired:
        True
    sha1:
        CB:5E:DE:50:57:99:51:87:8E:2E:67:13:C5:3B:E9:38:EB:23:7E:40
    type:
        TrustedCertEntry
    valid_start:
        August 22 2012
    valid_until:
        August 21 2017

The state module contains the keystore.managed state:

define_keystore:
  keystore.managed:
    - name: /tmp/statestore.jks
    - passphrase: changeit
    - force_remove: True
    - entries:
      - alias: hostname1
        certificate: /tmp/testcert.crt
      - alias: remotehost
        certificate: /tmp/512.cert
        private_key: /tmp/512.key
      - alias: stringhost
        certificate: |
          -----BEGIN CERTIFICATE-----
          MIICEjCCAX
          Hn+GmxZA
          -----END CERTIFICATE-----

PR #54991 by Christian McHugh (docs: salt.modules.keystore, salt.states.keystore)

IIS webconfiguration settings

A module and state to manage IIS WebConfiguration settings:

PR #54879 by Thomas Lemarchand (docs: salt.modules.win_iis, salt.states.win_iis)

Elasticsearch functions

PRs #54917 and #55721 by Proskurin Kirill by Proskurin Kirill (docs: salt.modules.elasticsearch)

RabbitMQ upstreams management

PR #55767 by Herbert (docs: salt.modules.rabbitmq, salt.states.rabbitmq_upstream)

Django migrations

A little helper function to apply Django migrations:

% sudo salt minion1 django.migrate <settings_module>

Or via state:

From South to North:
  module.run:
    - name: django.migrate
    - settings_module: my_django_app.settings

PR #54632 by @jrbeilke (docs: django.migrate)

Custom module helpers

saltutil state

Previously, it was only possible to run saltutil.sync_* functions via module.run, and it always reported changes. Now there is a corresponding state module that provides the following states:

The usage is straightforward:

sync_everything:
  saltutil.sync_all:
    - refresh: True

Test mode (test=True) is supported, but it doesn’t actually test for possible changes and just prints saltutil.sync_* would have been run.

PRs #50197 by Christian McHugh and #51900 by Max Arnold (docs: salt.states.saltutil)

Ability to sync custom executor modules

Allow syncing custom executor modules from salt://_executors directory via the saltutil.sync_executors or saltutil.sync_all functions. Executor modules (do not confuse them with execution modules) wrap state execution calls globally, for example, to use sudo or docker.call.

PR #55190 by Matt Phillips and Max Arnold

__utils__ in grains

Since custom utils (synced via saltutil.sync_utils) modules aren’t importable; it was impossible to use them in custom grain modules. Now you can call custom utility functions from grain functions via the loader:

# _utils/my_util.py
def my_func():
    return 'abc'
# _grains/my_grain.py
def my_grain():
    return {'my_grain': __utils__['my_util.my_func']()}
% sudo salt minion1 saltutil.sync_utils,saltutil.sync_grains ,

minion1:
    ----------
    saltutil.sync_grains:
        - grains.my_grain
    saltutil.sync_utils:
        - utils.my_util
% sudo salt minion1 grains.get my_grain

minion1:
    Hello there!

This is useful if you have a couple of custom grains that use common utility functions.

PR #49128 by @mirceaulinic

Date-based deprecation warning

This is a new utility function to emit date-based deprecation warnings in Salt modules:

import salt.utils.versions

salt.utils.versions.warn_until_date(
    '20201201',
    'Please stop using X before {date} and instead use Y'
)

After the specified date, the warning will turn into a RuntimeError.

Previously it was only possible to emit warnings based on release code names:

import salt.utils.versions

salt.utils.versions.warn_until(
    'Neon',
    'RAET is deprecated and will be removed in Salt {version}'
)

PR #55047 by Pedro Algarvio

Version-aware dependency decorator

The @salt.utils.decorators.depends decorator gained an optional version keyword. This decorator will check the module when it is loaded, check that the dependencies passed in are in the globals of the module and that the version requirements are met. If not, it will cause the function to be unloaded (or replaced if fallback_function is specified).

import salt.utils.decorators

@depends('botocore', version='1.12.21')
def a_function_that_depends_on_botocore():
    pass

PR #55590 by Herbert

filter_falsey

The salt.utils.data.filter_falsey function filters values from an iterable that evaluate to false (a typical use-case for most Boto calls). Removes None, {} and [], 0, '', but does not remove False.

In [1]: import salt.utils.data

In [2]: salt.utils.data.filter_falsey({'foo': None, 'bar': True})
Out[2]: {'bar': True}

In [3]: salt.utils.data.filter_falsey([True, False, None, {}, [{}, 'blah', '']])
Out[3]: [True, False, [{}, 'blah', '']]

In [3]: salt.utils.data.filter_falsey(
   ...: [True, False, None, {}, [{}, 'blah', '']],
   ...: ignore_types=[dict],
   ...: recurse_depth=True)
Out[3]: [True, False, {}, [{}, 'blah']]

PR #52499 by Herbert

recursive_diff

The salt.utils.data.recursive_diff function is able to produce a dict with old and new keys for any combination of nested maps or iterable types:

In [1]: import salt.utils.data

In [2]: salt.utils.data.recursive_diff(
   ...: {'foo': [1, 2, 3]},
   ...: {'foo': [3, 2, 1]})
Out[2]: {'new': {'foo': [3, 1]}, 'old': {'foo': [1, 3]}}

PR #55759 by Herbert

Jinja helpers

json_query filter

This is a port of Ansible json_query Jinja filter to make complex queries against JSON data structures. The query language parser depends on jmespath python library.

It could be used to filter pillar data, event data, yaml maps, and in combination with http_query. Can replace lots of ugly Jinja loops and simplify data parsing (especially if you use the TOFS pattern). In some cases, it can help to avoid writing trivial custom modules.

# json_query_example.sls
{% set services = '
  {"services": [
    {"name": "http", "host": "1.2.3.4", "port": 80},
    {"name": "smtp", "host": "1.2.3.5", "port": 25},
    {"name": "ssh",  "host": "1.2.3.6", "port": 22}
  ]}' | load_json %}

{% set ports = services | json_query("services[].port") %}
{% do salt.log.warning(ports) %}


% sudo salt-call state.apply json_query_example -l warning

[WARNING ] [80, 25, 22]

PR #50428 by Max Arnold (docs: json_query)

A helper to debug Jinja map files

Troubleshooting TOFS-based formulas could be tricky, especially when multiple yaml maps are merged and filtered inside a map.jinja file. The jinja execution module can help you debug these data structures:

% sudo salt minion1 jinja.import_yaml template-formula/defaults.yaml

minion1:
    ----------
    template:
        ----------
        added_in_defaults:
            defaults_value
        config:
            /etc/template
        pkg:
            ----------
            name:
                template
        rootgroup:
            root
        service:
            ----------
            name:
                template
        winner:
            defaults
% sudo salt minion1 jinja.load_map template-formula/map.jinja defaults

minion1:
    ----------
    config:
        /etc/template.d/custom-ubuntu-18.04.conf
    pkg:
        ----------
        name:
            template-ubuntu

PR #51047 by Erik Johnson (docs: salt.modules.jinja)

New dictupdate helpers

This feature adds a few functions to modify nested dicts, and exposes several Jinja filters (with support of configurable key delimiters and optional use of ordered dicts):

PR #52455 by Herbert

hmac_compute filter

The new hashutil.hmac_compute function and @hmac_compute Jinja filter to calculate an HMAC digest using SHA-256:

% sudo salt-call hashutil.hmac_compute "Hello there!" "A secret"

local:
    0c9d76aa8bcb2c4138af07e1f9d6a7393352948f42f3b18f3d8043c9242a260c

% sudo salt-call slsutil.renderer default_renderer=jinja \
  string='{{ "Hello there!" | hmac_compute("A secret") }}'

local:
    0c9d76aa8bcb2c4138af07e1f9d6a7393352948f42f3b18f3d8043c9242a260c

PR #55506 by @Ajnbro

Camels and snakes

The following two functions could be used either in Jinja templates or directly in Python modules. The primary use-case seems to be converting Boto parameter names. The functions named camel_to_snake_case and snake_to_camel_case are located in the salt.utils.stringutils module. They are also exposed as Jinja filters to_snake_case and to_camelcase (yes, the naming is inconsistent). The usage is simple:

% sudo salt-call slsutil.renderer default_renderer=jinja \
  string="{{ 'snake_case_for_the_win' | to_camelcase }}"

local:
    snakeCaseForTheWin
% sudo salt-call slsutil.renderer default_renderer=jinja \
  string="{{ 'camelsWillLoveThis' | to_snake_case }}"

local:
    camels_will_love_this

PR #52458 by Herbert

macOS

PyObjC bindings

The PyObjC library is now shipped with the macOS package. It allows calling native Objective-C APIs from Python without shelling out. AFAIK, there are no builtin modules in Salt that use it (yet). However, you can check out the mosen/salt-osx repo to see what is possible.

PR #49657 by Wesley Whetstone (also backported to 2019.2.0)

Homebrew improvements

List the right namespace for cask packages from a tap different from the default one. The caskroom/cask/ namespace for brew-cask packages is deprecated in favor of homebrew/cask/ and will be removed in Sodium release.

PR #54216 by Carlos D. Álvaro

Virt improvements

Enhancements in package modules

Grains

HTTP improvements

Multimaster tweaks

Performance tweaks

Filesystems and partitions

Files and archives

Other notable features

Nifty tricks

loop.until_no_eval state

This is a generic state that blocks the execution process until a specific Salt function returns an expected result. Unlike the loop.until state it doesn’t use potentially unsafe Python eval() function. Instead, it can use a function from Python’s operator module, __salt__, or __utils__. Below is an example of how to wait until an Amazon ELB instance is healthy:

Wait for service to be healthy:
  loop.until_no_eval:
    - name: boto_elb.get_instance_health
    - expected: '0:state:InService'
    - compare_operator: data.subdict_match
    - period: 5
    - timeout: 20
    - args:
      - {{ elb }}
    - kwargs:
        keyid: {{ access_key }}
        key: {{ secret_key }}
        instances: "{{ instance }}"

PR #55639 by Herbert (docs: salt.states.loop)

salt_version execution module

Did you ever have to invent a workaround in a state file to support older Salt versions? Or maybe wrote formulas that worked across Salt versions? You might find this new module quite useful:

% sudo salt minion1 salt_version.equal Neon

minion1:
    True
# And also in Jinja:
{% if salt['salt_version.equal']('Sodium') or
      salt['salt_version.greater_than']('Sodium') %}
superseded_syntax_for_module_run:
  module.run:
    - test.echo:
      - text: 'Salt Sodium or newer'
{% else %}
legacy_syntax_for_module_run:
  module.run:
    - name: test.echo
    - text: 'Older than Salt Sodium'
{% endif %}

PR #55195 by Nicole Thomas and Max Arnold

Custom state warnings

You may want to set up a highstate process warning to notify yourself about something important. Below is a little helper that allows you to propagate custom warnings back to salt CLI output (unlike the salt.log.warning jinja function that is only visible in a minion log):

# warning.sls
A warning:
  test.configurable_test_state:
    - result: True
    - warnings:
        - Hello
        - there
% sudo salt minion1 state.apply warning

minion1:
----------
          ID: A warning
    Function: test.configurable_test_state
      Result: True
     Comment:
     Started: 06:25:55.749509
    Duration: 2.846 ms
     Changes:
              ----------
              testing:
                  ----------
                  new:
                      Something pretended to change
                  old:
                      Unchanged
    Warnings: Hello there

Summary for minion1
------------
Succeeded: 1 (changed=1)
Failed:    0
Warnings:  1
------------

PR #53959 by Max Arnold

Enhanced config.option

The config.option function has been enhanced:

To view the sane defaults, you can use the following master and minion commands respectively:

% sudo salt-run salt.cmd config.option '*' omit_all=True wildcard=True

% sudo salt minion1 config.option '*' omit_all=True wildcard=True

Additionally, configuration for Docker registries is no longer restricted only to pillar data and is now loaded using config.option (see #51531).

PR #55637 by Erik Johnson (docs: salt.modules.config.option)

Attaching grains to minion start events

Device onboarding often requires getting initial system attributes and storing them in a CMDB. Previously, the common way to do that was through the Salt reactor, which required an additional round trip. The new feature reduces the master load by automatically attaching any grains to the salt/minion/*/start event. To enable it, add the list of grains to a minion config file:

start_event_grains:
  - machine_id
  - uuid

After you restart the minion you’ll see the following events on the master bus:

minion_start	{
    "_stamp": "2020-01-15T07:40:13.655558",
    "cmd": "_minion_event",
    "data": "Minion minion1 started at Tue Jan 14 23:40:13 2020",
    "grains": {
        "machine_id": "599118d5b9619443ac3166fb0e59349e",
        "uuid": "599118d5-b961-9443-ac31-66fb0e59349e"
    },
    "id": "minion1",
    "pretag": null,
    "tag": "minion_start"
}
salt/minion/minion1/start	{
    "_stamp": "2020-01-15T07:40:13.662584",
    "cmd": "_minion_event",
    "data": "Minion minion1 started at Tue Jan 14 23:40:13 2020",
    "grains": {
        "machine_id": "599118d5b9619443ac3166fb0e59349e",
        "uuid": "599118d5-b961-9443-ac31-66fb0e59349e"
    },
    "id": "minion1",
    "pretag": null,
    "tag": "salt/minion/minion1/start"
}

The event output above is a reminder to set the enable_legacy_startup_events: False in a minion config to get rid of the legacy event.

PRs #54948 and #55885 by Abid Mehmood (docs: start_event_grains)

minion_id domain removal

When there is no explicit id defined in a minion config, Salt uses the following algorithm:

  1. If an id_function name is defined in the minion config (with optional kwargs), it will be called to generate an id
  2. Otherwise, the salt.utils.network.generate_minion_id() will be invoked to generate an id based on host’s FQDN
  3. Then, if the minion_id_lowercase option is true, the resulting id will be lowercased
  4. Then the new minion_id_remove_domain option is considered
  5. And finally, the optional append_domain value is appended

The minion_id_remove_domain option can take the following values:

  1. false (default) - do nothing
  2. A specific domain name (e.g., example.com) - the domain name will be removed from a minion id (minion.example.com -> minion, minion.foo.bar -> minion.foo.bar)
  3. true - anything after the first dot will be removed from a minion id (e.g., minion.example.com -> minion, minion.foo.bar -> minion)

PR #54622 by @markuskramerIgitt (docs: minion_id_remove_domain)

Saltenv support in slsutil.renderer

The slsutil.renderer function is a useful way to debug Jinja templates. Now it can handle saltenvs:

% sudo salt minion1 slsutil.renderer salt://state.sls default_renderer=jinja saltenv=base

PR #52293 by Alexander Fischer

DSON outputter

Salt Doge Meme

kabosu112.exblog.jp

This outputter deserves to be highlighted. It uses DSON format (Doge Serialized Object Notation) to represent the output of any Salt command. So needed! Just run pip install dogeon and then:

% sudo  salt minion1 test.echo 'Such salty!' --out dson

such
    "minion1" is "Such salty!"
wow

PR #49338 by Erik Johnson

Also, you can write your Salt states using DSON. The corresponding renderer was added in Salt 2016.11 after SaltConf 16. Much readable!

#!dson
such
    "Very id. So unique." is such
        "test.configurable_test_state" is so
            such
                "comment" is "Much readable!"
            wow
        many
    wow
wow

See also: