What's new in Salt 3001 Sodium

27 minute read Updated:

Salt Sodium

This is an unofficial summary of new features in the Salt Sodium release. Due to time constraints, the post is not as detailed as I initially planned. If you want to read about other changes and deprecations, then go read the official release notes and the changelog.

As it often happens with major Salt releases, there are a couple of known issues in 3001. The standard advice is to test the new release as much as you can, but avoid upgrading your infra till the next point release.

What's new in Salt 3001 Sodium: Python 3, Developer-related changes, Performance, Cloud, Networking, SaltSSH, Grains, Saltcheck, Jinja, Vault, Certificates, Nifty tricks

Python 3

There are a couple of significant changes:

  1. Python 2 is no longer supported. See SEP-5 and the Python 2 deprecation FAQ for more details.
  2. The minimum supported Python version is 3.5. It defines the baseline Python features to use (e.g., you can’t use f-strings). Also, some features that depend on Py2-only modules will stop working. The exact Python version used depends on the OS and the packaging method. See SEP-20 for more details.
  3. Python 3.8 and Ubuntu 20.04 LTS are finally supported!
  4. To support older systems that do not ship with Python >= 3.5, Tiamat will be used to bundle Python 3 with Salt packages. This is going to be a beta test for all future packaging of Salt (see the daily builds and the resulting artifacts).
  5. SaltSSH gained a new feature to manage Py2-only systems. See SEP-8 for more details.

If you haven’t switched your Salt install to Python 3, I have a handy guide available.

Packaging and dependencies

Unfortunately, the switch to Tornado >= 5 is still in progress and won’t happen until Salt Magnesium. So far, there is at least one change that prevents starting multiple ioloops in a single thread. You can track the progress here.

A couple of logs are properly rotated now:

FreeBSD platform support became a bit closer to “officially blessed”:

There are a couple of changes that are important to Salt contributors:

Black 🖤

The Salt codebase and docs have been blackened! The intent was to spend less time fixing formatting issues in pull requests and instead focus more on tests and code quality. Also, say hello to double quotes everywhere (or write your own formatter to format the code back to your liking before editing it) 😬

The commit hashes were added to .git-blame-ignore-revs to simplify the git blame workflow.

See SEP-15 for more details. PRs #55765 by Wayne Werner and #57596 by Pedro Algarvio

Documentation

A couple of development-related pages were improved:

The documentation quality control process became a bit stricter:

The included checks are:

Pytest & nox

The transition from the custom runtests.py runner to Pytest/nox started somewhere in 2017 and spanned a few releases. The majority of work was done after the Salt 2019.2.0 Fluorine release.

As far as I know, the switch to Pytest didn’t happen in Salt Sodium, and both runtests and pytest will be running for some time. However, the amount of changes is quite impressive (see these PRs by Pedro Algarvio ).

For more info, read the Salt’s Test Suite web page that describes the new nox workflow. Also check out the Pytest - a journey of three years blog post.

Pre-commit hooks

A couple of pre-commit hooks were added to automate the manual steps:

Salt pre-commit hooks

Below is a little cheat sheet on how to start using these new tools.

Bootstrap the development environment:

mkvirtualenv -p $(which python3.7) salt
setvirtualenvproject
hash -r
pip install nox
pip install pre-commit
pre-commit install-hooks

A couple of useful nox commands:

Documentation commands:

Run the pre-commit hook manually (only check staged files):

It is also possible to run the pre-commit hook on arbitrary files:

Run a specific test:

Faster commits:

Performance

salt-call --local speedup

This one-line fix results in almost 3-fold performance increase (its author reports even more significant speedup):

# BEFORE:
% time sudo salt-call --local test.ping
local:
    True

real    0m2.961s
user    0m2.733s
sys     0m0.201s


# AFTER:
% time sudo salt-call --local test.ping
local:
    True

real    0m1.022s
user    0m0.756s
sys     0m0.197s

PR #57062 by Ivan Babrou and PR #57172 by Erik Johnson

esxi grain optimization

According to the PR, this short fix shaves 3⁄4 of a test.ping duration on Windows, by making the vsphere module import conditional.

PR #57579 by Markus

Async minion ping on key rotation

When the ping_on_rotate master option is set to true, the master will ping all connected minions to trigger a key refresh. This change avoids blocking the master Maintenance process when doing so.

PR #56791 by @pengyao

Lazily list available states

This change significantly improves the state.apply performance when using GitFS and a lot of branches. This is particularly useful when top_file_merging_strategy=same, and there are many environments.

PR #54468 by Mathieu Parent

Salt Master subprocess niceness

A couple of new settings to set niceness of various salt-master processes. It could be useful under high load to ensure that critical parts of Salt aren’t resource-starved. The settings are available only on POSIX platforms and set to None by default:

PR #57365 by Matt Phillips

Reactor leader

In multi-master hot-hot environments, all events are streamed to all masters for a given minion. This introduces the problem of a reactor being executed on each hot master, regardless of if the reaction is idempotent or not. This patch adds a simple is_leader runtime flag (True by default) that can be controlled via the reactor runner (or emitting the equivalent event to toggle) whether a reactor in hot-standby mode should be reacting to events coming into the bus. This provides a simple integration point for an external leader election tool, such as consul-template.

To query the flag, run the salt-run reactor.is_leader command. To change it run the salt-run reactor.set_leader True or salt-run reactor.set_leader False command.

Unfortunately, the feature didn’t work for me:

% sudo salt-run reactor.is_leader

event:
    ----------
    _stamp:
        2020-06-02T10:22:42.528292
    key:
        REDACTED
suffix:
    salt/reactors/manage/is_leader

Passed invalid arguments: 'NoneType' object is not subscriptable

Usage:

    Return whether the running reactor is acting as a leader (responding to events).

    CLI Example:

    .. code-block:: bash

        salt-run reactor.is_leader

PR #56856 by Matt Phillips

Parallel slots

Not exactly a performance improvement, but now slots are actually expanded and parallelized with parallel: true.

PR #56221 by Max Arnold

Faster fqdns grain

Gate the fqdns grain behind the enable_fqdns_grains setting (true by default, except for Windows) and calculate its values in parallel. This helps to reduce or completely avoid blocking other core grains.

As a side effect, the time to run the fast subset of tests on Windows went down from ~4 hours to ~50 minutes!

PRs #55581 by Pablo Suárez Hernández , #57576 by Shane Lee

Faster grain.setval

Add refresh_pillar kwarg (true by default) to grains.setval and grains.setvals functions. By setting it to false, you can skip the pillar refresh.

PR #56573 by @jtraub91

Reactor queue overflow warning

This is a minor change, but it will help you notice that the reactor TaskPool queue is full. The reactor_worker_threads and reactor_worker_hwm settings could be used to remediate that.

PR #56787 by @geekinutah

GPG renderer cache

A master can cache GPG data locally to bypass the expense of having to render them for each minion on every request. This feature should only be enabled in cases where pillar rendering time is known to be unsatisfactory, and any attendant security concerns about storing decrypted GPG data in a master cache have been addressed. The caching process is controlled with the following settings:

PR #55772 by Mathieu Parent

Cloud

Kubeadm module

An initial version of the kubeadm execution module has the following functions:

There are many missing functions, see the various TODOs in the source code if you want to help.

PR #53345 by Alberto Planas

Helm support

A new module and state to manage Helm. Requires Helm-CLI v3.0.

Execution module functions:

States:

PR #56081 by Grumpy

Virt changes

Other cloud changes

Grains

State system enhancements

Unless/onlyif/creates requisite unification

The ability to use execution modules in unless/onlyif requisites (introduced in Salt Neon) has been enhanced. Now it also works within the following states:

This change also introduces global creates requisite (previously was only available within the cmd state):

Contrived creates example:
  file.touch:
    - name: /path/to/file
    - creates: /path/to/file

PRs #55974 and #56381 by Christian McHugh

onfail_all requisite

The onfail_all requisite uses AND logic when multiple states are referenced (unlike the onfail_any that uses OR):

test_site_a:
  cmd.run:
    - name: ping -c1 10.0.0.1

test_site_b:
  cmd.run:
    - name: ping -c1 10.0.0.2

notify_site_down:
  hipchat.send_message:
    - room_id: 123456
    - message: "Both primary and backup sites are down!"
  - onfail_all:
    - cmd: test_site_a
    - cmd: test_site_b

In this contrived example, notify_site_down will run when both 10.0.0.1 and 10.0.0.2 fail to respond to ping.

PR #56831 by Matt Phillips

Ability to disable requisites

This is a new minion config option to disable certain kinds of state requisites:

disabled_requisites:
  - require
  - require_in

It is hard to come up with any use-case for this feature and the PR is not very verbose (Adding the ability to disable requisites during state runs). If the feature was more granular and it was possible to disable specific state IDs, then it potentially could be used to disable certain parts of upstream formulas.

My only guess is that right now, it may be used to unit-test some complex states in isolation. However, the only unit test in Salt Sodium that uses this option is the one that checks the feature itself…

PR #56815 by Gareth J. Greenaway

state.test function

The state.test function as an alias to state.apply test=True. It is easier to type and is less susceptible to typos.

PR #56298 by Ryan Addessi

SaltSSH

Saltcheck

Multiple Saltcheck assertions

Allow Saltcheck tests to specify multiple assertions against the output of a single module_and_function call. The assertion, expected_return, assertion_section, and assertion_section_delimiter keys can be placed in a list under an assertions key.

multiple checks on complicated output:
  module_and_function: network.netstat
  assertions:
    - assertion: assertEqual
      assertion_section: "0:program"
      expected_return: "systemd-resolve"
    - assertion: assertEqual
      assertion_section: "0:proto"
      expected_return: "udp"
  output_details: True
  print_result: True

PR #56101 by Christian McHugh

Parallel Saltcheck tests

Saltchecks tests will be run in parallel by adding saltcheck_parallel: True in minion config. Setting this value to an integer will set the maximum parallel processes. Otherwise, the number of processes will be the minimum between the number of CPUs or the number of tests.

PR #56097 by Christian McHugh

Jinja

IP filtering by network

The filter_by_network Jinja filter returns the list of IPs filtered by the network list.

{% set networks = ['192.168.0.0/24', 'fe80::/64'] %}
{{ grains['ip_interfaces'] | filter_by_networks(networks) }}
{{ grains['ipv6'] | filter_by_networks(networks) }}
{{ grains['ipv4'] | filter_by_networks(networks) }}

PR #56394 by Alexander Graul

method_call filter

Another interesting Jinja filter allows you to call object methods. Using it directly is not very practical. However, in combination with the map filter it enables some cool tricks without resorting to Jinja loops:

# method_call.sls
{% set hostnames = [
  'web01.example.com',
  'db01.example.com'
] %}

{% set names = hostnames |
  map('method_call', 'split', '.', 1) |
  map('method_call', '__getitem__', 0) |
  list
%}

{% do salt.log.warning(names) %}
% sudo salt-call state.apply method_call -l warning

[WARNING ] ['web01', 'db01']

PR #56765 by @zer0def

Other Jinja changes

macOS

Windows

Vault

Networking

Cisco NX-OS

Huge NX-OS patchset:

PR #54931 by Mike Wiebe , @tstoner , Nicole Thomas , and David Hilton

Junos

Many bugfixes in Junos-related components:

Other networking changes

Gitfs

Filesystems and partitions

Files and directories

Package managers

Certificates

Nifty tricks

API call to fetch the master public key

The goal of this feature is to simplify programmatic minion bootstrapping/pre-seeding without hardcoding a master public key. The new wheel.key.master_key_str function can return an existing pubkey through the python API or netapi.

PR #56550 by @alexz71

file.keyvalue state

The state was announced at SaltConf18 and is intended to work with default configuration files, which have the defaults commented out (not all defaults might be in there, though). It can be used instead of a more low-level file.replace, file.append, file.blockreplace, and file.line states to:

sshd_config_harden:
  file.keyvalue:
    - name: /etc/ssh/sshd_config
    - key_values:
        permitrootlogin: 'without-password'
        LoginGraceTime: '1m'
        DisableForwarding: 'yes'
    - separator: ' '
    - uncomment: '# '
    - key_ignore_case: True
    - append_if_not_found: True

PR #56770 by Mattijs

Support extra modules in sys.path

Salt can sync custom utils modules like any other module types (modules, states, etc.). However, salt.utils are often imported directly (instead of using the __utils__ dunder), and this prevents overriding them with custom versions. This improvement makes custom utils importable.

PR #53167 by Alberto Planas

Relative pillar includes

Sometimes it takes seven years to fix an issue. Now it is finally possible to use relative pillar includes (state files supported them since 0.16.0):

# foo/bar/init.sls
# Dot notation will include foo/bar/base.sls instead of base.sls

include:
  - .base

Unfortunately, multiple dots aren’t supported in pillars (in state files it is possible to use include: - ..base).

PR #56851 by Wayne Werner

Synchronous pillar refresh

This feature adds an async parameter to the saltutil.refresh_pillar function (on a par with the saltutil.refresh_modules). When set to False, it will block a minion until a /salt/minion/minion_pillar_complete event is received:

% sudo salt minion1 saltutil.refresh_modules async=False

minion1:
    True

It is not yet possible to specify this argument for the saltutil state functions, so in order to use it in a state file you still have to resort to module.run.

PR #56881 by Alexander Fischer

File banner

Have you ever put the “DO NOT EDIT THIS FILE BY HAND” header into a file managed by Salt? Well, there’s a feature for that:

% sudo salt minion1 slsutil.banner
minion1:
    ########################################################################
    #                                                                      #
    #              THIS FILE IS MANAGED BY SALT - DO NOT EDIT              #
    #                                                                      #
    # The contents of this file are managed by Salt. Any changes to this   #
    # file may be overwritten automatically and without warning            #
    ########################################################################


% sudo salt minion1 slsutil.banner width=64 commentchar='//'
minion1:
    //############################################################//
    //                                                            //
    //         THIS FILE IS MANAGED BY SALT - DO NOT EDIT         //
    //                                                            //
    // The contents of this file are managed by Salt. Any changes //
    // to this file may be overwritten automatically and without  //
    // warning                                                    //
    //############################################################//

% sudo salt minion1 slsutil.renderer default_renderer=jinja \
  string="{{ salt['slsutil.banner'](width=32) }}"
minion1:
    ################################
    #                              #
    # THIS FILE IS MANAGED BY SALT #
    #        - DO NOT EDIT         #
    #                              #
    # The contents of this file    #
    # are managed by Salt. Any     #
    # changes to this file may be  #
    # overwritten automatically    #
    # and without warning          #
    ################################

% sudo salt minion1 slsutil.renderer default_renderer=jinja \
  string="{{ salt['slsutil.banner'](text=salt['http.query']\
  ('https://salt.tips/whats-new-in-salt-sodium/pirate-flag.txt')['body'], width=47) }}"
minion1:
    ###############################################
    #                                             #
    #  THIS FILE IS MANAGED BY SALT - DO NOT EDIT #
    #                                             #
    # ........................................... #
    # .                                         . #
    # .         ___                             . #
    # .         \_/                             . #
    # .          |._                            . #
    # .          |'."-._.-""--.-"-.__.-'/       . #
    # .          |  \       .-.        (        . #
    # .          |   |     (@.@)        )       . #
    # .          |   |   '=.|m|.='     /        . #
    # .     jgs  |  /    .='`"``=.    /         . #
    # .          |.'                 (          . #
    # .          |.-"-.__.-""-.__.-"-.)         . #
    # .          |                              . #
    # .          |                              . #
    # .          |                              . #
    # .                                         . #
    # ........................................... #
    ###############################################

PR #56761 by @amendlik

boolstr helper function

A small helper to simplify converting boolean values to strings (useful in templated configs):

% sudo salt-call slsutil.renderer default_renderer=jinja \
  string="{{ salt['slsutil.boolstr'](True, 'yes', 'no') }}"

local:
    yes

The last two arguments are optional ('true' and 'false' by default).

PR #56761 by @amendlik

baredoc module

The existing sys.doc function is unable to return modules and functions that do not have their dependencies installed. The new baredoc module can show all the execution and state module functions that exist. It returns a dictionary keyed by module and function name, where each entry contains a list of arguments that the function can take. The module is also able to show state/module docstrings.

% sudo salt minion1 baredoc.list_states saltutil

minion1:
    ----------
    saltutil:
        |_
          ----------
          sync_all:
              ----------
              kwargs:
                  kwargs
              name:
                  None
        |_
          ----------
          sync_beacons:
              ----------
              kwargs:
                  kwargs
              name:
                  None
...
sudo salt minion1 baredoc.list_modules test names_only=true
minion1:
    ----------
    test:
        - missing_func
        - attr_call
        - module_report
        - echo
        - ping
        - sleep
        - rand_sleep
        - version
...
% sudo salt minion1 baredoc.module_docs test.ping

minion1:
    ----------
    test.ping:
        Used to make sure the minion is up and responding. Not an ICMP ping.

        Returns ``True``.

        CLI Example:

            salt '*' test.ping

The module is already used by the VSCode SaltStack plugin to provide state autocompletion. So far it looks like the plugin has the most comprehensive state autocompletion feature amongst other editors:

VSCode SaltStack autocompletion

Screenshot by Christian McHugh

The original PR #52044 was submitted by C. R. Oldham . Later it was significantly refactored and improved by Christian McHugh by utilizing a more robust ast parser in PRs #56685 and #56902.

CAVEAT: the module doesn’t show function aliases that use salt.utils.functools.alias_function, for example docker.cp or saltutil.sync_outputters.

Other notable changes

See also: