Writing backward compatible Salt states

13 minute read Updated:

Retro hardware

pxhere.com

Have you seen the dreaded function "module.run" is using its deprecated version and will expire in version "Sodium" (now Phosphorus) warning message? While it is quite easy to enable the new more powerful module.run syntax for all minions, doing so also requires you to:

If you write Salt states (formulas) that are used by other people on the internet, or by internal teams with their own Salt masters, you may also face the following problems:

I this article you’ll learn a couple of patterns that will help you write backward compatible Salt states. These patterns can turn a Salt upgrade from a big event that requires a lot of intensive work into a non-stressful gradual process.

A couple of common and uncommon patterns to introspect Salt versions and features

Three guiding principles

1. Optimize for code deletion

Write deletable code. Since the software is always evolving, it is much more practical to use modern features by default and de-emphasize the old ones.

2. Let it crash

Let it crash. If it doesn’t crash by itself but should, inject a controlled failure.

3. Use nudges and reminders

Add nudges to remind yourself or others to refactor/delete the code.

Version checks

Salt minion version number is available in two forms:

% salt-call --local grains.item saltversion saltversioninfo

local:
    ----------
    saltversion:
        2019.2.0
    saltversioninfo:
        - 2019
        - 2
        - 0
        - 0

Version list comparison

It is hard to compare the saltversion grain because it is a string. Comparing the saltversioninfo grain with a list is much more straightforward:

{% if grains.saltversioninfo >= [2017, 7, 0, 0] %}
...
{% endif %}

CAVEAT 1: List length does matter:

  1. [3000] >= [3000] is True
  2. [3000] >= [3000, 0] is False
  3. [3000, 0] > [3000] is True

CAVEAT 2: If you stuck with Salt 3000 (insecure!), you’ll need an ugly workaround for the buggy saltversion: [3000, None, None, 0] because you can’t compare None with an integer. Try this instead (should work with Jinja 2.7 and higher):

{% if (grains.saltversion.split('.') | map('int') | list) >= [3000, 1] %}
...
{% endif %}
Conditional pillars

It is also possible to check the minion version grain inside of the pillars. Here is how to conditionally enable the new module.run syntax for minions that are new enough to support it (assuming that your states support both syntax variants):

# /srv/pillar/common.sls
{% if grains.saltversioninfo >= [2018] %}
use_superseded:
  - module.run
{% endif %}

Codename comparison

Since Salt 3000 Neon it is possible to compare codenames using the new salt_version module:

{% if salt['salt_version.equal']('Neon') %}
...
{% endif %}
{% if salt['salt_version.greater_than']('Neon') %}
...
{% endif %}
{% if salt['salt_version.less_than']('Sodium') %}
...
{% endif %}

If your Salt is older than 3000, you can easily backport the module. If you do not want to do that, you can use a fall-back method if the module is not available:

{% if (salt.get('salt_version.greater_than', None) and salt['salt_version.greater_than']('Fluorine'))
   or (grains.saltversioninfo > [2019, 2, 0, 0]) %}
...
{% endif %}

Package version comparison

It is intended to compare platform-specific package versions, but also can be used with Salt versions. The first function is available since 2015.8.10, but is not implemented for each package manager:

{% if salt["pkg.version_cmp"](grains.saltversion, "3001") %}
...
{% endif %}

It returns -1, 0, or 1 depending on the result.

The second function appeared in Salt 3001 and returns a boolean value:

{% if salt['pkg_resource.version_compare'](grains.saltversion, '>=', '3001') %}
...
{% endif %}

Again, if the module is not available, you can backport it or use a fall-back method.

Minion targeting via top.sls

Using grain-based targeting, you can apply different states and pillar values to different minion versions without polluting your sls files with Jinja conditionals. Just assign states to minions based on the saltversion grain:

# /srv/salt/top.sls
base:
  '*': []
  'saltversion:3000.3':
    - match: grain
    - example_state_3000_3
  'saltversion:3001':
    - match: grain
    - example_state_3001
Salt environments

If you prefer to keep the same state name for different minion versions, try Salt environments:

# /etc/salt/master.d/saltenvs.conf
file_roots:
  base:
    - /srv/salt
  v3000_3:
    - /srv/salt_3000_3
  v3001:
    - /srv/salt_3001
# /srv/salt/top.sls
base:
  '*': []
v3000_3:
  'saltversion:3000.3':
    - match: grain
    - example_state
v3001:
  'saltversion:3001':
    - match: grain
    - example_state

Do not forget to read how top files are compiled.

Master version

If you need to provide different pillar values depending on Salt master version, try using the test.version function inside of pillar files:

# /srv/pillar/master-version.sls
{% if (salt['test.version']().split('.') | map('int') | list) >= [3000, 1] %}
...
{% endif %}

Codename comparison will work too:

# /srv/pillar/master-version.sls
{% if salt['salt_version.equal']('Neon') %}
...
{% endif %}

Jinja and other dependencies

And finally, it is possible to compare a dependency version, for example, Jinja:

{% if (salt['test.versions_information']()['Dependency Versions']['Jinja2'].split('.') |
map('int') | list) < [2, 11] %}
...
{% endif %}

Please note, that it is not possible to conditionally use different Jinja filters (e.g., tojson vs json). See the jinja/#842 issue for an explanation. One way to solve this problem is to use different Salt environments for different minion versions.

Configuration introspection

This pattern should be used when a state depends on some config option. Here is how to check if the new module.run syntax is enabled:

{% if 'module.run' in salt['config.get']('use_superseded', []) %}
...
{% endif %}

Another example:

{% if salt['config.get']('use_yamlloader_old') %}
...
{% endif %}

It is possible to use opts.get() instead of salt['config.get'](). However, it won’t return options that are defined through pillar/grains, plus the default value is an empty string instead of None.

{% if opts.get('use_yamlloader_old') %}
...
{% endif %}

Module and function introspection

Execution modules/functions

Module loading in Salt depends on a couple of factors like OS and installed dependencies. It is quite easy to check if a function (e.g., test.ping) is loaded:

{% if salt.get('test.ping', None) %}
...
{% endif %}

This particular syntax works for both salt and salt-ssh.

It is also possible to check if a function signature has a specific argument:

{% if 'extra_args' in salt['sys.argspec']('pip.install')['pip.install']['args'] %}
...
{% endif %}

State modules/functions

State modules aren’t exposed in Jinja, so it is a bit harder to check if a state module or function exists:

{% if 'xml' in salt['sys.list_state_modules']() %}
...
{% endif %}
{% if 'xml.value_present' in salt['sys.list_state_functions']() %}
...
{% endif %}

It is also possible to check if a function signature has a specific argument:

{% if 'extra_args' in salt['sys.state_argspec']('pip.installed')['pip.installed']['args'] %}
...
{% endif %}

Other types of modules

There are many other functions that could be used to introspect different types of modules:

Here is a particularly devious example of how to check the feature that is only mentioned in the function docstring:

{% if salt['sys.state_doc']('file.symlink')['file.symlink'] |
regex_search('(Force will now remove all types of existing file system entries)') %}
...
{% endif %}

And the new baredoc module (available in Salt 3001 Sodium) can inspect modules and states that aren’t even loaded yet:

{% if salt['baredoc.list_modules']('dockermod') %}
...
{% endif %}
{% if salt['baredoc.list_modules']('dockermod')['docker'] |
traverse('logout') %}
...
{% endif %}
{% if 'registries' in (salt['baredoc.list_modules']('dockermod')['docker'] |
traverse('logout')).get('args') %}
...
{% endif %}
{% if 'Force will now remove all types of existing file system entries' in
salt['baredoc.state_docs']('file.symlink')['file.symlink'] %}
...
{% endif %}

CLI detection

Salt states can be applied through various means: salt, salt-call (or salt-call --local), salt-run, salt-ssh. The available features can vary depending on the CLI utility being used. Below are a couple of patterns to detect a Salt utility.

WARNING: Some of those checks aren’t very specific (e.g., the salt-call ones), and could clash with salt-ssh checks. When you use them in multiple if branches, place the more specific checks first.

salt MINION state.apply

{% if opts.get('__cli') == 'salt-minion' %}
...
{% endif %}

salt-call state.apply

{% if opts.get('__cli') == 'salt-call' %}
...
{% endif %}

salt-call --local state.apply (masterless mode)

{% if opts.get('__cli') == 'salt-call' and opts.get('file_client') == 'local' %}
...
{% endif %}

salt-run state.cmd state.apply (and also orchestrate states)

This is helpful if you apply a state to a Salt master via salt-run and need to change the state behavior.

{% if opts.get('__cli') == 'salt-run' and opts.get('file_client') == 'local' %}
...
{% endif %}

salt-ssh MINION state.apply

{% if 'salt-ssh' == opts | traverse('__master_opts__:__cli', '') %}
...
{% endif %}

The traverse Jinja filter was added in Salt 2018.3.3. For older versions use opts.get('__master_opts', {}).get('__cli').

SSH via enable_ssh_minions: true

{% if (opts.get('__cli') == 'salt-call') and ('salt-master' == opts | traverse('__master_opts__:__cli', '')) %}
...
{% endif %}

A more universal way to detect both salt-ssh and ssh minions is to use one of the following tests:

{% if opts.get('root_dir').endswith('/running_data') %}
...
{% endif %}
{% if (opts.get('__cli') == 'salt-call') and (opts | traverse('__master_opts__:__cli', '') in ('salt-ssh', 'salt-master')) %}
...
{% endif %}

The libsaltcli.jinja macro

If you need to repeat those checks in multiple states, you can try the Jinja macro from the saltstack-formulas GitHub community.

Injecting warnings

Often it is necessary to trigger conditional warnings from states, to remind yourself (or others) to take some action (upgrade Salt, rewrite a state, etc.). Below are two ways to do that.

Log-based warnings

The easiest way to emit a warning from any *.sls file (state or pillar) is to use Jinja:

{% if SOME_CONDITION %}
{% do salt.log.warning('Attention!') %}
{% endif %}

The resulting warning will be visible in the minion log file (if used in states), or in the master log file (if used in pillars).

CAVEAT: salt.log is not available before 2017.7.0

State warnings

The problem with log-based warnings is that you need to watch the logs in order to notice them. Fortunately, it is also possible to inject a warning into a state. Since Salt 3000 you can do this:

# warning.sls
{% if SOME_CONDITION %}
A warning:
  test.configurable_test_state:
    - result: true
    - changes: false
    - warnings: Attention!
{% endif %}

Here is how the warning looks like:

% sudo salt minion1 state.apply neon.warning

minion1:
----------
          ID: A warning
    Function: test.configurable_test_state
      Result: True
     Comment:
     Started: 06:08:37.709704
    Duration: 1.791 ms
     Changes:
    Warnings: Attention!

Summary for minion1
------------
Succeeded: 1
Failed:    0
Warnings:  1
------------
Total states run:     1
Total run time:   1.791 ms

It is very visible, and highlighted in red (although the blog syntax highlighter can’t display it).

If your Salt version is older than 3000, you can easily backport this feature, or just copy it into a custom module.

Injecting failures

Occasionally it is necessary to inject something more noticeable than a warning. There are two ways to do that.

Fail state

# failure.sls
{% if SOME_CONDITION %}
A failure:
  test.configurable_test_state:
    - result: false
    - changes: false
    - comment: You need to do something right now!
{% endif %}
% sudo salt minion1 state.apply failure

minion1:
----------
          ID: A failure
    Function: test.configurable_test_state
      Result: False
     Comment: You need to do something right now!
     Started: 06:07:16.292593
    Duration: 2.663 ms
     Changes:

Summary for minion1
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   2.663 ms
ERROR: Minions returned with non-zero exit code

If you also add - failhard: true option to the failure.sls state, then the remaining state execution process will be stopped.

One practical example is to fail hard on all minions that do not use the new module.run syntax:

{% if 'module.run' not in salt['config.get']('use_superseded', []) %}
module.run check:
  test.configurable_test_state:
    - result: false
    - changes: false
    - comment: Please enable the new module.run syntax in the minion config
    - failhard: true
{% endif %}

Jinja raise

{% if SOME_CONDITION %}
{{ raise('You need to do something right now!') }}
{% endif %}

Here is how it fails:

salt.exceptions.TemplateError: Custom Error

; line 2

---
{% if true %}
{{ raise(' You need to do something right now!') }}    <======================
{% endif %}

A couple of module.run examples

I’ve chosen this example because it is a common stumbling block for many Salt users. Here is how the older (default) syntax looks like:

Example of the obsolete module.run syntax:
  module.run:
    - name: test.echo
    - text: "Using the obsolete variant of module.run"

And this is the modern syntax (requires you to add use_superseded: [module.run] to the minion config, and yes, the setting name is confusing):

Example of the modern module.run syntax:
  module.run:
    - test.echo:
        - "Using the modern variant of module.run"

Here is a little bit of Git archaeology to show you how this feature has been evolving:

Over the years it generated a lot of confusion: #42148, #42272, #43130, #43770, #44425, #44882, #45360, #47033, #49851, #50348, #51349, #53504, #55551, #56422, #57919, #58705…

The current sentiment is to keep the status quo (#58705, #55551). My personal opinion is that the modern syntax is much better, and with enough willpower it is possible to eventually phase out the old syntax.

Below are possible steps to switch to the new syntax.

1. Enable the feature toggle

You can do this through pillar or via the minion config:

# /etc/salt/minion.d/superseded.conf
use_superseded:
  - module.run

For salt-ssh you need to add the following configuration snippet:

# /etc/salt/master.d/superseded.conf
ssh_minion_opts:
  use_superseded:
    - module.run

The same can be done on a per-minion basis in a roster file:

MINION1:
  host: interceptor.micros.local
  minion_opts:
    use_superseded:
      - module.run

2. Decide how to update your states

Using the config check
Example of backward compatible module.run:
  module.run:
{% if 'module.run' in salt['config.get']('use_superseded', []) %}
    - test.echo:
        - "Using the modern variant of module.run"
{% else %}
    - name: test.echo
    - text: "Using the obsolete variant of module.run"
{% endif %}
Using the top.sls condition
# /srv/salt/top.sls
base:
  '*': []
{% if 'module.run' in salt['config.get']('use_superseded', []) %}
    - state_that_uses_the_new_syntax
{% else %}
    - state_that_uses_the_old_syntax
{% endif %}
Using Salt environments

You can rewrite your states in a separate saltenv (a fileserver directory or git branch). Then assign the saltenv to minions using the saltversion grain or any other targeting criteria (including a custom grain).

3. Add some nudges

DRY with Jinja macro

Below is an example of how to reduce the boilerplate code using a Jinja macro:

{% macro superseded(state_id) -%}
{% set superseded = 'module.run' in salt['config.get']('use_superseded', []) -%}
{% set new_style, old_style = caller().split('# ---') -%}
{% if superseded -%}
{{ new_style }}
{% if not salt['salt_version.less_than']('Phosphorus') -%}
{{ state_id }} - nudge to remove the setting:
  test.configurable_test_state:
    - result: true
    - changes: false
    - warnings: 'Please remove the deprecated "use_superseded: [module.run]" config option'
{% endif -%}
{% else -%}
{{ old_style }}
{{ state_id }} - nudge to add the setting:
  test.configurable_test_state:
    - result: true
    - changes: false
    - warnings: 'Please enable the "use_superseded: [module.run]" config option'
{% endif -%}
{%- endmacro %}

Usage example:

{% from superseded_macro.jinja' import superseded -%}

backward-compatible-module-run:
  module.run:
{%- call superseded('backward-compatible-module-run') %}
    - test.echo:
      - "Using the modern variant of module.run"
# ---
    - name: test.echo
    - text: "Using the obsolete variant of module.run"
{% endcall %}

DRY with custom modules

Even more DRY option is to use a custom module, but this topic is out of the article’s scope. For example, folks at SUSE (who work on Uyuni) implemented a custom module.run replacement that handles the syntax differences automatically: uyni/states/mgrcompat.py.

A possible alternative

Things could be a lot simpler if the Minion Capabilities (#50617) RFC was implemented. The idea was to allow a module/function/docstring/parameter introspection inside of Jinja templates:

Unfortunately, the RFC author no longer contributes to Salt. In addition to that, the Salt release strategy has changed, and it is unlikely that this feature (if implemented) will be backported to older versions.

See also: