Patching Salt Modules

20 minute read Updated:

Salt modules

Let’s say you were bitten by a critical bug in Salt (2019.2.0 & 2019.2.1 were painful for many of us) that had an existing but unreleased fix. Or a cool feature that you badly need has landed in the old develop branch and is waiting forever to be migrated to the new master. Even if it was released, maybe you can’t yet upgrade Salt across the whole minion fleet. Or maybe you were wrongly expecting that your PR against the master branch would be included in the nearest release but were told that the upcoming release is feature frozen, and you have to wait another 4 months for the next one.

As a DevOps engineer, you need to get things done quickly. Fortunately, SaltStack has a built-in way to patch itself! After reading this guide, you’ll be able to patch almost any part of Salt code, even if you run multiple versions simultaneously. Some of the ways are hacky, but bugs are bugs, and you need to fix them in the most efficient way.

The methods described are mostly suitable for applying a couple of simple fixes. If you have a large or constantly evolving patchset, you’ll keep yourself sane by making your own fork and installing it via pip or custom packages.

As a DevOps engineer, you need to fix problems quickly. Fortunately, SaltStack has a built-in way to patch itself!

Contents

Determining affected minions

The first thing to do is understand what minion versions you have (you can discover that there are older minions you forgot to upgrade):

% sudo salt '*' test.version

minion3:
    2019.2.1
minion2:
    2019.2.1
minion1:
    2019.2.3

Another way to do it is the manage.versions runner (it includes the master version too):

% sudo salt-run manage.versions

Master:
    2019.2.3
Minion requires update:
    ----------
    minion2:
        2019.2.1
    minion3:
        2019.2.1
Up to date:
    ----------
    minion1:
        2019.2.3

If that is too verbose, use the survey runner:

% sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion2
      - minion3
  result:
      2019.2.1
|_
  ----------
  pool:
      - minion1
  result:
      2019.2.3

And if all you want is just version counters, use the following command:

% sudo salt '*' test.version --out json --static | jq '.[]' | sort | uniq -c
      2 "2019.2.1"
      1 "2019.2.3"

Now you need to understand which of these versions you need to target:

The general advice is to look at the PR contents and see if the changes are applicable to your installed Salt versions.

How to sync Salt modules

In the official docs, you can find the up-to-date list of syncable modules (contributed by Jamie Bliss in #50633). The majority of module types can be loaded (synced) from a fileserver, except the ones with a footnote.

The simplest way to sync them all across a minion fleet is to use the following command:

% sudo salt '*' saltutil.sync_all

Alternatively, you can use the individual saltutil.sync_* commands to sync different kinds of modules.

In Salt Neon, you can also use the new saltutil.sync_* states to do the same.

Some module types could be synced to a master:

% sudo salt-run saltutil.sync_all

Or individually via the saltutil.sync_* runner commands.

Also, custom modules are automatically synced whenever a highstate is performed.

Extracting upstream patches

The general advice is to avoid grabbing the latest modules from the upstream Git repo and putting them into your salt:// file tree. The only scenario when it’ll likely end up well is when a module doesn’t exist yet in your Salt version and doesn’t depend on any other modules. In all other cases, it is much safer to extract a specific patch and adapt it to apply cleanly to your specific Salt version.

Let’s say you want to backport the ability to pass arbitrary pip arguments through pip.installed from Neon to 2019.2.3 (PR #52327).

A simple way to extract the feature is to open the https://github.com/saltstack/salt/pull/52327.diff link, download the patch and remove unnecessary hunks/files that are related to tests and docs. You can see the resulting patch here: pip-extra-args-orig.diff

The patch won’t apply cleanly; however, if you inspect the salt/modules/pip.py.rej and salt/states/pip_state.py.rej, you’ll see that they contain non-functional changes and could be ignored. For more complex conflicts, you’ll need to spend some time to port the patch.

By comparing the original salt-2019.2.3 tree with the patched one (using the diff -Naur -x '*.orig' -x '*.rej' salt-2019.2.3{.orig,} command), you can get the patch that applies cleanly to 2019.2.3: pip-extra-args-2019-2-3.diff

Overriding a whole module

  1. Grab the patched modules. For the example above, that would be salt/modules/pip.py and salt/states/pip_state.py.
  2. Put them into the salt://_modules/ and salt://_states/ folders respectively.
  3. Sync the modules:
% sudo salt minion1 saltutil.sync_modules,saltutil.sync_states ,

minion1:
    ----------
    saltutil.sync_modules:
        - modules.pip
    saltutil.sync_states:
        - states.pip_state

Viola! Now you can use the extra_args option:

$ sudo salt minion1 sys.state_doc pip.installed | grep -A 8 extra_args

            extra_args
                pip keyword and positional arguments not yet implemented in salt

                    pandas:
                      pip.installed:
                        - name: pandas
                        - extra_args:
                          - --latest-pip-kwarg: param
                          - --latest-pip-arg

                Warning:

                    If unsupported options are passed here that are not supported in a
                    minion's version of pip, a `No such option error` will be thrown.

Below is an example of how to use this new option to install a package into the /root/.local directory (instead of /usr/local):

install_poetry:
  pip.installed:
    - name: poetry
    - extra_args:
      - --user

Adding a new function

Sometimes you want to extend a module with just one new function, but do not want to override the whole module because you are afraid of possible future version conflicts. Let’s say you want to add the delete function to sdb.sqlite3 module (PR #51479).

The original module is named sqlite3.py. Custom one should have a unique file name that contains the original one (see #50812 and #52521 to understand why). Let’s name it as sqlite3_custom.py:

# The imports below are copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__() return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount

Put the following configuration snippet to /etc/salt/minion.d/mysqlite.conf on a minion and restart it:

mysqlite:
  driver: sqlite3
  database: /tmp/sdb.sqlite
  table: sdb
  create_table: True

Then put the above sqlite3_custom.py module into the salt://_sdb/ directory and sync it:

% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom

Let’s try it:

% sudo salt minion1 sdb.set sdb://mysqlite/password 12345678

minion1:
    True
% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    12345678
% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 58, in delete
        return salt.utils.sdb.sdb_delete(uri, __opts__, __utils__)
      File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 108, in sdb_delete
        return loaded_db[fun](query, profile=profile)
      File "/var/cache/salt/minion/extmods/sdb/sqlite3_custom.py", line 28, in delete
        conn, cur, table = _connect(profile)
    NameError: name '_connect' is not defined
ERROR: Minions returned with non-zero exit code

Oops! Our custom function needs to have access to _connect, any related imports, private functions, and module-level variables. Salt loader only exposes public functions (without leading underscores), so any public function calls from the same module should be changed to __salt__['sqlite3.function']() syntax.

Let’s fix this specific example to make it work:

# The import below is copied from the original module
from __future__ import absolute_import, print_function, unicode_literals

# This private function is imported from the original module
from salt.sdb.sqlite3 import _connect

# The import below is copied from the original module
try:
    import sqlite3
    HAS_SQLITE3 = True
except ImportError:
    HAS_SQLITE3 = False

# The original module doesn't define a virtualname. However, it is absolutely
# critical to define it in a custom module, otherwise the loader
# won't be able to find our delete function in the same sqlite3 namespace
# (this will happen when the __virtual__() return value is not a string)
__virtualname__ = 'sqlite3'

# This function is also copied from the original module
def __virtual__():
    '''
    Only load if sqlite3 is available.
    '''
    if not HAS_SQLITE3:
        return False
    return True

# This is the new function that we want to add
# See https://github.com/saltstack/salt/pull/51479
def delete(key, profile=None):
    '''
    Delete a key/value pair from sqlite3
    '''
    if not profile:
        return None
    conn, cur, table = _connect(profile)
    q = profile.get('delete_query', ('DELETE FROM {0} WHERE '
                                     'key=:key'.format(table)))
    res = cur.execute(q, {'key': key})
    conn.commit()
    return cur.rowcount

Let’s try it again:

% sudo salt minion1 saltutil.sync_sdb

minion1:
    - sdb.sqlite3_custom
% sudo salt minion1 sdb.delete sdb://mysqlite/password

minion1:
    1
% sudo salt minion1 sdb.get sdb://mysqlite/password

minion1:
    None

It works! Please note, that because the overlay module is named differently than the original one, all that initialization code in the upstream module is still triggered. Also, direct imports could potentially have unintended side effects.

CAVEAT: your custom delete function will be silently ignored (even with a version-aware guard) if the upstream sqlite3.py module gets its own delete function (i.e., after a Salt upgrade).

Version-aware guard

It would be nice if a patched module just stopped working with any Salt version that doesn’t match the target one. Then any upgrades would lead to a loud failure, and you won’t forget to remove or update the patched module. Let’s implement exactly that!

For example, there is an issue #54755 (pip failures even when not using pip) that badly affected Salt 2019.2.1. The fix is located in #54826; however, it can’t be applied cleanly on top of 2019.2.1 because the module was changed multiple times between the releases. Fortunately, all these changes are relevant, so we can just grab the state/pip_state.py module from 2019.2.2 or 2019.2.3. You can inspect the changes using the following command on two extracted release tarballs:

% diff -u salt-2019.2.{1,3}/salt/states/pip_state.py

...
DIFF HERE
...

Put the pip_state.py module from 2019.2.3 into the salt://_states/ directory and sync it:

% sudo salt minion1 saltutil.sync_states

minion1:
    - states.pip_state

After that, the state that triggered the error will run just fine:

# pip_test_state.sls
/tmp/file.txt:
  file.managed:
    - contents: |
        Content
    - unless:
      - 'grep "Content" /tmp/file.txt'
% sudo salt minion2 state.apply pip_test_state

minion1:
----------
          ID: /tmp/file.txt
    Function: file.managed
      Result: True
     Comment: unless condition is true
     Started: 06:47:19.338294
    Duration: 4058.666 ms
     Changes:

Summary for minion1
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   4.059 s

Now comes the most important part. We need to add a version-aware guard to the module, so it will only work on 2019.2.1. There are multiple ways to check Salt version:

  1. Compare __grains__['saltversion'] with a string like '2019.2.1'
  2. Compare __grains__['saltversioninfo'] with a list like [2019, 2, 1, 0]
  3. Compare the salt.utils.versions.LooseVersion instances
  4. Use the salt.utils.versions.warn_until helper
  5. Compare salt.version.__saltstack_version__ with an instance of salt.version.SaltStackVersion
  6. Compare release codenames using the new salt_version execution module in Salt Neon
  7. Use the new salt.utils.versions.warn_until_date helper in Salt Neon

Option (5) seems to be the most robust (it should work with release candidates and git snapshots too, and also support the new versioning scheme).

We need to place the version comparison in a module function, which is triggered as early as possible. For the majority of modules, that would be __virtual__() or __init__(opts). The added lines are marked with --- VERSION CHECK ---:

def __virtual__():
    '''
    Only load if the pip module is available in __salt__
    '''
    # --- VERSION CHECK ---
    from salt.version import SaltStackVersion, __saltstack_version__ as sv
    ver = SaltStackVersion(2019, 2, 1, 0, u'', 0, 0, None)
    if sv != ver:
        msg = 'The %s module is not compatible with %s' % (__name__, ver)
        log.critical(msg)
        raise Exception(msg)
    # --- VERSION CHECK END ---

    if HAS_PKG_RESOURCES is False:
        return False, 'The pkg_resources python library is not installed'
    if 'pip.list' in __salt__:
        return __virtualname__
    return False

Now, if you apply the pip.installed state to 2019.2.3 minion, it will fail because the module is not loaded:

% sudo salt minion1 state.single pip.installed ipython

minion1:
    Data failed to compile:
----------
    Specified state 'pip.installed' was not found

Another option is to place this check into the installed(...) function (right after the docstring and before the function body). This is more risky, because the module will be loaded and some parts could be executed before the check. On the positive side, the failure is more visible:

% sudo salt minion1 state.single pip.installed ipython

minion1:
----------
          ID: ipython
    Function: pip.installed
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 1933, in call
                  **cdata['kwargs'])
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "/var/cache/salt/minion/extmods/states/pip_state.py", line 675, in installed
                  raise Exception(msg)
              Exception: The salt.loaded.ext.states.pip_state module is not compatible with 2019.2.1
     Started: 07:49:24.007851
    Duration: 4.435 ms
     Changes:

Summary for minion1
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   4.435 ms

A couple of essential details to summarize:

Version-aware override

So, we got a little safety guard that helps us to detect when our custom modules are outdated. But the rabbit hole is much deeper because we can run multiple Salt versions at the same time and may need to have different versions of custom modules.

The solution is to use Salt environments. First, set up the file_roots master setting:

file_roots:
  base:
    - /srv/salt
  v20190201:
    - /srv/salt_20190201

Second, make sure that any custom modules for 2019.2.1 are placed into the corresponding /srv/salt_20190201/ subdirs (_modules, _states, _grains, etc.) and not into the base environment /srv/salt dir:

% tree /srv
├── pillar
│   └── top.sls
├── salt
│   ├── top.sls
├── salt_20190201
│   └── _modules
│       └── test.py

There are two ways to pin minions to specific saltenvs:

  1. Using the top file
  2. Using the saltenv minion setting

Unfortunately, the second approach doesn’t affect how the saltutil.sync_* functions work (no modules are synced). And the explicit saltutil.sync_all saltenv=v20190201 will sync the modules to all minions, ignoring the saltenv setting. So, the only option left is to specify saltenvs in the top file:

base:
  '*': []
v20190201:
  'saltversion:2019.2.1':
    - match: grain

Now the saltutil.sync_modules function will work as intended and sync our custom test.py only to minion2:

% salt '*' test.version
minion1:
    2019.2.3
minion2:
    2019.2.1

% sudo salt '*' saltutil.sync_modules
minion1:
minion2:
    - modules.test

By adding more environments, you can distribute different custom modules to different minions using any targeting criteria (including the compound matchers).

Two caveats

CAVEAT 1: custom modules are cached and won’t be automatically deleted when you update Salt (at least for Ubuntu packages; other distros may contain uninstaller scripts that clear these caches). To mitigate that, run saltutil.clear_cache before the update (and maybe saltutil.refresh_* if necessary).

CAVEAT 2: Fileserver environments won’t be usable for proxymodules until the #56222 is merged. However, you can fix this problem by using the self-patching method and then proceed as usual.

If you are paranoid, you can use environments in combination with version-aware guards to prevent modules from being run on updated minions or if you accidentally mess up Salt environments.

Non-syncable modules

Some module types do not have a corresponding saltutil.sync_* command. For example, before Salt Neon, it was impossible to sync custom executors.

Below is a customized direct_call.py executor that logs the function name:

# -*- coding: utf-8 -*-
# direct_call.py
from __future__ import absolute_import, print_function, unicode_literals
import logging

log = logging.getLogger(__name__)

def execute(opts, data, func, args, kwargs):
    log.info("Executing the %s function", func)
    return func(*args, **kwargs)

There are three ways to distribute this custom executor module.

1. Place the module into a cache dir

The cache dir is defined by the extension_modules config option (/var/cache/salt/minion/extmods, there is no need to change it).

# override_executor.sls
/var/cache/salt/minion/extmods/executors:
  file.directory:
    - makedirs: True

/var/cache/salt/minion/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /var/cache/salt/minion/extmods/executors

Run sudo salt minion1 state.apply override_executor to apply the state.

The biggest downside is that the cache directory is volatile and may be cleaned (e.g., if you run saltutil.clear_cache, saltutil.sync_all, state.highstate, or state.apply commands).

2. Module search path

Salt has multiple settings to control the search path for each module type (extension_modules and various *_dirs options). First, you need to configure the minion. Place the following snippet into /etc/salt/minion.d/extmods.conf and restart the minion (unfortunately, pillar-based minion configuration doesn’t work in this case):

# /etc/salt/minion.d/extmods.conf
executor_dirs:
  - /srv/extmods/executors

Then run sudo salt minion1 state.apply override_executor to apply the following state:

# override_executor.sls
/srv/extmods/executors:
  file.directory:
    - makedirs: True

/srv/extmods/executors/direct_call.py:
  file.managed:
    - source: salt://_executors/direct_call.py
    - require:
        - file: /srv/extmods/executors

3. Setuptools entry points

This is the most complex method; however, it allows installing arbitrary sets of custom modules along with their dependencies. I started writing some examples for this blog post but then abandoned the idea for the sake of brevity. If you are interested in how to do that, visit the new Salt Extensions and the older Packaging Modules documentation pages. Also check out the Salt Extensions section from my Salt 3003 Release Notes

Do not forget to add a version-aware guard whenever you override a built-in module.

The dreaded utils namespace

Utils are a special breed of modules (despite being syncable through saltutil.sync_utils). Sometimes they are used through the loader (the __utils__ dunder), but most of the time they are imported directly:

% find salt/ -name '*.py' -exec grep -l '__utils__\[' \{\} \; | sort -u | wc -l
     159

% find salt/ -name '*.py' -exec grep -l ' salt\.utils' \{\} \; | sort -u | wc -l
    1101

There are a couple of problems with utils:

This explains the long list of issues that people have when they try to override utils: #32500, #46841, #46911, #51719, #51958, #52136, #52222, #53666, #55232…

There are two attempts to partially solve this use-case (#52001 and #53167 by Alberto Planas ), ald the last one has been released.

I was able to find two ways to deal with this mess when a patch touches both regular modules and utils:

  1. Copy all the changed functions (and functions that use these functions!) from the utils patch into the modules that use it, and change all the calls accordingly
  2. Alternatively, put the patched utils module into the _utils dir, then change all the calls to use the __utils__ notation

With both methods, you’ll end up running two versions of functions from the utils module (the old one is imported as usual, and the newer one is inlined or called from the overlay module).

Self-patching

The trick is described by DatuX in #51932. For example, let’s fix the horrible typo that prevented Salt 2019.2.1 from starting when the ping_interval option was set (see #54777). The patch is simple:

diff --git a/salt/minion.py b/salt/minion.py
index d1c0c00751..28b4956a49 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -2730,7 +2730,7 @@ class Minion(MinionBase):
                     self._fire_master('ping', 'minion_ping', sync=False, timeout_handler=ping_timeout_handler)
                 except Exception:
                     log.warning('Attempt to ping master failed.', exc_on_loglevel=logging.DEBUG)
-            self.remove_periodic_callbback('ping', ping_master)
+            self.remove_periodic_callback('ping')
             self.add_periodic_callback('ping', ping_master, ping_interval)

         # add handler to subscriber

Put the patch into the salt://patches/files/2019.2.1-periodic-callback.diff file, then create the following state and put it into salt://patches/periodic_callback.sls:

{% if grains.saltversion == "2019.2.1" %}

# Just in case there is no patch command installed
patch:
  pkg.installed

# We use directory name here (saltpath) to apply patches that touch multiple files
periodic_callback_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/2019.2.1-periodic-callback.diff
    - strip: 2
    - require:
        - pkg: patch

# Set the ping_interval option after the patch is applied
set_ping_interval:
  file.managed:
    - name: '{{ salt["file.join"](salt["config.get"]("config_dir"), "minion.d", "ping_interval.conf") }}'
    - contents: |
        ping_interval: 60
    - onchanges:
        - periodic_callback_patch

# Restart salt minion after the patch is applied and the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltproject.io/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: set_ping_interval

{%- endif %}

The state will apply the patch, set the required config option, and restart the minion. However, there are a couple of things that could be improved:

Bootstrapping patched minions

All these patching methods are super useful, but it is also essential to know how to apply them to new minions. Earlier I described a few commands that you can use to sync the modules manually. Now it is time to automate that.

There are many ways to bootstrap Salt minions: salt-bootstrap, salt-ssh, different cloud drivers (including the saltify one), salt-run manage.bootstrap, and countless custom scripts. The most important thing is to apply any overrides/patches to minions as early as possible (before any other states are run). There are multiple ways to do so:

  1. You do not need to do anything when you apply a highstate. With the autoload_dynamic_modules minion config option (True by default), Salt runs saltutil.sync_all automatically before applying the highstate.
  2. If for some reason, you disabled the above option, you can place a state that syncs modules at the beginning of the top.sls file, so it will be applied first
  3. Configure the startup_states minion setting
  4. Add a master reactor config for the salt/minion/*/start event
  5. Hook into the salt-cloud bootstrap process
  6. Write a custom bootstrap script

If you need to restart a minion once for the patch to take effect, you can combine methods (3) or (4) with the restart technique. Methods (5) and (6) allow you to patch minions before they are started.

Useful commands

A couple of commands to troubleshoot custom modules:

  1. salt-call together with print or logging statements (or with any interactive Python debugger like ipdb)
  2. saltutil.clear_cache to clear the extmods directory
  3. cp.list_master prefix=_modules/ (or any other directory) to see which custom modules are available on the fileserver
  4. saltutil.list_extmods to see the list of synced modules
  5. sys.list_* to list different kinds of modules

Future developments

If you need more help

Hopefully, this guide has enough details to help you apply any patches you want. However, if you need more help to do that, I’m available for part-time consulting.

Unexplored areas

There are a couple of rabbit holes I wanted to explore but had no time to do so:

This is not a commitment, but I hope to cover these topics in the future (if this post turns out to be useful).

See also: