What's new in Salt 3003 Aluminium release

18 minute read Updated:

Salt Aluminium

The Salt Aluminium release is rather modest with new features. There are multiple possible reasons for that: post-acquisition turbulence/reorganization under VMware, rebranding and the new saltproject.io domain, a couple of severe CVEs, etc. In fact, at the end of 2020, the list of notable changes was so small that I almost decided to skip writing this post altogether.

Fortunately, a couple of interesting features landed in the following weeks, plus some noteworthy changes happened under the hood as well. So, please enjoy the next installment of these unofficial Salt release notes!

What's New in Salt 3003 Aluminium: Beacons, Cloud, Development, Salt Extensions, Performance and caching, Juniper minion, FIPS mode, FreeBSD and macOS, Rich Slack messages

Docs

Salt User Guide

It is a new Salt User Guide that was created during February 10 Docs Jam. It covers lots of topics, but there are a couple of them that caught my eye:

Go read it, it is awesome!

The guide is temporarily hosted on GitLab Pages, and its source code can be found in the corresponding repository. For more background about the Docs Jam event you can read the blog post by Alyssa Rock (the event organizer).

Other improvements

If you know some obscure and undocumented config options, please add them to this epic.

Development

runtests removal 🎉

It is a huge milestone! I already covered the migration progress in the past (1, 2). Now the runtests.py test runner was finally removed, and lots of tests (but not all) were moved to PyTest.

The new Contributing guide has all the details on how to run the tests, plus I have my own cheatsheet with a couple of useful commands.

PR #59377 by Pedro Algarvio , also many PRs by Kirill Ponomarev and other developers.

Backward compatibility tests

This is a new test suite to test backward compatibility between newer masters and older minions. I haven’t tried to actually run it, but this what I learned by looking at the code:

  1. It runs tests in dynamically generated Docker containers, using the saltstack/ci-centos-7 base image.
  2. The minion test matrix includes 2017.7.8, 2018.3.5, 2019.2.4, 3000.6 with Py2/Py3, and 3001.4, 3002.2 with Py3.
  3. The test suite is fairly minimal (for now) and checks the key acceptance procedure, test.ping, state.highstate (that installs one package and calls a custom module), and a salt-cp utility test. I’m pretty sure that the test suite will be significantly expanded in the future.

PR #59548 by Pedro Algarvio

Hooks and linters

Stricter object lifecycle control

It seems to be a part of ongoing efforts to eradicate memory leaks.

Loader contexts

Prior to these changes, Salt’s “magic dunders” (__utils__, __salt__, __opts__, etc…) would be added as global attributes to a loaded module’s namespace. It had the unintended side effect of allowing a method call from one loader to change something in another loader. This can be particularly problematic for __context__ and __opts__, plus it can also cause problems for loaders added with unique configs.

Now loaders use the contextvars library to provide a context that is specific to the loader that loads the function being called. Every function is wrapped with the new LoadedFunc class.

There is also an addition of a new pack_self attribute to Salt’s LazyLoader class. It allows the loader to expose itself as a magic dunder without creating circular references in the loader’s pack dictionary. Those circular references are the cause of some memory leaks. The pack_self argument should be used to avoid memory leaks.

I haven’t run any benchmarks, but I expect these changes to make the loader somewhat slower. There is just too much magic inside… On a positive note, it was reported that this change fixes the long-standing reactor race conditions: #54045, #57626. It looks like the fixes shipped in Salt Magnesium weren’t sufficient for that.

PR #58853 by Daniel Wozniak

Other changes

Salt Extensions

For many years it was possible to install custom Salt modules via pip. In Salt Aluminium this feature was simplified a bit and gained some additional tooling:

The PR itself is fairly simple. Previously it was necessary to specify loader functions to define custom modules using setuptools entry-points in setup.cfg:

[options.entry_points]
salt.loader=
  runner_dirs = my_package.loader:func_to_get_list_of_runner_dirs
  module_dirs = my_package.loader:func_to_get_list_of_module_dirs

Now they can be defined in a more succinct way as well:

[options.entry_points]
salt.loader=
   namedoesnotmatter = my_package

With this new approach, when the target is a package, Salt looks for the ext_type being loaded as submodule of the package, and if it exists, it loads from there. If the target is a callable, then that callable should return a dictionary where the keys are the several salt loader names. For salt execution modules, it would be modules; for salt state modules, it would be states, etc. The values for those keys need to be iterable (i.e., a list, a generator, etc.), whose values are the paths from where to load the salt loader modules. If a function returns a list, then this is an old-style entry-point, and its name must match the value of ext_type.

Salt’s versions report salt --versions-report now includes all installed extensions as well. It may be necessary to restart Salt to activate an installed extension.

PR #58943 by Pedro Algarvio . For more information you can watch this presentation.

P.S. I’m aware of several open source projects that use entrypoints to ship custom Salt modules:

Cloud

Hetzner Cloud provider

The Hetzner Public Cloud provider for salt-cloud has been added in Salt Aluminium. Internally it uses the official hcloud-python library. The list of initially supported functions:

By the way, the spreadsheet I made shows which functions are implemented in various Salt cloud drivers and is somewhat useful for feature parity tracking. If you want to implement a new cloud provider or enhance an existing one, here is a good guide that contains a minimal set of functions the provider needs to have: Writing Cloud Driver Modules. Alternatively, you might want to look at Idem.

PR #59301 by @fkantelberg

Libvirt performance tuning options

It is a GSoC 2020 project that makes lots of existing libvirt tunables configurable through Salt (I won’t even try to list them all!).

PR #58196 by Guoqing Li , #59007 by Cedric Bosdonnat

Enhanced libvirt network options

It is a really huge PR that adds tons of new options to virt.network_defined and virt.network_running states.

Execution modules:

State modules:

Utils:

PR #59146 by Cedric Bosdonnat . Hat tip to Wayne Werner for reviewing this monster!

Other cloud changes

Juniper native minion

Juniper native minion support was merged into the master branch. Previously, it was available in private branches for Neon, Sodium, and Magnesium releases, which were tested and certified by Juniper. Now it is a part of the mainline release process and will be tested using regular CI/CD runs. A brief list of new features and improvements:

PRs #58979 and #59114 by David Murphy . More documentation is available on juniper.net.

Beacons

A mod_beacon state function

It is an initial implementation of SEP-30 - Event Driven State Enforcer. It can automatically add associated beacons to monitor changes to the pkg, service, and file resources in the state file and fire events to the event bus when changes occur.

# aluminium/modbeacon.sls
/tmp/hello.txt:
  file.managed:
    - contents: Hello there!
    - beacon: true
    - beacon_data:
        coalesce: false
        interval: 60
        mask:
          - create
          - delete
          - modify

/tmp/hello_dir:
  file.directory:
    - beacon: true
    - beacon_data:
        auto_add: true
        recurse: true
        exclude: []
        coalesce: false
        interval: 60
        mask:
          - create
          - delete
          - modify

zsh:
  pkg.installed:  # also works with pkg.removed
    - beacon: true
    # beacon_data keyword is not supported here

cron:
  service.running: # also works with service.dead
    - beacon: true
    - beacon_data:
        onchangeonly: true
        interval: 60
        delay: 0
        emitatstartup: false
        uncleanshutdown: /run/crond.pid

The downside is that all these automatically created beacons will be lost when the minion is restarted, so you need to ensure this state is run again after the restart.

PR #59559 by Gareth J. Greenaway

The pkg beacon improvements

Improve the pkg beacon to fire events not only when there are upgrades to packages, but also when watched packages are installed or removed.

beacons:
  pkg:
    - pkgs:
        - zsh
    - refresh: True
salt/beacon/master/pkg/	{
    "_stamp": "2021-03-13T12:05:01.648082",
    "id": "master",
    "pkg": "zsh",
    "status": "installed",
    "version": "5.8-3ubuntu1"
}
salt/beacon/master/pkg/	{
    "_stamp": "2021-03-13T12:05:20.867170",
    "id": "master",
    "pkg": "zsh",
    "status": "not-installed",
    "version": null
}

PR #59463 by Gareth J. Greenaway

The swapusage beacon

The new swapusage beacon complements the existing memusage beacon (and depends on the psutil library as well).

beacons:
  swapusage:
    - percent: 10%
    - interval: 60

Once the threshold is exceeded, the beacon will emit a salt/beacon/*/swapusage/ event:

salt/beacon/master/swapusage/	{
    "_stamp": "2021-03-13T10:54:38.640698",
    "id": "master",
    "swapusage": 16.8
}

PR #59460 by Imran Iqbal

The salt_monitor beacon

A beacon to execute salt execution module functions. This beacon will fire only if the return data is “truthy”. The function return, function name, and args and/or kwargs, will be passed as data in the event. The configuration can accept a list of salt functions to execute every interval.

beacons:
  salt_monitor:
    - salt_fun:
      - slsutil.renderer:
          args:
            - salt://aluminium/parallel.sls
          kwargs:
            - default_renderer: jinja
      - test.ping
    - interval: 3600

The resulting event will look like this:

salt/beacon/master/salt_monitor/	{
    "_stamp": "2021-03-13T11:33:43.558114",
    "args": [
        "salt://aluminium/parallel.sls"
    ],
    "id": "master",
    "kwargs": {
        "default_renderer": "jinja"
    },
    "ret": "barrier:\n  test.nop\nblah-1:\n  cmd.run:\n    - name: sleep 10\n    - require:\n      - barrier\n    - parallel: true\n    - require_in:\n      - barrier2\n\nblah-2:\n  cmd.run:\n    - name: sleep 10\n    - require:\n      - barrier\n    - parallel: true\n    - require_in:\n      - barrier2\n\nblah-3:\n  cmd.run:\n    - name: sleep 10\n    - require:\n      - barrier\n    - parallel: true\n    - require_in:\n      - barrier2\n\n\nbarrier2:\n  test.nop\n",
    "salt_fun": "slsutil.renderer"
}

PR #56575 by @jtraub91

Performance and caching

Parallel requisites

States that use parallel: true and have requisites now run in parallel as they should:

# aluminium/parallel.sls
barrier:
  test.nop

{%- for x in [1,2,3] %}
blah-{{x}}:
  cmd.run:
    - name: sleep 10
    - require:
      - barrier
    - parallel: true
    - require_in:
      - barrier2
{% endfor %}

barrier2:
  test.nop

As you can see, the actual time to run is ~11s instead of 30s (although the built-in total time reporting doesn’t make a lot of sense anymore):

time salt-call state.apply aluminium.parallel --state-output terse

local:
  Name: barrier - Function: test.nop - Result: Clean Started: - 18:45:07.663767 Duration: 0.544 ms
  Name: sleep 10 - Function: cmd.run - Result: Changed Started: - 18:45:07.665067 Duration: 10027.307 ms
  Name: sleep 10 - Function: cmd.run - Result: Changed Started: - 18:45:07.669409 Duration: 10046.531 ms
  Name: sleep 10 - Function: cmd.run - Result: Changed Started: - 18:45:07.673804 Duration: 10035.852 ms
  Name: barrier2 - Function: test.nop - Result: Clean Started: - 18:45:17.740429 Duration: 1.334 ms

Summary for local
------------
Succeeded: 5 (changed=3)
Failed:    0
------------
Total states run:     5
Total run time:  30.112 s

real    0m11.486s
user    0m1.071s
sys     0m0.206s

PR #58976 by Gareth J. Greenaway

That said, it looks like there is still some room for improvement: #59959.

Clear and show the pillar cache

This is a much-needed feature for Salt environments with lots of pillar data and enabled pillar caching (pillar_cache: true). It allows to inspect and purge pillar cache for any specific minion (or multiple ones).

salt-run pillar.show_pillar_cache '*'
salt-run pillar.clear_pillar_cache 'minion'

The pillarenv and saltenv keywords are also supported.

PR #59411 by Gareth J. Greenaway

Other performance changes

TCP transport

FIPS mode

This change adds a new master/minion option named fips_mode (False by default). When enabled, it will disable non-fips compliant crypto libraries such as libnacl. For example, FIPS is enabled on VMware PhotonOS 3 image on CI.

List of affected modules:

PR #59833 by Daniel Wozniak

FreeBSD

macOS

Other OSes

Rich Slack messages

SadServer quote in Slack

This feature adds two optional parameters to slack.post_message execution function:

To understand how to format Slack messages using these arguments, read the Composing messages documentation page. The corresponding slack.post_message state does not support these new parameters yet; hopefully, it will be addressed later.

sadserver_quote:
  # NOTE: Put use_superseded: ['module.run'] into the minion config
  module.run:
    - slack.post_message:
      - "#random"
      - " "
      - Salt
      - api_key: xoxb-123456789012-0123456789012-ABcdEFghIJklMNopQRstUVwx
      - icon: https://saltproject.io/favicon.png
      - blocks: >-
          [
          {"type": "section", "text": {"type": "mrkdwn", "text": "Your daily *Sad Server* quote:"}},
          {"type": "divider"},
          {"type": "section", "text": {
           "type": "plain_text",
           "text": {{ salt['cmd.run']('python3 -c \'import json, random, sys, urllib.request; sys.stdout.buffer.write(("{}".format(random.choice(json.loads(urllib.request.urlopen("https://brokenco.de/files/sadserver.json").read().decode("utf-8")))["full_text"].strip())).encode("utf-8"))\'') | yaml_encode }}
           },
           "accessory": {"type": "image", "image_url": "https://pbs.twimg.com/profile_images/581161893219323904/eGnWc30X_400x400.png", "alt_text": "Sad Server"}
          }
          ]

PR #59428 by Ryan Addessi

P.S. Apparently, this feature will only work with Slack legacy tokens, which are deprecated and no longer can be created. And to be honest, the overall integration with Slack needs some refactoring - there are multiple modules that use different ways to communicate with Slack: modules.slack_notify, states.slack, engines.slack, returners.slack_return, returners.slack_webhook_return, and utils.slack. Also, on Feb 24th, the Slack engine stopped working due to deprecation of the API

Other notable changes

You can find other changes and bugfixes in the official CHANGELOG.md.

See also: