What's new in Salt 3007 Chlorine STS

20 minute read Updated:

Salt Chlorine

This post is a bit unusual because it contains not only the new features that went into the Salt 3007 short term support release (master cluster, mTLS, WebSocket transport, salt-ssh changes, etc), but also the VMware acquisition news and its consequences. Plus, I allowed myself to delve into the problem-solution fitness theories and JTBD in particular. For a shorter summary, check out the official announcement.

What's new in Salt 3007 Chlorine: VMware acquisition, migration to Salt Extensions, Master cluster, TCP mTLS and WebSocket transports, Vault, Salt SSH

The most prolific community contributor to this release is jeanluc . He resolved many long-standing issues, implemented some features that were expected to work but didn’t, and even improved the tools for extension maintainers!

VMware acquisition by Broadcom

Well, it definitely made a splash… Here are a couple of headlines (including Reddit and Slack):

It is difficult to comment on this, other than with an old Russian saying:

Новая метла по-новому метёт (А new broom sweeps clean)

Unfortunately, the acquisition also had an effect on the Salt Project. The following folks from the core team were laid off:

This is a huge loss. Gareth, Megan, and Wayne Werner (who left VMware at the beginning of 2023 for reasons unknown to me) were some of the most responsive core developers on Salt Community Slack and have deep knowledge of Salt internals:

I hope they all found good new opportunities and impactful projects to work on.

These events led to several important policy changes (you can watch the Salt Project Community Open Hour 12-21-23 where Tom Hatch makes the announcements). More on that below.

New maintenance policies

I’m not going to cite the changes; just go and read them on your own time (the policies are quite radical, TBH):

Expedited migration to Salt Extensions

Salt Extensions have a long history:

Migration-related changes:

Outstanding problems:

I believe that from a Salt user point of view, this transition from a single tool to multiple building blocks can be a disaster, unless Broadcom executes it well enough. Do not get me wrong - I totally understand how hard and costly it is to maintain a cross-platform codebase like Salt. But the “user point of view” part is critical there. Users can be very fickle, even if they do not state that explicitly:

Customers are always beautifully, wonderfully dissatisfied, even when they report being happy and business is great. Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf. © Jeff Bezos

They will:

All of that erodes the Salt user base, reduces the number of community contributors, and, in some ways, may even affect the revenue from commercial products that utilize Salt under the hood.

So, to execute the transition well enough, Salt maintainers and the extension developer community need to care about user needs first, while being strategically smart about the efforts required to maintain the distributed codebase.

A long diversion into JTBD

I’m a huge fan of JTBD (Jobs to be Done) and ODI (Outcome Driven Innovation) theories, and I believe that they can be used to develop mitigations for many usability problems with Salt. Below are the main concepts:

  • The primary objective is to understand the “underlying process” (JTBD) the user is trying to execute when they are using a product or service
  • Every process has some steps that can be described in an almost generic way (Define, Locate, Prepare, Confirm, Execute, Monitor, Modify, Conclude). This is a job map for the core functional job - the stable, long-term focal point around which all other needs (related, emotional and consumption chain jobs, and financial outcomes) are defined
  • The consumption chain job is a journey a user goes through to receive, set-up, use, upgrade, clean and maintain a product. By exaggerating a bit, one can say that the whole SaaS model became so widespread because it eliminated many tedious parts of the software consumption chain job (download, install, upgrade, etc.)
  • Each step in a job map has many (usually tens) user needs or outcome statements, and the whole job map can have around 100-150 needs. It is possible to uncover an exhaustive list of needs.
  • Different user segments tend to care about different clusters of outcomes. By segmenting by outcomes, you can develop well-targeted solutions more efficiently (without bloat)
  • The job steps and user needs need to be written in a solution-free form to be stable over time (this allows different solutions to be invented as technologies evolve)
  • Besides the primary job executor (user), there are other roles that can exist - the purchase decision maker, the product life cycle support team, the product vendor, etc. They have their own jobs and needs

Because Salt is an extensible framework, it does many core functional jobs:

  • Execute commands remotely
  • Manage software components on a host
  • Manage laptops that are not always online (a quite common goal that is not well-supported by the existing Salt documentation and features)
  • Provision cloud environments
  • Orchestrate jobs across many hosts
  • React to events

…etc. It will take many months and a user research team to map them all.

Instead, I did a relatively limited exercise and partially mapped the Salt consumption chain job (i.e. the important outcomes for setting up, upgrading and maintaining a Salt minion). The goal is to uncover the most critical user needs, which can be used by the Salt developer community to brainstorm and implement some solutions that make the Salt user experience better while keeping the maintainer’s efforts within reasonable bounds.

Roles

  • Salt User
  • Salt Extension Developer
  • Salt Core Developer

These are the most basic ones, but obviously more roles exist - 3rd Party Salt Package Maintainer (for Debian, FreeBSD, etc), Commercial Software Vendor who relies on Salt (VMware/Broadcom, SUSE, etc), and probably others.

User needs

To come up with a list of user needs, I used the following (free) template by Michael Boysen (who worked with the ODI creator Tony Ulwick at Strategyn): JTBD Framework: 20 Contexts Related to Installing or Setting up a New Product.

The user needs are not exhaustive (I only spent 2-3 hours to construct them), aren’t grouped by job steps, and use a less formal structure. Still, these are a great starting points, and as an engineer, you can likely imagine different solutions that can satisfy them. One possible solution mapping process is the Opportunity Solution Tree by Teresa Torres.

  1. There should be an easy way to find an extension that can manage any particular “thing”, as well as the extension docs
  2. It should be clear how well an extension is maintained (stability, test coverage, maintainer responsiveness, code reviews, etc)
  3. If there are multiple extensions that can manage a “thing”, it should be easy to determine which is best (features, popularity, number of downloads, etc)
  4. It should be clear whether an extension supports a specific “thing to manage” version, OS/arch version, Salt version, and also works with the latest Salt development version
  5. The same for extension dependencies (a user does not want to spend time guessing the right dependency version combinations)
  6. An extension license should be clearly stated and compatible with the Salt one
  7. An extension should not destabilize/break the whole Salt process (prevent Salt from starting, produce memory leaks)
  8. It should be easy to install/upgrade/uninstall Salt minion with any extensions needed, including their dependencies
  9. The installation/upgrade process should be repeatable (extensions, including their transitive dependencies, must be pinned), otherwise there will be problems
  10. The installation/upgrade process should be unattended
  11. The time required to upgrade and the frequency of upgrades should be minimized
  12. The installation should not only be repeatable, but also cryptographically signed (dependency hashes)
  13. The installation process should support various network environments (proxy settings, offline mode)
  14. The migration path from a monolithic salt minion installation to the one with extensions should be straightforward and error-free (for any supported OS)
  15. Extension-specific configuration settings should follow some common convention or have established namespaces
  16. It should be easy to define extension settings before they are installed, so they start with the right settings
  17. Users would like to never deal with a scenario where two extensions do not work together (a transitive dependency conflict, config/module namespace/virtualname clash, etc)
  18. It should be easy to see what extensions and dependencies (including their versions) are installed, including a programmatic way to do that
  19. It should be easy to determine if there are security updates available for Salt or any extensions/dependencies
  20. Extensions should conform to upstream documentation guidelines, logging conventions, testing frameworks, deprecation helpers, OS/Salt support lifecycle
  21. Extensions should quickly adapt to changes, deprecations and improvements in the “things” managed by them or used by them (underlying platforms)
  22. Extensions should conform to the major/minor versioning scheme, to allow safe upgrades
  23. Users need to be able to test their states in an automated way (unit/integration tests) against new Salt versions and new extension versions
  24. It should be easy to understand where to file a specific issue a user has (whether it is a Salt failure or an extension failure)
  25. The extension error messages and logs should be clear

This list is not exhaustive - I didn’t mention Salt learning needs (there are lots of issues that contribute to Salt’s steep learning curve), state development needs (text editor support has not been improved for years, state linter needs some work, state troubleshooting is hard), etc. By the way, Salt issue tracker is an excellent source to uncover many hidden user needs (this is why it is not a good idea to aggressively close feature requests and stale bug reports).

I spent much less time synthesizing an Extension maintainer and a Salt core developer needs, but below are some of them to start with:

Extension maintainer needs

  1. The extension should be easy to maintain and update according to the best upstream Salt practices, helpers and automations
  2. Not everyone uses Github, some pre-made pipelines should exist for Gitlab as well
  3. When releasing an extension, a maintainer wants to notify its potential users
  4. It should be easy to keep up with Salt updates and make sure the extension stays compatible and stable
  5. An upstream framework/convention to define configuration settings for new extensions
  6. An extension should be easy to keep secure and update if necessary
  7. The documentation should be easy to maintain and publish
  8. CI runner expenses should be minimized, but the test coverage should be well maintained
  9. A community of extension maintainers (to agree on common stuff, ask for help, etc)
  10. Be able to influence upstream Salt development and have access to Salt core developers

Salt core developer needs

  1. Minimize the number of non-relevant issues to deal with
  2. Easily keep track of relevant issues for the official extensions and Salt itself (having a monorepo might be easier)
  3. Reduce the test feedback cycle length
  4. Easily coordinate Salt/Extensions release cycles if there are cross-cutting features that need to be released
  5. Fast and safe release procedures
  6. Try not to break existing extensions with new Salt releases
  7. Keep the common code (CI pipelines, dependencies) up to-date across repos
  8. Keep the projects free of security issues

References

If you want to discuss any of the above, ping me (@marnold) on Salt Community Slack.

Some notes on working with polyrepos

Tools to do polyrepo changes:

How big orgs approach polyrepos:

Master cluster

Over the years, there were a couple of requests to allow running multiple Salt Masters behind a load balancer (see #43368, #54297). Now it is possible!

The architecture is described in the master cluster SEP:

Salt Master Cluster architecture diagram

There are new master settings to configure the cluster:

A shared filesystem like Gluster is required. All of the masters in a cluster are assumed to be running on a reliable local area network (there is no partition tolerance). For more information, visit the Master Cluster page.

The feature was implemented by Daniel Wozniak in PRs #64705, #64936, #65840, #65138, #65228, and #65754

TCP mTLS and WebSocket transports

Quoting the feature author:

One benefit of WebSocket over TCP is the ability of smarter routing of requests with load balancers that support TLS termination and routing based on headers. For instance, you could have both request server and publish server behind a load balancer that is listening on 443.

PR #64937 by Daniel Wozniak

Vault

If you use Vault, please go read the pull request for many additional details (like this discussion).

PR #62684 by jeanluc

Salt SSH

There are several great improvements in salt-ssh:

Also, a long-standing issue with git pillar branches was fixed in Salt 3006.5 by the same author: #61861.

Removals

GPG

Saltcheck

Add an optional junit keyword to Saltcheck, to generate reports in JUnit format. GitLab should understand it out of the box; GitHub has a number of third-party actions for that.

PR #63464 by Meghann Cunningham

Cloud

Package managers

Windows

Small feature highlights

Other notable features

You can find other changes and bugfixes in the official CHANGELOG.md and Release Notes

P.S. It is unlikely that I’ll be able to track and cover these countless new extensions and their features from now on. It requires too many efforts to be sustainable. Something like Django Packages needs to be invented by the community to fill the gap. For my unofficial release notes series, I will see whether there are enough changes in Salt itself to continue publishing them, or if the official changelog will suffice.

See also: