Salt is buggy. There are many threads (1, 2, 3, 4, …) about that. If you dig deep enough into any Salt feature, you are almost guaranteed to find bugs. This results in de-facto issue bankruptcy, plus it looks like Salt open source team is severely understaffed, and many talented folks recently left the company (Daniel Wallace, Nicole Thomas, Mike Place, Brett Benassi, Erik Johnson).
If you dig deep enough into any Salt feature, you are almost guaranteed to find bugs. How to deal with that?
Salt core tends to have fewer bugs than other modules contributed by the community. The main reason for that is the very liberal merging policy: right now there are ~2200 open issues, and just ~90 open pull requests. Salt team does code reviews and usually asks a submitter to add some test coverage, but obviously can’t test every feature against a real (and often highly specialized) system. So, because the original submitter may be no longer active, these bugs tend to stay open unless someone from the community steps in.
If you are lucky, your bug may be already fixed on GitHub. However, this doesn’t mean that the fix is already included in the latest stable release. Is there a way to find out if a particular PR got merged into a specific Salt release? Depending on the release ETA, you may decide either to invent a hacky workaround or just simply wait.
Alternatively, you may want to submit a fix yourself. However, it is not immediately clear on which branch you should base the fix. How the branches evolve? What is the difference between the version number (e.g., 2018.3.4) and codename (Oxygen)? Is there a mapping between them or a calendar that contains future release dates?
By closely watching the official GitHub repo, reading the documentation and hanging out in the Slack channel, I was able to collect and summarize various useful bits into this guide. I hope it helps.
When is an existing merged PR will be released?
There are several methods which you can use to find tags or branches that contain the PR you want.
1. ZRELEASED labels on GitHub
If a PR contains a yellow
ZRELEASED - YYYY.MM.X label, then it is already released, and you need to upgrade your Salt version.
2. Git tags on Github
You can look for Git tags that contain the necessary commits:
If there are no tags and the commit was merged into the
develop branch, then you probably have to wait for the next major release. Bugfixes for older branches are eventually merged into newer ones. As for bugfixes in the
develop branch, you can politely ask to backport them or do it yourself (see How to submit a PR).
3. Git branches or tags
Let’s say you want to find when the PR #46435 is going to be released. First, you need to clone the repo:
% git clone https://github.com/saltstack/salt % cd salt
Then you need to find a commit that belongs to this PR. You can do this on GitHub (see the method 2 above), or using CLI:
% git log --grep '#46435' commit d6263a0a4fcd2573afc5177831c20f003cbc74ce Merge: f9c60ad6f7 b48255e101 Author: Nicole Thomas <firstname.lastname@example.org> Date: Fri Mar 9 09:52:22 2018 -0500 Merge pull request #46435 from onlyanegg/serializer_extmod Adding serializer extmod
The last step is to find which branches (or tags) contain the specific commit:
% git branch -a --contains d6263a0a4fcd2573afc5177831c20f003cbc74ce * develop remotes/origin/2019.2 remotes/origin/2019.2.0.rc1 remotes/origin/HEAD -> origin/develop remotes/origin/develop remotes/origin/hotfix/pytest remotes/origin/revert-50798-45839-develop remotes/origin/revert-51444-develop.py2 % git tag --contains d6263a0a4fcd2573afc5177831c20f003cbc74ce v2018.11 v2019.2 v2019.2.0rc1 v2019.2.0rc2
From the output above we can infer that the feature will be released in version
2019.2.0, which is based on the
2019.2 feature branch (also known as Fluorine). The
v2018.11 tag should be ignored because there are no branches with the same version (see the full
git branch -a output).
4. Version-check tool
The process above could be simplified using the
version_check tool and Docker. To install it clone the repo:
% git clone https://github.com/rallytime/version-check.git % cd version-check/
And build the image:
% docker build -t version_check .
Finally, to use it run the following command (46435 is the PR number):
% docker run --rm -it version_check -p 46435 Branches: 2019.2 2019.2.0.rc1 develop revert-50798-45839-develop revert-51444-develop.py2 Tags: v2018.11 v2019.2 v2019.2.0rc1 v2019.2.0rc2
For usage details see the project’s README.
The highest layer of abstraction is available as the online tool http://versions.gtmanfred.com (internally it utilizes the above-mentioned
version_check script). Unfortunately, it is no longer online. You could try to host it yourself or ping Daniel Wallace on Twitter and politely ask if he can bring it back to life.
How to submit a PR
Many times you’ll have to submit a fix yourself. Below is a simplified Salt branching model:
- New features should be submitted against the
developbranch. At some point, the branch is forked into a feature frozen codename branch. When a release date is decided upon, codename branch is renamed into the main release branch.
- Bug fixes and docs should go into the oldest affected (and supported) main release branch (e.g.,
2017.7). This is a base branch for dot release branches.
- Hotfixes should go into the corresponding dot release branch (e.g.,
- Eventually, fixes will be merged into the newer branches by SaltStack’s release team
- Occasionally, some fixes that went into the
developbranch will be back-ported to an older branch
The official documentation explains Salt branches in a more detailed fashion. To get more information on the release process, consult the Salt Git Policy and Salt Release Process pages.
Salt doesn’t currently have a scheduled release cadence. However, according to PyPI data, the expected time between releases is:
- 229 days on average for major (feature) releases (varies from 126 to 329 days)
- 48 days on average for bugfix releases (varies from 1 to 272 days)
These are rough estimates, to stay in the loop you can subscribe to the salt-announce mailing list, or join the
#releases channel on SaltStack Community Slack.
I haven’t found any reliable data that reflects the SaltStack feature planning process, but below are some clues you can use:
- Product support lifecycle table that outlines support phases and EOL dates. In short: there are two major active branches; oldest previous active is CVE-only; previous CVE-only is EOL.
- The most up-to-date list of codenames
- Github labels to track the progress on issues assigned to the specific tag (release candidates in particular)
- Github PR search to see how many open PRs are left against a specific branch
Just before a new release, you can see it being tagged on GitHub. After that it is typically a matter of days for packages to be built and uploaded. A release is usually happens in the first half of the week (70% of PyPI uploads happened in the first three days).
If you have some free time and want to improve Salt codebase, go to the issue tracker and help to fix a bug!
Also, you can follow me on Twitter where I periodically post things like this: