Upgrading Salt to Python 3

21 minute read Updated:

Upgrading Salt

As of January 1st, 2020, Python 2 has officially reached the end of life and will no longer receive any updates (except the last one in April). Various Linux distributions will support it for a while; however, many organizations already switched to Python 3 or plan to do so in the near future.

It is time to switch your Salt install to Python 3:

This guide describes how to switch the repo from Py2 to Py3, identify and uninstall the old packages, then install the Py3 version. It contains detailed checklists that will help you to plan and perform the migration. It can be also applied to regular upgrades. However, it is not a 100% automatic foolproof recipe you can apply blindly.

10 key principles

Over the years of using Salt, I developed a set of ten opinionated guiding principles that will help you maintain and upgrade it. You do not have to follow these guidelines, and I often break them too. However, they are implied throughout the article.

1. Have a QA environment

QA environment can be manual or automated. The goal is to be able to test the required combinations of Operating Systems, Salt versions (master & minion), and possible highstates. Always do staged/gradual rollouts. You’ll save a lot of time if your QA environment supports snapshots.

2. Run the same Salt version everywhere

Master and minion versions should be the same (the only exceptions are [1] and [5]). This is the most traveled path. You won’t need to adapt your states to different feature sets. Also, the upgrade process is much easier.

3. Avoid rare operating modes

Less traveled paths and settings that significantly change how Salt operates (transport, multiprocessing, zmq filtering, caching, undocumented things), tend to result in nasty surprises. If you want to change a setting, give it a try in your QA environment for a while.

4. Control the bootstrap/upgrade process

Yes, a minor upgrade will require more work. But it is work you can control and decide when to do. You can’t control work that appears out of the blue when minions automatically upgrade and something breaks.

5. Upgrade the master first

When you upgrade Salt, upgrade the master first. Older master can (and eventually will) break things.

6. Back up the keys

Set up periodic backups of the /etc/salt/pki master directory. Other than that, your Salt master should be disposable, and states/pillars should be kept in a version control system (or pulled from external data sources).

7. Have a backup access path

Have a plan B to access your servers when Salt goes down. Use salt-ssh, fabric, ansible, ssh, winrm, or any other tool. It is also helpful to keep a list of minion IPs (you can periodically extract them from the master grain cache).

8. Use Salt as an O&M control plane

Avoid making your production workload depend on Salt. If you uninstall Salt, your production environment should remain operational (at least for a while).

If you have the resources, you may also want to consider the last two principles:

9. Mirror the Salt repo

I never had to do this, but I’ve seen rare connectivity issues with the official repo. Also, for some folks, the minion bootstrap process was broken when SaltStack removed (and then quickly returned back) the buggy 2019.2.1 release. This is less of an issue if you follow the principle [4].

10. Build your own packages

Bugs will happen, and you need to fix them quickly without waiting for the next release. Also, you may want to integrate new features before they are shipped. Custom packages are a way to achieve that. They give you full control over the Salt code you run.

I ran a patched salt-master from a virtualenv for some time in 2012 or 2013, but I no longer do that. Instead, I prefer the more lightweight ways to patch Salt.

Over the years of using Salt, I developed a set of ten opinionated guiding principles that will help you maintain and upgrade it.

Plan the upgrade process

1. Survey your master and minion fleet

A list of useful commands is provided below. There are many dimensions you should inspect:

The main goal is to find all the existing combinations, so you can exhaustively test the upgrade process. Also check out the Py3 support list.

2. Prepare the QA environment

Ideally, you want to be able:

  1. Test the same OS/Salt versions
  2. Test the existing highstates
  3. Test the actual workload that runs on top of your infra

You’ll need to test the upgrade process multiple times, so a fast feedback cycle is necessary (e.g., it would be nice to quickly snapshot/restore your QA environment).

3. Update the states

Read the official release notes and search for any open issues. Make sure your states are compatible with the target Salt version (see this handy guide on how to write backward compatible state files).

4. Decide on the upgrade process

For example:

  1. Perform the backups (master/minion keys, alternate connection paths, etc.)
  2. Upgrade OS packages, run autoclean/autoremove. Optionally reboot if you have new kernels.
  3. Disable any periodic Salt jobs and other stuff that can interfere with the upgrade process. Try minion blackouts
  4. Upgrade master to the target version, while staying on Py2
  5. Ensure that the existing process to bootstrap new minions uses the target Salt version (Py3)
  6. Upgrade minions to the target version, while staying on Py2. If that is not possible, upgrade to the latest available for each distro.
  7. Upgrade Salt master to Py3
  8. Upgrade Salt minions to Py3 for each OS separately
  9. Enable periodic Salt jobs and other things you disabled on step 3

Consider the minion upgrade order. Your environment may require a specific upgrade sequence.

The goal is to remove the old Salt version and its dependencies (to avoid mixing packages from different repos), then bootstrap the Py3 one from scratch. If your minions are disposable, you can avoid the in-place upgrade process and instead rebuild them from scratch. Another option is blue/green deployment.

5. Test the upgrade process multiple times

  1. Manually upgrade your masters
  2. Make sure their minions are reachable after the upgrade
  3. Manually upgrade a single minion for each OS
  4. Make sure the upgraded minions are reachable, the highstates can be applied, and the business apps are functional
  5. Automate the above steps
  6. Test the automation multiple times
  7. Test the backup options (master/minion keys, alternate connection paths, etc.)
  8. Test the bootstrap process for new minions

6. Do the upgrade

  1. Decide how many upgrade stages you want
  2. Decide on the appropriate schedule
  3. Inform the team(s)
  4. Perform and verify the backups (master/minion keys, alternate connection paths, etc.)
  5. Switch the bootstrap for new minions to the target version
  6. Upgrade the master
  7. Upgrade the minions
  8. Monitor the result

Survey your master and minion fleet

Listing versions

% sudo salt '*' test.version
% sudo salt '*' pkg.version salt-minion
% sudo salt '*' test.versions_report

(you may find a lot of variation in the installed dependencies!)

% sudo salt '*' grains.item pythonversion
% sudo salt-run manage.versions

Master:
    2019.2.4
Minion requires update:
    ----------
    minion1:
        2018.3.5
    minion2:
        2018.3.5
    minion3:
        2018.3.5
    win10:
        2018.3.5
sudo salt-run survey.hash '*' test.version
|_
  ----------
  pool:
      - minion1
      - minion2
      - minion3
      - win10
  result:
      2018.3.5

Listing repos and keys

% sudo salt '*' pkg.list_repos
% sudo salt '*' pkg.get_repo_keys

Combine with jq to get more compact output:

% sudo salt -G "os:Ubuntu" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltproject.io/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main"
"deb https://repo.saltproject.io/apt/ubuntu/18.04/amd64/archive/2019.2.4 bionic main"
% sudo salt -G "os:Debian" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[][].line'

"deb https://repo.saltproject.io/apt/debian/9/amd64/archive/2018.3.5 stretch main"
"deb https://repo.saltproject.io/apt/debian/9/amd64/archive/2019.2.4 stretch main"
% sudo salt -G "os:CentOS" pkg.list_repos --static --out json | jq '.[] | with_entries(select(.key|contains("saltstack"))) | .[].baseurl'

"https://repo.saltproject.io/yum/redhat/7/$basearch/archive/2018.3.5/"
"https://repo.saltproject.io/yum/redhat/7/$basearch/archive/2019.2.4/"

Targeting py2 minions

% sudo salt -C "G@os:Ubuntu and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Debian and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:CentOS and G@pythonversion:0:2" grains.item pythonversion saltversion
% sudo salt -C "G@os:Windows and G@pythonversion:0:2" grains.item pythonversion saltversion

Other useful grains for targeting

Surveys

Any module calls (test.version, cmd.run, grains.item, etc.) can be combined with the survey runner:

sudo salt-run survey.hash '*' grains.item os osfinger pythonversion saltversion
|_
  ----------
  pool:
      - minion3
  result:
      {u'osfinger': u'Ubuntu-18.04', u'saltversion': u'2018.3.5', u'os': u'Ubuntu', u'pythonversion': [2, 7, 17, u'final', 0]}
|_
  ----------
  pool:
      - minion2
  result:
      {u'osfinger': u'CentOS Linux-7', u'saltversion': u'2018.3.5', u'os': u'CentOS', u'pythonversion': [2, 7, 5, u'final', 0]}
|_
  ----------
  pool:
      - minion1
  result:
      {u'osfinger': u'Debian-9', u'saltversion': u'2018.3.5', u'os': u'Debian', u'pythonversion': [2, 7, 13, u'final', 0]}
|_
  ----------
  pool:
      - win10
  result:
      {u'osfinger': u'Windows-10', u'saltversion': u'2018.3.5', u'os': u'Windows', u'pythonversion': [2, 7, 14, u'final', 0]}

Listing package origins

The goal is to find packages that were installed from a specific origin, so that you can remove them during the upgrade process.

Ubuntu/Debian

I found the following snippet on Ask Ubuntu. If you know a more straightforward way to do the same without installing aptitude or other packages, I’m all ears.

% apt-cache policy $(dpkg -l | awk 'NR >= 6 { print $2 }') |
  awk '/^[^ ]/    { split($1, a, ":"); pkg = a[1] }
    nextline == 1 { nextline = 0; printf("%-40s %-50s %s\n", pkg, $2, $3) }
    /\*\*\*/      { nextline = 1 }' | grep saltstack

The output can vary depending on the OS:

salt-common                              https://repo.saltproject.io/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
salt-minion                              https://repo.saltproject.io/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic/main
python-jinja2                            https://repo.saltproject.io/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-common                              https://repo.saltproject.io/apt/debian/9/amd64/archive/2018.3.5 stretch/main
salt-minion                              https://repo.saltproject.io/apt/debian/9/amd64/archive/2018.3.5 stretch/main

Additionally, you can inspect how a Salt minion was installed in the first place. You need to figure out which packages are installed explicitly (the rest of the dependencies can be auto removed). Check out the /var/log/apt/history.log:

Debian:

Start-Date: 2020-02-18  05:15:02
Commandline: apt-get install -y -o DPkg::Options::=--force-confold procps pciutils python-yaml
Requested-By: vagrant (1000)
Install: python-yaml:amd64 (3.12-1), libyaml-0-2:amd64 (0.1.7-2, automatic)
End-Date: 2020-02-18  05:15:02

Start-Date: 2020-02-18  05:15:05
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg2 apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: gnupg2:amd64 (2.1.18-8~deb9u4), apt-transport-https:amd64 (1.4.9)
End-Date: 2020-02-18  05:15:06

Start-Date: 2020-02-18  05:15:35
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion
Requested-By: vagrant (1000)

Ubuntu:

Start-Date: 2020-02-18  05:15:08
Commandline: apt-get install -y -o DPkg::Options::=--force-confold wget gnupg apt-transport-https ca-certificates
Requested-By: vagrant (1000)
Install: apt-transport-https:amd64 (1.6.12)
End-Date: 2020-02-18  05:15:08

Start-Date: 2020-02-18  05:15:49
Commandline: apt-get install -y -o DPkg::Options::=--force-confold python2.7 python-apt python-requests python-yaml procps pciutils
Requested-By: vagrant (1000)
Install: python-six:amd64 (1.11.0-2, automatic), python-openssl:amd64 (17.5.0-1ubuntu1, automatic), python-yaml:amd64 (3.12-1build2), python-requests:amd64 (2.18.4-2ubuntu0.1), python-certifi:amd64 (2018.1.18-2, automatic), python-chardet:amd64 (3.0.4-1, automatic), python-enum34:amd64 (1.1.6-2, automatic), python-cryptography:amd64 (2.1.4-1ubuntu1.3, automatic), python-cffi-backend:amd64 (1.11.5-1, automatic), python-ipaddress:amd64 (1.0.17-1, automatic), python-pkg-resources:amd64 (39.0.1-2, automatic), python-apt:amd64 (1.6.5ubuntu0.2), python-urllib3:amd64 (1.22-1ubuntu0.18.04.1, automatic), python-idna:amd64 (2.6-1, automatic), python-asn1crypto:amd64 (0.24.0-1, automatic)
Upgrade: python2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), python2.7:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-minimal:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04), libpython2.7-stdlib:amd64 (2.7.15-4ubuntu4~18.04.2, 2.7.17-1~18.04)
End-Date: 2020-02-18  05:15:56

Start-Date: 2020-02-18  05:16:11
Commandline: apt-get install -y -o DPkg::Options::=--force-confold salt-minion

The bootstrap-salt.sh script also logs everything to /tmp/bootstrap-salt.log:

pciutils is already the newest version (1:3.5.2-1).
procps is already the newest version (2:3.3.12-3+deb9u1).
...
Selecting previously unselected package python-yaml.
Preparing to unpack .../python-yaml_3.12-1_amd64.deb ...
Unpacking python-yaml (3.12-1) ...
Setting up libyaml-0-2:amd64 (0.1.7-2) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Setting up python-yaml (3.12-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version (20161130+nmu1+deb9u1).
ca-certificates set to manually installed.
wget is already the newest version (1.18-5+deb9u3).
The following NEW packages will be installed:
  apt-transport-https gnupg2
...
The following additional packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common
Suggested packages:
  debtags dbus-user-session pinentry-gnome3 tor apache2 | lighttpd | httpd
  python-apt-dbg python-apt-doc python-cryptography-doc
  python-cryptography-vectors python-enum34-doc python-jinja2-doc
  python-openssl-doc python-openssl-dbg python-psutil-doc doc-base
  python-socks python-setuptools-doc python-mysqldb python-pycurl
  python-tornado-doc python-twisted python-ntlm python-augeas
Recommended packages:
  sfdisk e2fprogs
The following NEW packages will be installed:
  dctrl-tools debconf-utils dirmngr javascript-common libjs-jquery
  libjs-sphinxdoc libjs-underscore libpgm-5.2-0 libsodium18 libzmq5 python-apt
  python-backports-abc python-cffi-backend python-chardet
  python-concurrent.futures python-croniter python-cryptography
  python-dateutil python-enum34 python-idna python-ipaddress python-jinja2
  python-markupsafe python-msgpack python-openssl python-pkg-resources
  python-psutil python-pyasn1 python-requests python-setuptools
  python-singledispatch python-six python-systemd python-tornado python-tz
  python-urllib3 python-zmq salt-common salt-minion

So, for Debian/Ubuntu you may want to remove the salt-minion, python-requests, python-yaml, and python-apt explicitly, then auto remove the dependencies (please double-check that it uninstalls the salt-common package).

CentOS/Amazon

% yum list installed | grep saltstack

PyYAML.x86_64                          3.11-1.el7                     @saltstack
python-zmq.x86_64                      15.3.0-3.el7                   @saltstack
salt.noarch                            2018.3.5-1.el7                 @saltstack
salt-minion.noarch                     2018.3.5-1.el7                 @saltstack
zeromq.x86_64                          4.1.4-7.el7                    @saltstack

The /var/log/yum.log is not very informative:

Feb 18 08:15:37 Updated: rpm-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:37 Updated: rpm-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-build-libs-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Updated: rpm-python-4.11.3-40.el7.x86_64
Feb 18 08:15:38 Installed: python-chardet-2.2.1-3.el7.noarch
Feb 18 08:15:38 Installed: python-kitchen-1.1.1-5.el7.noarch
Feb 18 08:15:38 Installed: libyaml-0.1.4-11.el7_0.x86_64
Feb 18 08:15:38 Updated: libxml2-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Installed: libxml2-python-2.9.1-6.el7_2.3.x86_64
Feb 18 08:15:38 Updated: python-urlgrabber-3.10-9.el7.noarch
Feb 18 08:15:39 Updated: yum-3.4.3-163.el7.centos.noarch
Feb 18 08:15:39 Installed: yum-utils-1.1.31-52.el7.noarch
Feb 18 08:15:39 Installed: PyYAML-3.11-1.el7.x86_64
Feb 18 08:15:39 Updated: chkconfig-1.7.4-1.el7.x86_64
Feb 18 08:16:15 Installed: lz4-1.7.5-3.el7.x86_64
Feb 18 08:16:15 Updated: systemd-libs-219-67.el7_7.3.x86_64
Feb 18 08:16:15 Installed: python-ipaddress-1.0.16-2.el7.noarch
Feb 18 08:16:15 Installed: python-markupsafe-0.11-10.el7.x86_64
Feb 18 08:16:15 Installed: python-six-1.9.0-2.el7.noarch
Feb 18 08:16:15 Installed: libtommath-0.42.0-6.el7.x86_64
Feb 18 08:16:15 Installed: libtomcrypt-1.17-26.el7.x86_64
Feb 18 08:16:16 Installed: python2-crypto-2.6.1-16.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-1.0-8.el7.x86_64
Feb 18 08:16:16 Installed: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
Feb 18 08:16:16 Installed: python-tornado-4.2.1-5.el7.x86_64
Feb 18 08:16:16 Installed: python-urllib3-1.10.2-7.el7.noarch
Feb 18 08:16:16 Installed: python-requests-2.6.0-8.el7_7.noarch
Feb 18 08:16:17 Installed: python-babel-0.9.6-8.el7.noarch
Feb 18 08:16:18 Installed: python-jinja2-2.7.2-4.el7.noarch
Feb 18 08:16:18 Installed: libsodium-1.0.18-1.el7.x86_64
Feb 18 08:16:18 Installed: python2-msgpack-0.5.6-5.el7.x86_64
Feb 18 08:16:18 Updated: cryptsetup-libs-2.0.3-5.el7.x86_64
Feb 18 08:16:20 Updated: systemd-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: systemd-python-219-67.el7_7.3.x86_64
Feb 18 08:16:20 Installed: openpgm-5.2.122-2.el7.x86_64
Feb 18 08:16:20 Installed: zeromq-4.1.4-7.el7.x86_64
Feb 18 08:16:21 Installed: python-zmq-15.3.0-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-futures-3.1.1-5.el7.noarch
Feb 18 08:16:21 Updated: pciutils-libs-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: pciutils-3.5.1-3.el7.x86_64
Feb 18 08:16:21 Installed: python2-psutil-2.2.1-5.el7.x86_64
Feb 18 08:16:26 Installed: salt-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Installed: salt-minion-2018.3.5-1.el7.noarch
Feb 18 08:16:26 Updated: systemd-sysv-219-67.el7_7.3.x86_64
Feb 18 08:16:26 Updated: libgudev1-219-67.el7_7.3.x86_64

However, the /tmp/bootstrap-salt.log shows precisely which packages were requested and which are merely dependencies:

================================================================================
 Package               Arch       Version                   Repository     Size
================================================================================
Installing:
 PyYAML                x86_64     3.11-1.el7                saltstack     160 k
 yum-utils             noarch     1.1.31-52.el7             base          121 k
Updating:
 chkconfig             x86_64     1.7.4-1.el7               base          181 k
Installing for dependencies:
 libxml2-python        x86_64     2.9.1-6.el7_2.3           base          247 k
 libyaml               x86_64     0.1.4-11.el7_0            base           55 k
 python-chardet        noarch     2.2.1-3.el7               base          227 k
 python-kitchen        noarch     1.1.1-5.el7               base          267 k
Updating for dependencies:
 libxml2               x86_64     2.9.1-6.el7_2.3           base          668 k
 python-urlgrabber     noarch     3.10-9.el7                base          108 k
 rpm                   x86_64     4.11.3-40.el7             base          1.2 M
 rpm-build-libs        x86_64     4.11.3-40.el7             base          107 k
 rpm-libs              x86_64     4.11.3-40.el7             base          278 k
 rpm-python            x86_64     4.11.3-40.el7             base           83 k
 yum                   noarch     3.4.3-163.el7.centos      base          1.2 M

Transaction Summary
================================================================================
Install  2 Packages (+4 Dependent packages)
Upgrade  1 Package  (+7 Dependent packages)

================================================================================
 Package                              Arch    Version          Repository  Size
================================================================================
Installing:
 salt-minion                          noarch  2018.3.5-1.el7   saltstack   37 k

Below is a similar log for Amazon Linux:

Installing:
 m2crypto            x86_64     0.31.0-4.el7           saltstack-repo     287 k
 python-crypto       x86_64     2.6.1-2.el7            saltstack-repo     470 k
 python-msgpack      x86_64     0.4.6-1.el7            saltstack-repo      73 k
 python-zmq          x86_64     15.3.0-3.el7           saltstack-repo     520 k
Updating for dependencies:
 PyYAML              x86_64     3.11-1.el7             saltstack-repo     160 k
 procps-ng           x86_64     3.3.10-26.amzn2        amzn2-core         292 k
 python-requests     noarch     2.6.0-7.amzn2          amzn2-core          95 k
Installing for dependencies:
 libsodium           x86_64     1.0.16-1.el7           saltstack-repo     140 k
 openpgm             x86_64     5.2.122-2.el7          saltstack-repo     172 k
 python2-typing      noarch     3.5.2.2-4.el7          saltstack-repo      39 k
 zeromq              x86_64     4.1.4-7.el7            saltstack-repo     555 k

So, for CentOS/Amazon you may want to remove the salt-minion, PyYAML, m2crypto, python-zmq, python-crypto, and python-msgpack explicitly, then auto remove the dependencies.

Another option is to run sudo yum history list or sudo yum history package-list '*' to inspect the YUM transaction history and find what was installed.

Listing dependencies using pip

If you installed some Salt dependencies via pip, it is worth inspecting them too:

% sudo salt -C "G@kernel:Linux" cmd.run 'pip freeze --local'
% sudo salt -t 30 -C "G@os:Windows" cmd.run 'pip.exe freeze --local' cwd='C:\salt\bin\scripts\'

On CentOS, Amazon, and Windows, the command shows a full list of python packages, so you’ll need to compare it with a default one to find what is different.

Upgrade the Salt Master

From the FAQ:

Q: Can I run different versions of Salt on my Master and Minion?

A: This depends on the versions. In general, it is recommended that Master and Minion versions match. When upgrading Salt, the master(s) should always be upgraded first. Backwards compatibility for minions running newer versions of Salt than their masters is not guaranteed.

There are multiple ways to upgrade a Salt master:

Option 1

The in-place upgrade process:

  1. Backup the /etc/salt/pki directory (just in case)
  2. Remove the old Salt repo
  3. Uninstall the old salt-master package and all of its dependencies
  4. Add the new repo (py3)
  5. Install the new salt-master

Option 2

If you can’t disconnect all minions at once during the upgrade process, it is possible to bring up an additional master server and migrate minions one-by-one:

  1. Set up new salt-master (py3)
  2. Copy the /etc/salt/pki directory from old master to the new one, to ensure that master and minion keys are the same
  3. Migrate the minions:
# migrate_minion.sls
migrate_to_new_master:
  file.replace:
    - name: "{{ salt['config.get']('conf_file') }}"
    # Alternative variant if you keep master address in a separate config file
    # - name: "{{ salt['file.join'](salt['config.get']('config_dir'), 'minion.d', 'master.conf') }}"
    - pattern: '^master:\s+{{ salt["config.get"]("master") | regex_escape }}$'
    - repl: 'master: new-master.example.com'

# Restart salt minion after the the config option is set
# A naive module.run with service.restart stopped working in 2019.2.1
# (https://github.com/saltstack/salt/issues/55201), so we have to use salt-call
# https://docs.saltproject.io/en/latest/faq.html#restart-using-states
restart_salt_minion:
  cmd.run:
    - name: 'salt-call --local service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: migrate_to_new_master

To be safe, you can migrate minions in batches:

% sudo salt -G 'master:old-master.example.com' --batch-size 10% state.apply migrate_minion

Option 3

Another option was suggested by Carlos D. Álvaro on Twitter. The idea is to keep the master in a Docker container and mount the logs, keys, and states/pillars from a host machine. You can find an example of such a container on Carlos’ GitHub.

Upgrade Salt on Windows

With winrepo

Things are very easy in Windows land if you use winrepo-ng:

1. Update your winrepo definitions

If you do not want to bring the whole winrepo-ng, just put the following two files into salt://win/repo-ng:

2. Rebuild the package database

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.refresh_db

3. List the currently installed version

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.version salt-minion

3. Make sure the target version is available

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.list_available salt-minion-py3

4. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2" pkg.install salt-minion-py3 version=2019.2.4

You might want to add the --batch-size X argument for a staged rollout.

Without winrepo

Do not want to use winrepo at all? Below is an alternative way to upgrade Windows minions.

1. Fetch the installer

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url https://repo.saltproject.io/windows/Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltproject.io\windows\Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe

You can also include a checksum to prevent redownloads (works in Salt 2018.3.0 or greater):

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" \
  cp.get_url https://repo.saltproject.io/windows/Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe \
  source_hash=$(curl -s https://repo.saltproject.io/windows/Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe.sha256 |\
  iconv -f utf-16 -t utf-8 | awk '{print $1;}' | tr A-Z a-z)

win10:
    c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltproject.io\windows\Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe

The default download location is fine, because the Salt installer doesn’t remove the cache dir since 2017.7.2.

2. Run the upgrade command

% sudo salt -t 30 -C "G@os:Windows and G@pythonversion:0:2 and G@cpuarch:AMD64" cmd.run \
  'START /B c:\salt\var\cache\salt\minion\extrn_files\base\repo.saltproject.io\windows\Salt-Minion-2019.2.4-Py3-AMD64-Setup.exe /S'

Upgrade Salt on Ubuntu and Debian

There are a few key pieces:

1. Remove the old repo and key

remove_salt_repo:
  pkgrepo.absent:
    - name: deb https://repo.saltproject.io/apt/ubuntu/18.04/amd64/archive/2018.3.5 bionic main
#   - name: deb https://repo.saltproject.io/apt/debian/9/amd64/archive/2018.3.5 stretch main
    - keyid: 0E08A149DE57BFBE

It is also possible to delete all existing Salt repos and keys on both Ubuntu and Debian:

{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

apt-get -y remove --auto-remove salt-minion
apt-get -y remove --auto-remove python-requests python-yaml python-apt

4. Install the minion

While it is possible to just apt-get install the necessary packages:

apt-get -y -o DPkg::Options::=--force-confold install python3-requests python3-yaml python3-apt
apt-get -y -o DPkg::Options::=--force-confold install salt-minion

I feel that using salt-bootstrap.sh is simpler and supports more platforms:

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltproject.io | sh -s -- -x python3 stable 2019.2.4

Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.

5. The resulting state

# upgrade/ubuntu_debian.sls
{% for repo_key, repo_data in salt['pkg.list_repos']().items() -%}
{% for repo in repo_data -%}
{% if not repo.get('disabled') and 'saltstack' in repo.get('line', '') -%}
Remove repo {{ repo['line'] }}:
  pkgrepo.absent:
    - name: {{ repo['line'] }}
    # - keyid: 0E08A149DE57BFBE
{% endif %}
{% endfor %}
{%- endfor %}

{% for key_id, key_data in salt['pkg.get_repo_keys']().items() -%}
{% if 'saltstack' in key_data.get('uid', '') -%}
Remove key {{ key_data['keyid'] }}:
  module.run:
{%- if 'module.run' in salt['config.get']('use_superseded', []) %}
    - pkg.del_repo_key:
        - keyid: {{ key_data['keyid'] }}
{%- else %}
    - name: pkg.del_repo_key
    - kwargs:
        keyid: {{ key_data['keyid'] }}
{% endif %}
{% endif %}
{%- endfor %}

install_at:
  pkg.installed:
    - name: at

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        apt-get -y remove --auto-remove salt-minion
        apt-get -y remove --auto-remove python-requests python-yaml python-apt
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltproject.io | sh -s -- -x python3 stable 2019.2.4" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

6. Run the upgrade command

% sudo salt -C "( G@os:Ubuntu or G@os:Debian ) and G@pythonversion:0:2" state.apply upgrade.ubuntu_debian

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on CentOS and Amazon Linux

There are a few key pieces:

1. Remove the old repo

remove_salt_repo:
  pkgrepo.absent:
    - name: saltstack

It is also possible to delete all existing Salt repos on any rpm-based distro:

{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

2. Stop the salt-minion service

systemctl stop salt-minion.service

3. Remove the packages

The snippet below should be adapted to your specific environment (i.e., remove all the dependencies, but do not touch any packages that are critical for production workload):

yum -y remove salt-minion
yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
yum -y autoremove

4. Install the minion without starting it

rm -rf /var/cache/salt/minion
curl -Ls https://bootstrap.saltproject.io | sh -s -- -X -x python3 stable 2019.2.4

Removing the cache dir is optional, however it was recommended some time ago on Windows (no longer so since 2017.7.2). Also, this comment hints on possible incompatibilities between cache serialization formats. And finally, it could be important if you use version-specific custom modules.

5. Reuse the old config

The snippet below should be adapted to your specific environment:

[ -f /etc/salt/minion.rpmsave ] && \
( cp -f /etc/salt/minion /etc/salt/minion.bak; \
mv -f /etc/salt/minion.rpmsave /etc/salt/minion )

6. Start the salt-minion service

systemctl is-enabled salt-minion.service || \
(systemctl preset salt-minion.service && \
systemctl enable salt-minion.service)
systemctl start salt-minion.service

7. The resulting state

# upgrade/centos_amazon.sls
{% for key, repo in salt['pkg.list_repos']().items() -%}
{% if repo.get('enabled', '1') == '1' and 'saltstack' in key -%}
Remove repo {{ key }}:
  pkgrepo.absent:
    - name: {{ key }}
{% endif -%}
{% endfor %}

install_at:
  pkg.installed:
    - name: at

atd_service:
  service.running:
    - name: atd
    # - enable: True

upgrade_salt_minion:
  cmd.run:
    - name: |
        echo "systemctl stop salt-minion.service
        yum -y remove salt-minion
        yum -y remove PyYAML m2crypto python-zmq zeromq python-crypto python-msgpack
        yum -y autoremove
        rm -rf /var/cache/salt/minion
        curl -Ls https://bootstrap.saltproject.io | sh -s -- -X -x python3 stable 2019.2.4
        [ -f /etc/salt/minion.rpmsave ] && \
        ( cp -f /etc/salt/minion /etc/salt/minion.bak; \
        mv -f /etc/salt/minion.rpmsave /etc/salt/minion )
        systemctl is-enabled salt-minion.service || \
        (systemctl preset salt-minion.service && \
        systemctl enable salt-minion.service)
        systemctl start salt-minion.service" | at now

The at trick was once recommended in the official docs. See the following issues if you like some archaeology: #5721, #7997, #22993, #32593, #39952, #43340, #46709.

You can also try the systemd-run --scope method.

8. Run the upgrade command

% sudo salt -C "( G@os:CentOS or G@os:Amazon ) and G@pythonversion:0:2" state.apply upgrade.centos_amazon

You might want to add the --batch-size X argument for a staged rollout.

Upgrade Salt on other operating systems

If you have done this on any other operating system (Red Hat, Fedora, SUSE, Gentoo, Arch, Raspbian, macOS, Solaris, FreeBSD, OpenBSD, etc.), feel free to share your experience and possible gotchas. I’ll be happy to update the article with any specific recipes!

Monitor the upgrade process

The upgrade process can take some time. There are three ways to monitor minions as they reconnect:

Watch the master log

% sudo tail -f /var/log/salt/master

Watch the master event bus

% sudo salt-run state.event tagmatch='salt/minion/*/start' pretty=true

List the Py2/Py3 minion count

% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0]).\(.[1]).\(.[2])"' | sort | uniq -c

      1 "2.7.13"
      1 "2.7.17"
      1 "2.7.5"
      1 "3.5.3"
      1 "3.6.8"
      1 "3.6.9"
% sudo salt "*" grains.get pythonversion --out json --static | jq '.[] | "\(.[0])"' | sort | uniq -c

      3 "2"
      3 "3"

Upgrade automation

If you need more control over the upgrade process (handle retries and upgrade failures, define specific orchestration sequence, etc.) you can use Salt orchestration or preferably write a custom runner. Another variant is to use salt-api and pepper.

An out-of-band upgrade using fabric or salt-ssh is also an option.

Future developments

In 2020 SaltStack introduced Tiamat to make self-contained Salt binaries. Hopefully, it will simplify the dependency handling and make the upgrade process easier.

See also:

Also, you can follow me on Twitter where I periodically post things like this:

SaltTips tweet