Announcing Salt Grafana

6 minute read Updated:

Salt Grafana dashboard

TLDR: This is an open-source set of components that uses Grafana stack to implement a visualization, observability and alerting tool for Salt. To start using it, please visit the documentation.

Open-source Salt observability tool based on Grafana

Purpose

If you run a non-trivial number of jobs on a Salt master (especially in an automated way), some of the following questions should look familiar:

If that rings true, then Salt Grafana is for you!

Design decisions

  1. Once configured, Salt Master settings don’t need to be changed to add a new metric, transformation or event destination. All transforms are done outside of Salt.
  2. Salt orchestrations are the primary units of work (but this constraint might be relaxed later).
  3. The default event parsers and Grafana dashboards were designed to deliver the visualizations required by one particular company, but the project is quite flexible and extensible. You are encouraged to tweak the default dashboards and panels, add new Vector transformations, metrics and alerts.
  4. The components are optional and are considered to be the building blocks you can mix and match. You do not have to use PostgreSQL job cache if you can live without the TreeView panel and do not need to ask complex questions that require ad hoc SQL queries. If you do not need to visualize orchestrations as traces, you can drop the Tempo Relay service and Grafana Agent.
  5. Some questions are easier to answer with SQL queries, some with LogQL.
  6. Grafana Cloud is used to reduce the amount of infrastructure to maintain. There is nothing that prevents using on-premise Grafana stack (although this wasn’t tested).
  7. PostgreSQL is used because Loki can’t store large blobs of data (the limit is 64 kb). Salt job returns (esp. for orchestrations) can easily exceed that.
  8. Although it wasn’t tested, the system should support multiple Salt masters (with the exception of a singular Job Cache DB that should be dedicated for each master unless it is a shared cache for HA setup).
  9. Vector engine should be usable not only on masters, but also on minions as well (although this wasn’t tested).

Architecture

Here is how the full system is wired together:

Salt Grafana architecture

Future ideas

If Salt Grafana doesn’t do what you want out of the box, feel free to submit a merge request to the salt-grafana repo. If you want to contribute a complex or breaking change, please file an issue first to discuss it. Also, you can hire me to customize the tool for your specific needs.

Similar projects

I’m aware of three other projects that add metrics to Salt:

If Salt Grafana is not the right tool for you, maybe other visual UIs for Salt that solve different use-cases might be useful. In the past, I’ve already compiled a list of Salt GUIs, but it is worth repeating it here for better discoverability:

It would be unfair not to mention a commercial offering from VMware called SaltStack Config (SSC, previously called SaltStack Enterprise or SSE), but it is quite enterprisey/expensive and seems to become more and more integrated into the VMware ecosystem.

This project was made possible by:

The project links are:

You can follow me on Twitter to find more useful tips like this:

SaltTips tweet