NetFoundry is a heavy user of SaltStack to help manage software on third-party networks. Many businesses, especially their ops teams, also benefit from configuration management software like SaltStack.
Although SaltStack itself is very secure, in May 2020, SaltStack was exploited from the networks due to a vulnerability exploit (RCE authentication bypass of high severity designated CVE-2020-11651 and CVE-2020-11652). In this blog, we will focus on why we picked SaltStack, and how we secured it with the open source platform we maintain – OpenZiti to minimize the chances of being compromised from the networks.
In fact, by dogfooding our technology, we use SaltStack without exposing any SaltStack masters to the internet. Let’s briefly introduce the main actors before we dive in:
- SaltStack is a Python-based, open-source software for event-driven IT automation, remote task execution, and configuration management. Built on Python, Salt uses simple and human-readable YAML combined with event-driven automation to deploy and configure complex IT systems.
- NetFoundry gives teams a platform to embed secure networking into any use case. NetFoundry SaaS is built on an open source platform, OpenZiti (which NetFoundry originated and is the leading contributor), and the SaaS options include free-forever tiers (for up to 10 nodes).
Why SaltStack?
SaltStack excels in two areas in which NetFoundry needs to effectively manage virtual machines (VMs) on our customer and partner networks:
1. Remote task execution to harvest quasi-real-time information about the current status of infrastructure.
2. Configuration management to maintain the integrity of the infrastructure we support.
SaltStack not only excels in these two areas, but SaltStack is open source based – just like NetFoundry. That combination of remote task execution functionality, configuration management functions, and open source made it relatively easy to take SaltStack for a whirl!
What was the initial SaltStack implementation?
NetFoundry started with a relatively standard SaltStack deployment, with an overall SaltStack architecture which looked like this:
Salt uses the master-client model in which a master issues a command to a client, and the client executes the command. In the Salt ecosystem, the Salt master is a server running the salt-master service. It issues commands to one or more Salt Minions – servers running the salt-minion service and registered with that particular Salt master. In our case, the minions run in third-party networks, such as customer and partner networks – i.e., its inherently multi-cloud networking.
What we learned about SaltStack
We learned that SaltStack, particularly in the Master/Minion operating mode, provides a very powerful infrastructure management system to meet our management and automation needs, specifically configuration management and remote task execution across different networks and clouds. However, as a zero trust provider, we also needed airtight security – for NetFoundry, our partners and our customers.
As we dug into SaltStack, we learned that the SaltStack security itself is excellent. We, therefore, turned our attention to securing the network. Our goal was ambitious, to completely close the communication ports of the Salt master – make it very difficult to attack the Salt master from the networks! In the SaltStack networking architecture, admins need to open inbound ports in the firewalls in front of the Salt master to allow communication from the Salt minions. If we somehow enable admins to close those inbound ports, then we could minimize the attack surface – minimize the chances that attackers could discover, connect to and attack the Stack architecture.
We learned that leveraging OpenZiti’s zero trust networking platform functions could effectively make SaltStack invisible to the outside world – we could close all the inbound firewall ports. This ensures that we don’t have to worry about scan and exploit attacks if another SaltStack exploit occurs.
How we implemented secure-by-design SaltStack
To make SaltStack unreachable from the networks, we had to integrate two solutions:
1. SaltStack
2. Openziti platform
Let’s look at each:
SaltStack
We started with the phase one implementation described above (you can also consult the SaltStack Fundamentals pages and documentation for more resources). As a reminder of the basic Salt communication structure:
Our distributed, zero trust networks are deployed between customers, partners and all types of cloud platforms, so security by design is mandatory for us – we ‘shift left’ to put security directly into the nucleus of our development and delivery lifecycle. So, our goal was to take the fantastic SaltStack architecture and leverage the extensibility and flexibility of its open source core while making it secure-by-default from a networking perspective.
This ‘shift left’ approach strengthens security, but it also improves business velocity. Secure-by-default enables our DevOps and SRE teams to meet their automation and agility goals without needing to take on the burden of implementing bolted-on type security measures. For example, bolted-on security such as maintaining ACLs, VPNs and security groups is complex, time-consuming and error-prone. Read more on this from Mike, our Head of DevOps.
In summary, we wanted a secure and reliable system to keep our Salt deployment unreachable from the networks while allowing us to deploy and maintain the communications between all these systems quickly. Enter OpenZiti.
Openziti meets SaltStack
Integration with OpenZiti for secure by design, zero trust networking security
To remove the attack surface, an open listening port of a SaltStack master node would provide all communication between SaltStack components through OpenZiti’s zero trust overlay network. The controller of each NetFoundry Ziti network functions as the SaltStack master for its network. NetFoundry’s NaaS engine manages the configuration of those SaltStack master nodes.
The following diagram shows a fine-grained sequence of commands and messages orchestrating the setup.
- the provisioning of a NetFoundry OpenZiti network (upper part)
- adding managed edge components (lower part)
A NetFoundry OpenZiti edge component needs to register with its network controller to become part of the secure network. Each component leverages a registration script to automate that process. This script registers OpenZiti first, then configures SaltStack to use a Ziti-enabled service. All following communication within SaltStack is secured and invisible. Further, we benefit from other OpenZiti superpowers including no port inference and private DNS. We love to dogfood OpenZiti so much, we did the same to other remote monitoring, management, and configuration tools such as our bastions, Jenkins and Prometheus.
Alternatives
Before we started applying OpenZiti to SaltStack, we considered a few alternatives. Firstly SaltStack recommends hardening your master. After having our master hacked within hours of the May 2020 vulnerability, we concluded this did not go far enough. We had to take it off the internet.
Many other organizations use VPNs such as Wireguard or ZeroTier. We felt this had a few drawbacks, including their security approach was not as comprehensive (e.g., neither provides a deny all inbound rule), not wanting to manage complex firewall port rules, and ensuring it worked everywhere by using (e.g., being based on TCP rather than UDP). We have not made a direct comparison on a solution such as Wireguard vs OpenZiti, but we have compared Tailscale (a commercial SaaS of Wireguard) vs NetFoundry (a commercial SaaS of OpenZiti).
Feedback
I would love to hear from you on what you think about how we have used SaltStack and applied zero trust networking to it using OpenZiti. Do you think it’s an effective VPN replacement? What could we do better? OpenZiti directly into SaltStack or ‘zitify’ ZeroMQ to secure its transport?
