Ever had that feeling your stomach is about to drop to the floor?
It’s what happens when you implement a configuration management system and then figure out it’s been affected by a zero-day vulnerability.
I’m sure you’ve heard of SaltStack; it’s one of the de facto configuration management tools in the market and provides the much-needed infrastructure automation that is a must in today’s computing environment.
In May 2020, SaltStack gave a whole lot of systems and network administrators that sinking feeling as they released the news of a zero-day exploit. Attackers found a flaw that allowed them to bypass the authentication and send root-level commands directly to the master.
Follow that up with the ability to traverse the local file system & essentially, the master was at the attackers’ disposal with just a few easy commands.
Many systems got pwned by this Saltstack exploit!
Read more on the above issues here:
*SaltStack authorization bypass
*Critical SaltStack vulnerability affects thousands of datacentres
How could this have affected so many systems?
How on earth were these systems exposed directly to the internet?
In today’s reality that’s what you call a distributed system. Networks are not isolated or local. They span onsite networks with multiple clouds & are spread across large geographical areas. Securing distributed configuration management tools and their connections is expensive and very, very complicated to deploy and maintain.
Before proceeding, let’s review the SaltStack architecture at a high level.
SaltStack Architecture 101:
Distributed systems that are SaltStack managed run the minion process and establish outbound connections to the master. Those connections back to the master led to this infamous disaster; the master had to accept inbound connections from the minions.
As we saw from the outcome of the SaltStack vulnerabilities, many SaltStack master servers were sitting directly on the internet, waiting for those incoming connections and therefore vulnerable to exploit.
It’s how it works, right? If you want a connection back to the master, you need to open the inbound firewall ports, right?
Not necessarily. In comes OpenZiti & Zero Trust.
At Netfoundry, we solved our security concerns by ‘dogfooding’ our own technology, OpenZiti.
- OpenZiti is an open source application-defined networking overlay solution built on zero trust principles.
- Netfoundry is a orchestration & management platform providing a SaaS version of OpenZiti.
We deployed OpenZiti to ensure one of the most important (and yet advanced) principles of Zero Trust networks – demystify definitions of zero trust here.
No listening and inbound ports, aka Dark Services.
Leveraging OpenZiti, we could configure the SaltStack master only to accept connections from the localhost!
Here’s the old architecture versus the new:
In the new architecture, the master only allows inbound connections from OpenZiti. No inbound Firewall ports are open & the systems are still able to connect.
It’s like magic…except it’s not! It’s just one of the principles of Zero Trust.
OpenZiti makes an outbound connection to a secure public router removing exposure to the public internet.
Portability is another issue we face, as all others do with today’s distributed systems. It’s not enough to set this up in a local network; it needs to be able to span private networks and multiple cloud platforms and reach all the way into different types of deployments. Luckily, Netfoundry and OpenZiti provide just that.
A cloud-native solution that works across both public and private cloud providers.
Instead of having all the minions in various clouds & private networks making direct connections to the SaltStack Master, they now make connections to the local instance of OpenZiti running on the host & are transported automagically across the network to reach the Master – it really is Harry Potter style magic!
OpenZiti also adds other components of Zero Trust, including end-end encryption, micro-segmentation, and many other superpowers, that make it the solution of choice to protect any tooling that requires connectivity across multiple networks. While we applied this to SaltStack, it could easily be applied to any other distributed system and/or to secure configuration management tools – for example, at NetFoundry we made our Bastions Dark, Jenkins Invisible and directly embedded OpenZiti into Prometheus. Do we even care about zero days and CVEs if we make our servers dark?
This makes that feeling in my stomach finally settle down.
If you are already convinced, check out our free-forever tiers of NetFoundry or head over to our OpenZiti quickstarts to get started and secure your SaltStack installations.
If you want to go deeper on understanding how we implemented secure-by-design SaltStack, read technical blog next week.
