Transparent Bastions

Our Zero Trust Journey (ep.2) :
Transparent Bastions

How can we eliminate SSH proxy configurations and improve security?

Welcome to episode 2 of our Zero Trust Journey! Our point of arrival will be applying the principle of zero trust to our existing platform infrastructure. When we’re done things like production databases and SSH servers will only be reachable on the OpenZiti Network. This eliminates network exposure to anonymous clients both public and private.

In case you missed episode 1 (https://netfoundry.io/bastion-dark-mode/), we started with a typical defensive strategy. Our bastion host was acting as an SSH jump box and SSH proxy to protect a private zone from direct internet exposure. We applied OpenZiti to make the SSH bastion host “dark” on all networks and we were able to keep working normally. We call this Bastion Dark Mode.

An SSH bastion addressable only via OpenZiti

In this episode, we’ll continue the bastion pattern: a host with special network access depicted in the diagrams as being behind the firewall. We’ll remove the dark SSH bastion from our OpenZiti overlay and use an OpenZiti Router that is logically transparent to the client. 

Dark bastions are great, but transparent bastions are even more flexible and much more convenient. Dark bastions allowed us to continue using our existing SSH proxy configurations without interrupting business as usual. This was important for the transition to dark bastions. After making our bastions dark, we realized the security provided by those SSH proxies would be made entirely redundant if we were to apply OpenZiti to the connection! So that’s what we set out to do. 

Transparent bastions allow us to connect to our protected resources without a client proxy configuration: no more SSH jump box. Additionally, there are certain problems that dark bastions never solved. For example, How can we send GitHub webhooks to our private Jenkins server? It just didn’t make sense to send the webhooks over SSH through a dark SSH bastion, so we used a GitHub Action built with OpenZiti’s NodeJS SDK to send the webhooks to Jenkins. You can read about how that works in another post (https://netfoundry.io/this-is-the-way-invisible-jenkins/).

Let’s Talk Proxies

When it comes to proxies there are two types, forward and reverse. A forward proxy is considered opaque because it requires the client to know the URL of the proxy in order to access the target resource. A reverse proxy is considered transparent. A transparent proxy does not require the client to be configured at all, so the user may not realize they are using a proxy. 

When you say “proxy” out of context I would probably guess you’re talking about an opaque forward proxy, and an SSH bastion, i.e. jump box, is one example. Using a typical (opaque forward) HTTP proxy with your browser requires you to configure it with the URL of the proxy. In that sense, your browser is aware of the proxy and “sees” all requests and responses handled through that proxy, not the web server itself. It only sees the proxy, so the proxy is opaque. All requests are sent to the proxy which then selectively forwards the request to the destination. 

A reverse proxy on the other hand is positioned as a transparent receiver in front of the application server and clients don’t need any special configuration. A load balancer is an example of a transparent reverse proxy. The client only “sees” the web server, so the proxy is transparent.

A proxy is a means to an end, and a transparent proxy means you’ve eliminated a step because it just works without a special client configuration. You might even forget you’re connecting through a transparent proxy. This is great for the bastion use case because the user knows what they’re trying to connect to and doesn’t care how they get there.

Using the Transparent Bastion

From my perspective as a user, I don’t need to know anything about the transparent bastion to use it effectively. To start using the transparent bastion seamlessly I just need to stop using the opaque SSH bastion! 

Let’s say I’m using the laptop device shown on the left side of the diagram. In the episode 1 configuration, I used an SSH jump box configuration to reach our resources. An example configuration for a resource that is an SQL server is to save the SSH bastion’s domain name and my proxy username and private key file path into my SQL client application.

My OpenZiti badge there represents my tunneler which is the OpenZiti agent on my computer. I’ve been issued an identity for that tunneler which I have loaded. With that one step completed any application on my computer has access to the resources shown on the right side of the diagram. I no longer need an SSH jump box or proxy of any kind. The bastion is shown in the diagram below, but I don’t see it as a user because it’s part of the OpenZiti overlay which is transparent from one edge to the other.

A transparent bastion making a few apps directly addressable via OpenZiti

Setting up OpenZiti Services

A prerequisite for a direct, transparent connection is an OpenZiti service that specifies the destination. For now, we will be creating the few services we have manually. We plan to use the NetFoundry API to automate setting up these services in the future when we have a less static set of resources. 

Fortunately, it’s a one-time cost for each destination. When I want to SSH to a particular host or connect to a particular SQL server then I need to take a one-time administrative step in the NetFoundry web console to specify that connection. Here’s one example of specifying a production database server as a Ziti service.

Specifying an OpenZiti service in the NetFoundry web console

1. Apply some hashtag role attributes so that my new service aligns with the existing service policy for production databases.
2. Define a domain name and port pair that clients will use to connect to this service. This could be a fictitious name or the real name of the server.
3. Select the bastion host that has access to the application server we’re specifying.
4. Define the real domain name and port of the application server from the perspective of the bastion host where ziti-router is running.

Reference: Support Hub article about creating services and role attributes (https://support.netfoundry.io/hc/en-us/articles/360045503311-Create-and-Manage-Services) 

Going Further with OpenZiti

Transparent bastions are powerful and flexible, but there’s still a fundamental weakness in the bastion pattern. A bastion is a castle and once the enemy is inside the walls they can attack vulnerable resources directly. In our case, this could happen if the device where ziti-router software is running were compromised. The vulnerability exists in the leg of the journey between the OpenZiti router and the protected resources behind the firewall. That segment is protected only by the firewall, not OpenZiti. More complete adoption of zero trust will shrinkwrap the lines of defense around the defended resources so that it’s no longer possible to breach the firewall that creates our security zone and gain broad access to everything inside.

The next stop in our journey will be to extend the edge of the OpenZiti network to the protected resources. Then the OpenZiti connection will terminate on the same device where the resource is located, or inside a more narrow slice of the network if co-residency on the device is not practical as is the case for some cloud provider services where the endpoint is not a “device”, per se. We’ll need to install OpenZiti software and enroll an identity for each resource e.g. each SSH server, which will certainly entail some new automation with the NetFoundry API.