By Philip Griffiths and Kenneth Bingham
The Rise of the Supercloud
In December 2021 Dave Vellante wrote in siliconANGLE about “The Rise of the Supercloud”.
Supercloud describes an architecture (for companies like Snowflake and Databricks) that taps the underlying services and primitives of public cloud providers to deliver additional value above and beyond what’s available from those public cloud providers. A supercloud delivers capabilities through software, consumed as services, and can run on a single public cloud or span multiple clouds.
In this 3-part series we will learn that supercloud providers have challenges securing their data cloud, how multi-cloud networking is more difficult than it needs to be, and finally how supercloud providers can now control their own multi-cloud-native network fabrics to gain new superpowers of cloud-independent control and security. This will deliver Supercloud providers and their customers with better cost structures and revenue, far stronger security and greater end-to-end control independent of underlay networks and underlay clouds.
Prominent examples of superclouds include Databricks, DataRobot, HashiCorp, Confluent, and Snowflake. The facts of their widespread adoption and realized value are evident in each of these examples. There is however an important, unsolved problem with superclouds. In the absence of a solution to this problem, the direct and indirect consequences are inevitably passed along to the end-user. Superclouds are deeply, logically integrated with the underlying cloud infrastructure on which they are built and do not control. The problem is that the realms of responsibility between the supercloud operator and their customers are overlapping in such a way that it’s impossible to achieve a clear separation of concerns, so both supercloud operator and customer end up with expensive dependencies on clouds and telcos which compromise security, agility and control.
Dave’s follow-up article “Securing your Snowflake cloud data” maps this problem out nicely. As an example, Snowflake’s supercloud has “a shared responsibility model between Snowflake and its customers when it comes to fully securing their data cloud”. This model is a three-layered security approach – network, identity access management, and encryption. The customer has the responsibility to provision and manage fundamentals of infrastructure including DNS, public access controls, inter-cloud connectivity (e.g., MPLS, VPN, raw internet) as well as a private link to the cloud environment. This hurts the supercloud provider and the customer – costs are high, security is low and agility is compromised. Superclouds are inherently distributed systems with central control, but they do not control the network or how to securely interconnect the distributed system. Ben Herzberg surmises, “complexity is the enemy of security and having those multi-cloud operations, from a security perspective, definitely adds complexity, which adds risks, so simplifying that is really, really helpful”.
The Lost Opportunities
This problem profoundly impacts multiple dimensions of security (e.g. confidentiality, integrity, availability), but even that does not encompass the entire realm of effect. Viewed through the prism of business benefits, there is the prize of greater adoption, realized value, and minimized costs. Today’s superclouds writ large burden their customers with setting up security and connectivity for superclouds, all of which represent costs and risks for their customers. For example, all the services mentioned above are monetized by the cloud providers or other 3rd parties including telecom companies. This is a lost revenue opportunity as well as slowing down time to revenue for superclouds. They also incur greater costs from supporting their customers to fix issues that are outside of their control. For superclouds, this is a lost opportunity to create the most value for their customers.
The Network is the Computer
John Gage of Sun Microsystems once coined the term, “The network is the computer”. What if superclouds could specify the network to the application instead of relying on all these problematic, shared systems and network infrastructure that is external to the application? This would allow their applications to be multi-cloud native, secure-by-design, develop-once-deploy-anywhere (be it in hyperscalers or at the edge and in IoT). They would allow their customers to massively reduce complexity and deliver value faster to customers while providing greater revenue and lower costs for superclouds to benefit shareholders.
Supercloud innovators would get superpowers by controlling their networks and making them secure-by-design, programmable, API-first, automation-friendly; but that does not sound like the description of a network. In part 2 we will examine the problem with traditional network approaches which has led to the current shared responsibility model for superclouds.
Ken is crafting developer experiences with the NetFoundry API and OpenZiti. He is enthusiastic about Linux, security, and building things with free and open source software and hardware. You can find him in his native habitat talking and clowning around at tiny tech events all over.
