Javid Habibi
Staff Solutions Engineer
Platform Engineering
AWS
How the engineering discipline behind EC2's stability maps directly onto Crossplane composition architecture and what most platform teams get wrong on day one.
In 2017, AWS began one of the most ambitious infrastructure replacements in the history of cloud computing. The Nitro System moved virtualization off the host CPU and onto purpose-built hardware, replaced the Xen-based hypervisor with a lightweight KVM-based one, and re-architected how networking, storage, and security worked under every new EC2 instance.
The amazing part, your Terraform code didn't change. Your SDK calls didn't change. Your Instances kept doing what Instances have always done. The only applications that broke were those depending on undocumented behavior quirks of the old implementation that were never part of the contract.
This is the payoff of a deliberate engineering discipline that AWS has written about extensively: treat your API as a near-permanent contract, and never let the implementation leak through it.
If you're building an internal platform with Crossplane, you are in the API business whether you planned to be or not. Your XRDs are your Instances. Your application teams are your customers. The question that determines whether your platform earns trust or becomes a bottleneck is the same one AWS answered with Nitro: can you change everything underneath without your consumers noticing?
For most Upbound and Crossplane adopters today, the answer is no. This post is about why, and how to fix it.
The Failure Mode: Flat Compositions and Leaky Schemas
The most common composition pattern we see in the field are what we'd call a flat composition: a single composite resource that maps directly to many managed resources, with an XRD schema assembled by promoting provider fields upward. The XRD grows fields like dbInstanceClass, vpcCidrBlock, and assumeRolePolicyDocument not because anyone designed an API, but because those were the knobs the underlying provider CRDs happened to expose.
This works beautifully on day one. The trouble starts the first time the implementation needs to change:
A provider family migration renames fields or moves resources to new API groups and because those fields live in your XRD, your XRD has to change. A team needs a third availability zone, but the composition hardcodes a two-AZ subnet layout, requiring a schema change. Security wants to move from IRSA to EKS Pod Identity, another schema change. Every one of these ripples out to every team consuming the API.
Werner Vogels has a name for the root cause: the leaky implementation. His warning, delivered in the context of AWS's own API design, is that if customers can see how you've built something, they will start depending on how you've built it and at that moment you lose the ability to change it. A flat composition is a leaky implementation by construction. The XRD isn't an API it's a window into the plumbing.
What AWS Actually Did
AWS treats breaking changes as extraordinarily expensive but not impossible. S3's first-generation APIs from 2006 still work. But AWS has retired interfaces: the EC2 SOAP API was wound down, EC2-Classic with the original pre-VPC networking model was fully retired in 2022 after a multi-year migration with dedicated tooling and direct customer outreach. The lesson isn't "never break anything." It's that AWS prices breaking changes correctly: years of runway, migration tooling, and proactive communication. Most platform teams price them at zero, and their consumers pay the difference.
The contract and the implementation are strictly separated. The Nitro transition was possible because the EC2 API never exposed hypervisor details. The interface between the control plane and each host is narrow and well-defined. Nitro controllers sit in a listening state, receiving authenticated commands initiated by the external control plane so everything behind that interface could be rebuilt.
Change is shipped in containable units. Cell-based architectures, shuffle sharding, and zonal deployment calendars mean that when AWS changes internals, the blast radius of a bad change is bounded and rollback is cheap.
Every one of these maps onto composition design.
A Layered Architecture for Compositions
The structure that captures these lessons is three layers but only two API tiers. Layers 1 and 2 are contracts, Layer 1 is not an API at all. Hold onto that distinction, because this is not a call for three levels of XR nesting.
Layer 3 is your platform API. This is the XRD your application teams consume. It expresses the things your consumers actually vary, and nothing else. It changes rarely, and only additively. This is your Instances.
Layer 2 is your set of domain abstractions. These are XRDs for coherent infrastructure domains Network, Compute, Identity, DataStore each cloud-aware but provider version agnostic, each owned as an independent contract with its own lifecycle. Their status fields form the interface between domains: a Compute resource discovers subnets by reading the Network resource's published status, the same way AWS services discover each other through API responses rather than shared internal state.
Layer 1 is the implementation. These are the compositions and function pipelines behind each Layer 2 XRD and they are the only place provider managed resources and provider CRD fields appear. This layer is expected to change constantly. Provider migrations, field renames, new cloud features: all of it stops here, the way hypervisor changes stopped at the Nitro interface.
The test for whether a field belongs in an XRD, call it the Nitro test, is simple: if we swapped the implementation underneath, would this field still make sense? spec.size: large passes. spec.dbInstanceClass: db.r6g.xlarge fails; it belongs in Layer 1, where a function maps intent to the current right answer.
What This Looks Like in Practice
Consider the network domain, since it's where hardcoded implementations are most common. A typical flat approach bakes one topology into the composition a fixed CIDR, four subnets across two AZs, and any deviation means surgery. A Layer 2 Network XRD instead parameterizes intent:
Nothing in this schema names a provider CRD, an API group, or a field that only exists in one provider family. The composition function behind it iterates over the AZ count, computes subnet CIDRs, and emits the VPC, subnets, gateways, route tables, and security groups using whatever provider and API version is current. The XRD publishes status.networkId, status.privateSubnetIds, and friends for sibling domains to consume.
Notice natGateway: per-az. It's static stability surfaced as a consumer choice: one NAT gateway per zone means losing an AZ doesn't take out egress for the others. The platform encodes the principle, the consumer makes the cost/availability tradeoff explicitly.
Everything this Network resource provisions is a real cloud object that exists independently of Crossplane itself. If the control plane goes dark, reconciliation stops, meaning no drift correction and no new changes landing, but the VPC, subnets, and gateways keep doing their job, the same way a running EC2 instance outlives an impaired EC2 control plane. That survival isn't automatic, it's a design constraint. The discipline is to keep the Crossplane control plane out of the runtime critical path of what it provisions, and to be deliberate with deletion and management policies so a control plane disruption can't cascade into tearing down the data plane. It's also why a careful migration leans on managementPolicies: ["Observe"] to hand authority to a new implementation without ever putting live resources at the mercy of a reconcile.
Now let's look at the payoff scenario, your organization migrates from community providers to official provider families. This means different API groups, different field shapes, and different CRD structures. In the flat model, this is a breaking change to every XRD that leaked a provider field, which means claim migrations, consumer coordination, and a big-bang cutover. In the layered model, it's a Layer 1 change: new compositions, same XRDs, same XRs, consumers undisturbed. You can even run old and new implementations side by side during the transition, selected per resource, which is, almost exactly, how Xen and Nitro coexisted across instance families for years.
That is a platform team's Nitro moment, and it's the moment that earns lasting trust from application teams.
The Tradeoffs
This architecture is not free, and not heading it would repeat the mistake of the mythologized AWS story.
Nesting has real costs, and it is not community consensus. Every XR layer adds a reconciliation hop, deepens the crossplane trace output you'll debug through, and requires deliberate engineering for status propagation and cross-resource references. There is a credible school of thought in the Crossplane community that composition functions reduce the need for nesting at all, since a single function pipeline can encapsulate enormous complexity behind one XRD. Our guidance: functions first. Reach for a nested XR only at an ownership boundary, a domain with its own team, its own change cadence, or its own reuse story across multiple parent APIs. Nesting belongs at contract boundaries. Inside the plumbing, it's indirection, not abstraction.
Don't over-abstract Layer 3. The uncomfortable detail from inside the AWS story: most durable AWS APIs are the mid-level ones. EC2 and S3 expose plenty of implementation flavored concepts, instance types, availability zones, storage classes and have thrived for two decades. AWS's high-abstraction, intent-based offerings, like Elastic Beanstalk, stagnated when the abstraction couldn't keep pace with what users needed underneath it. The lesson for your Layer 3 is not "abstract everything to t-shirt sizes." It's "expose exactly what varies between your consumers, defend backward compatibility ruthlessly, and design a bounded escape hatch on purpose" because if you don't design the override mechanism, your power users will create one through pressure, and it will become the real API.
The discipline is organizational, not technical. Crossplane v2 makes this architecture markedly cleaner, namespaced composite and managed resources, no more claims, function pipelines as the standard composition model, and the ability to compose any Kubernetes resource. But no release flag enforces the contract boundary. The review-time question "does this field pass the Nitro test?" is a habit your platform team has to build, the same way "would this break a customer?" is a habit AWS had to build.
The Bigger Shift
AWS's durability teaches one thing above any individual pattern: a platform team operates an internal API program. Contracts are versioned, layers have independent lifecycles, deprecation follows deliberate policy, and implementations can evolve freely because the contracts stay stable.
This is the shift we're building for at Upbound: platforms whose contracts hold steady while AI agents and human engineers drive the same declarative APIs, and whose implementations stay free to change underneath both. We created Crossplane and remain among its core maintainers, and Modelplane, our open source control plane for AI inference, is built on the same principle.
Crossplane gives you the machinery for that program: XRDs as contracts, compositions as swappable implementations, functions as the translation layer, and a control plane that reconciles it all continuously. Whether your platform API survives its own implementation is a design choice and it's one you make in your very first XRD.
Choose the way AWS did.
About Authors
Javid Habibi
Staff Solutions Engineer
