I gave a talk in 2018 called Test Driven Infrastructure for Highly Performing DevOps/SRE Teams. Don't watch it unless you have an interest in ancient artifacts.

It was the Puppet and Chef era, and the tooling existed. Fast checks ran offline in seconds. Slow ones booted a real machine, applied your code and tested what came out. (Test Kitchen and serverspec, if you were there.)

The fast ones could not tell you much. Puppet would work out the full list of resources it was going to apply, and you could inspect that list without touching a machine - but the step from your code to that list is thin, so checking it mostly restated what you had already written. We used it to decide which slow tests a change needed, not to learn whether the result was right. Anything you actually doubted meant provisioning something real: one edit required one boot-and-check cycle before you learned anything, and a red result that could as easily be the environment as your code. So infrastructure testing became end-to-end testing, and the loop was never short enough to drive design. You wrote the code, then you found out.

What I wanted then was a fast tier worth trusting, so the slow tier could go back to being a gate rather than the only real test. What was missing was not another testing framework. It was meaningful output from my own logic, produced and asserted before any cloud API was called. The idea was never missing. The fast feedback loop was.

It exists now, and the rest of this post is what it looks like in practice: Crossplane composition functions for the logic, and Upbound's developer experience for building and testing them - the up CLI, a project structure that generates typed models from the providers' own schemas, and a test runner that renders the pipeline offline.

Test-driven development is not "we have tests". It is a loop: write a failing test, make it pass, clean up, repeat. The loop only works if the test has four properties.

1. It fails first, for the reason you expect. A test that has never failed proves nothing.

2. It runs fast enough to stay in the loop. A minute is already too long to keep the habit.

3. It is deterministic. Same input, same result, every time.

4. It asserts what the code produced: the rendered output itself, not the calls it would have made along the way.

Infrastructure has historically failed on the second and fourth at the same time, and the two are linked. What the code produces, in the end, is a VPC or a cluster or a database, and that only exists once something has been provisioned: minutes, sometimes hours, and a bill. So infrastructure testing tends to land somewhere adjacent:

• Plan-based assertions (Terraform's terraform test) are a genuine fast tier: assert against a plan without creating infrastructure. What bounds them is what a plan knows - computed attributes are unknown until apply - and running without credentials means mocking the provider, whose generated values, by HashiCorp's own documentation, rarely match what a real one returns.

• Provision-and-poke (Terratest and friends) asserts the real thing, at minutes to hours per run against a billable account. That is an integration test, and it belongs in CI rather than an inner loop.

• Static policy checks (OPA, conftest) are fast and deterministic, and assert the input you wrote rather than the output your logic produced. Useful, different job.

• Unit tests with provider mocks (Pulumi) get the speed back honestly, but you run your program against a mocked engine and the computed values come from the mock. Fidelity is bounded by the mock.

• Template synthesis (AWS CDK) runs your code for real: synth emits a template offline and the assertions module checks it, with no mocks and no substituted engine. You are asserting a document CloudFormation interprets later, not the objects that get applied.

None of these is wrong. Each trades something away: the real engine, every cloud but one, or the speed.

What changes with a declarative boundary

Here is the part that makes the fast tier worth having now, and it is two properties rather than one.

First, a composition function is code, and there is real logic in it. A composite resource (XR) - the API your platform team defines - might take a subnets list, and each entry in it becomes a subnet, a route table, a route and an association. One word, size: medium, becomes a node count, an instance type, a storage size and a backup window. Four resources from one list entry, four settings from one word: the output is the product of that logic rather than a copy of the input, so asserting it is a real check on what the code decided. That is not automatic. Where the step from input to output is thin, an assertion on the output mostly repeats what you already wrote - the Puppet case above, and the policy checks in the list before it.

Second, and this is the unusual one: the code does not call a cloud SDK. It returns data - a set of Kubernetes API objects describing desired state, which a controller applies later, separately. Pulumi is code too, and its unit tests can assert the resources a program declares - but only with the engine replaced by mocks, and with the computed values invented by those mocks. Terraform's plan is inspectable, but it is a prediction of what a provider will do rather than the product of your own logic. Crossplane moves that boundary back, and what your logic produces is an artifact that holds still long enough to inspect.

One more difference is easy to miss. The tools above test a deployment: code runs, resources are created, the run ends. A Crossplane composition never ends - it is the logic a controller runs on every reconcile, so the objects you assert offline are the ones a control plane will keep asserting against drift long after the change merged.

That is what makes a fast test meaningful, because you can run the real function, against the real provider schemas, and inspect the real output, without touching a cloud. Upbound's up CLI runs it:

up test run tests/test-network/

Nothing is mocked in that command. It runs your function in a container, renders the composition pipeline the way Crossplane would, and compares the resulting objects against what you asserted. The output it checks is the same data that would have been applied.

The loop, concretely

Start with the assertion. Here the VPC does not exist yet - no function code has been written:

vpc = vpcv1beta1.VPC(

    apiVersion="ec2.aws.m.upbound.io/v1beta1",

    kind="VPC",

    metadata=k8s.ObjectMeta(name="vpc-example-network", namespace=NAMESPACE),

    spec=vpcv1beta1.Spec(

        forProvider=vpcv1beta1.ForProvider(region=REGION, cidrBlock="10.0.0.0/16"),

    ),

)

Run it and read the failure:

ERROR: no actual resource found: ec2.aws.m.upbound.io/v1beta1/VPC/vpc-example-network

🔴 That is the first property: it failed, and it named exactly what is missing. Then the function:

desired_vpc = vpcv1beta1.VPC(

    metadata=k8s.ObjectMeta(name=f"vpc-{xr_name}", namespace=namespace),

    spec=vpcv1beta1.Spec(

        forProvider=vpcv1beta1.ForProvider(

            region=observed_xr.spec.region,

            cidrBlock=observed_xr.spec.cidr,

        ),

        providerConfigRef=vpcv1beta1.ProviderConfigRef(

            kind="ClusterProviderConfig", name="aws-provider",

        ),

    ),

)

resource.update(rsp.desired.resources["vpc"], desired_vpc)

🟢 Green, offline, with no AWS credentials involved. Two details make this higher fidelity than it looks. The types come from the provider's own CRD schemas, generated into Python models, so a field that does not exist fails at construction time rather than in a cloud API response. And the namespace, providerConfigRef and management-policy semantics are the real ones, because the renderer is the real renderer.

When you do want the cloud's opinion, the same manifests go to a real control plane:

up test run tests/e2etest-network/ --e2e

That one provisions real infrastructure and bills you for it, and it answers a different question: does AWS accept this? up test run rebuilds your function and renders, so the fast tier is tens of seconds. The full platform end-to-end - VPC, EKS, RDS, an application - takes about half an hour, most of it spent waiting for the EKS cluster and its node group. Two orders of magnitude is enough to change how you work. Those are not competing tests. They are different tiers, and the point of the fast tier is that you almost never need to guess in the slow one.

Day-two work gets the same loop

Everything so far has been provisioning. The interesting question is whether the loop extends to operational work - the tasks that happen once, or on a schedule, rather than being reconciled forever.

Crossplane's Operation, CronOperation and WatchOperation run function pipelines to completion - once, on a schedule, or when a watched resource changes - and they are testable the same way:

up test run tests/operationtest-snapshot-db/ --operation

The test supplies the resources the operation asks for, runs the pipeline, and asserts what it composed - in our case an RDS snapshot named from the database's real AWS identifier. Red, green, no cloud. "Back up before the upgrade" becomes a thing you can test rather than a runbook step you hope someone follows.

What it changes

The honest summary is not that infrastructure TDD became easy. It is that the loop became short enough to be a habit, because the thing your code produces is data you can inspect immediately.

In practice the ordering inverts. You write the assertion for the API you want, watch it fail with a message naming what is missing, then write the composition until it passes. Mistakes cost seconds instead of a provisioning cycle. The expensive end-to-end run stops being where you debug and becomes what it should be: a gate that confirms the cloud agrees.

This matters more, not less, when part of the implementation is generated. AI made infrastructure generation cheap. It did not make incorrect infrastructure cheap. A fast, deterministic test is an executable contract that a human or an agent has to satisfy before anything reaches a cloud API, and a failing test with a message naming what is missing is a better brief than a prose description of the intent.

That is ordinary test-driven development. The notable part is not the practice, which we were all attempting in 2018. It is that the fast tier finally has something substantial to assert - the output of real logic, rendered by the real engine - so the loop is short enough to drive design instead of confirming it afterwards.

Where the real examples are

This comes out of the day job at Upbound: building reliable abstractions for customers, teaching the workshops where they learn to build their own, and maintaining the reference platforms. Every technical claim above came from that work.

If you would rather run the loop than read it, Upbound's Builder's Workshop is the guided version: six chapters from an empty project to a pushed configuration, with the composition test as chapter three.

The closest thing to this post as running code is configuration-azure-network. It is Python, it is public, and it is the same shape as the example above: one list in the XR spec, each entry becoming a subnet and its delegation. The function is under 200 lines. The composition test beside it is over 600, covering four variants - a Postgres database subnet, a MySQL one, none at all, and several at once - and tests/e2etest-network-py provisions the same thing for real.

Both numbers are the argument in miniature. One of those four variants asserts that an empty list produces nothing extra - the branch least likely to break, and the one nobody writes when a test costs a provisioning cycle. Speed is not the only thing a cheap test buys: coverage reaches branches an expensive tier could never justify.

The reference platforms show the same structure at platform scale, in KCL rather than Python: platform-ref-aws, platform-ref-azure and platform-ref-gcp each keep tests/test-cluster and tests/e2etest-cluster side by side. The language is a choice rather than a constraint: up test generate scaffolds in KCL, Python, Go, Go templates or plain YAML for a purely declarative test, and up test run runs whatever comes out. The fast tier is a property of the boundary, not of a language binding.