The pattern repeated often enough that it became a running joke. A dashboard would surface a number, someone would ask the obvious follow-up question, and answering it meant writing a throwaway script against 24 control planes. Version drift exists, but does it matter and where? This composite is unhealthy, but which of its 56 children is actually broken? We have 230 resources out of sync, but how many distinct problems is that really?

With the addition of the Hub as part of Upbound v3, we now had an API endpoint that we could point our agent towards that had all of the same resources and data that Insights uses.  So we pointed Claude up to Upbound Hub and started asking. Hub is the centralized API that indexes and aggregates every control plane in the org, which makes it the one place where those cross-fleet questions are answerable at all. Claude talks to it through a skill that wraps the Hub API. The operator in the examples below holds nothing but a Hub token. There is no kubectl context into any individual control plane. Everything below comes out of the aggregated API.

What follows is our own internal dev fleet. Control plane names and other data have been obfuscated.  We keep a certain percentage of resources scrubbed in dev to enable testing.

Package drift

Upbound Insights flagged provider version drift immediately but that was the easy part.  We needed to understand which control planes are behind, by how much, and whether the gap is a routine patch bump or a breaking migration.

Claude pulled crossplanepackages and walked the distribution subresource for each package reporting more than one version. Eight of our 62 packages had drifted. provider-family-aws was the one worth stopping on:

Four versions across four control planes, spanning a major release boundary. provider-aws-ec2 mirrored it almost exactly at v1.2.1, v1.23.2, and v2.2.0. The useful finding was not the count, it was the ordering: dev/app-platform is the consistent laggard on v1.2.1 for both packages, and the v1 to v2 family jump is breaking rather than additive. That flips the upgrade plan. Unify provider-aws-iam first because most other AWS packages depend on it, bring the laggard forward through v1 to a common baseline, and sequence the v2 family migration last as its own piece of work rather than folding it into routine patching.

Because Hub does not scan anything (today), it has no vulnerability data, no severity field, no SBOM. Advisories reach us the way they reach everyone, from our own internal scanning, our Marketplace, a GitHub advisory, or an upstream release note, and they name a repository and a range of affected versions. What has always been slow is the next step, which is working out our actual exposure. Given a repo and a version range, Claude answers which control planes are running it, at what digest, and how widely, from usageCount and deploymentStatus in the image catalog.

The catalog also caught something we would not have gone looking for. The same image digest is published under two repository names, upbound/provider-helm and crossplane-contrib/provider-helm, and both appear in observedRefs. Two names, one artifact. If you map exposure by tag you will either double count it or miss half of it.

A 56-node graph with four broken nodes

Upbound Insights showed an XCore composite unhealthy in dev/app-platform. Its own status message said what these messages usually say, that it was waiting on composed resources. Useless.

Claude walked resourcerelationshiptrees down from the composite. The tree came back with 56 nodes, of which four were broken. Two Release resources had failed with update failed: failed to upgrade release: another operation (install/upgrade/rollback) is in progress, a stuck Helm lock rather than anything wrong with the chart or its values. Two Alert resources were unsynced behind them. Of the remaining 52, forty three reported healthy and nine reported nothing at all, those being the ProviderConfigUsage leaves that never carry conditions.

Finding four bad nodes in 56 by hand means expanding a tree in the console or dumping YAML and reading conditions until something looks wrong. Getting it as a ranked answer took one prompt and the fix was easy: clear the two Helm locks, and the alerts reconcile on their own.

The more instructive case was a different composite. XEKS/my-eks-xr in staging/cloud-v2 has only eight nodes, and every one of them was unsynced. Its root condition:

Nothing is wrong with any of those eight resources. An informer that will not sync is a provider and CRD layer problem, and the composite is just the place you happen to notice it. The shape of the graph is the diagnosis. Four broken nodes out of 56 points you at specific leaves; eight out of eight points you underneath the composite entirely. And staging/cloud-v2 is one of the drift laggards from the section above, sitting on provider-family-aws v1.23.2, which is where we started looking and where the actual fix landed.

230 failures, 11 causes

The last one is the case that changed how we triage. Insights reported 230 resources with Synced=False across the fleet. As a number it is meaningless, because it says nothing about whether that is 230 problems or a handful.

Claude pulled every unsynced resource with full status and grouped them by what their condition messages actually said. 181 of the 230, a little under 80 percent, traced to eleven missing Secrets. One dead credential per torn down space, each one taking down everything that referenced it.

By kind it was 173 Object, 26 Release, six Alert, and a short tail of AWS managed resources, which is to say almost all of it was provider-kubernetes and provider-helm unable to connect rather than any cloud API rejecting us. By control plane, 196 sat in dev/app-platform.

Eleven fixes clear roughly 80 percent of everything broken in the fleet, and one of them clears 61 resources on its own. That reframes the work from a 230 item backlog into a ranked list a single person finishes in an afternoon: recreate the credentials for spaces that are still live, and delete the orphaned ProviderConfigs and Objects pointing at spaces that are genuinely gone. We started with aws-dev-sandbox and watched the fleet number drop by 61.

No dashboard was going to hand us that. The grouping key was buried in text, in eleven variations of a message that reads cannot get credentials secret: Secret "X" not found, spread across 24 control planes.

What actually made this work

Upbound Insights is the signal and Hub is where you resolve it. That division held across everything and it is a more useful way to think about Insights and the Hub, rather than simply a dashboard.

Three things had to be true. Hub had to aggregate, so the question could be asked once instead of 24 times. It had to expose the relationship graph, because a composite’s own status message is almost never the answer. And the failures had to be legible as text, which is the part we did not expect to matter most. Grouping 181 failures by a secret name embedded in a condition message is not a query you can write, and it is the single biggest reason this felt different from the scripts we used to write.

If you are running more than a handful of control planes, the drift audit is the cheapest place to start. It took one prompt and it changed our upgrade order.