For two weeks, our engineers thought the open model behind their coding agents wasn’t ready. It made up tool names, dropped required fields and wandered through long reasoning loops. Then one of our pilot users suggested a test almost too simple to be useful: ask the model to copy back 50 names. On one serving engine it managed that correctly only 30% of the time. On another, using the same model weights and GPUs, it got every name right. The problem was not the model. It was the software around it.

That was one of the more memorable moments in Project Champagne, our effort to run real internal workloads on Modelplane rather than ask others to depend on something we hadn’t operated ourselves. It also gets to the reason Upbound has submitted Modelplane to CNCF for consideration as a Sandbox project: running inference reliably is a systems problem, and we think the operating layer for that systems problem needs to be built in the open.

AI infrastructure is becoming more horizontal. Models, serving engines, schedulers, accelerators and infrastructure providers evolve on different timelines, while GPU capacity remains scarce and expensive. Different workloads need different combinations; a coding agent with a long conversation has little in common with an interactive chat endpoint or a security scan in a build pipeline. The difficult part is no longer just standing up one cluster and getting one model to answer. It is keeping those combinations working across a heterogeneous fleet as capacity, traffic and software change.

That’s a familiar transition for platform engineers. Upbound created Crossplane so teams could manage resources across clouds through declarative APIs. Modelplane applies the same idea to inference: platform teams describe the models and infrastructure they need, and the control plane coordinates deployments, serving technology and GPU capacity across a fleet of Kubernetes clusters. Application teams can use a stable inference endpoint while the systems behind it evolve. The work underneath is still done by the model servers, schedulers, accelerators and providers that specialize in it.

An open ecosystem is the point

We saw the breadth of that work at Inference Day on November 5. Rather than put one company on stage to explain the whole inference stack, the event brought together the people building different pieces of it. Kelsey Hightower opened a conversation about what platform teams should build - and which cloud-native lessons they should carry forward. The panels then tackled three distinct problems: serving models at scale, building the platforms underneath, and operating inference in production. Viktor Farcic closed with an Inference 101 session for engineers new to the field.

Those conversations spanned maintainers and contributors from NVIDIA Dynamo, KServe, Envoy AI Gateway, Kubernetes, Metal3, Crossplane and Modelplane, alongside engineers focused on production telemetry and operations. That range is the point. A project focused on scheduling should not have to become a model server, and an accelerator provider should not have to own the enterprise control plane. The interesting work is making these independent pieces useful together at the fleet level. We also published an Open Field Guide to Inference Engineering, drawing on practices across the community and the lessons from Champagne, so platform engineers inheriting this work have a place to start.

Inference Day: modelplane.ai/inference-day

Field guide: docs.modelplane.ai/guides/inference-guide

What this looked like on stage

At KubeCon + CloudNativeCon North America, Bassam walked through that idea with infrastructure from several providers. The keynote showed NVIDIA GPUs on AWS and AMD GPUs on Vultr, with a single endpoint in front of two model deployments. As the scenario evolved and an AWS reservation approached expiration, CoreWeave was added and the AWS deployment was drained. The point wasn’t that every workload should be moved around constantly; it was that an application shouldn’t need to be rewritten every time the infrastructure behind its inference endpoint changes.

The same principle applies within the serving stack. One example is Modelplane’s integration with NVIDIA Dynamo, which brings together distributed inference, multi-node workload coordination using Grove and the KAI Scheduler, and model-weight distribution through ModelExpress. It’s one of the ways specialized open source technologies can work within a common fleet-level operating model rather than becoming a separate infrastructure stack.

Learning by running our own inference

Champagne began with three very different workloads: chat, coding agents and a build-security agent that finds and fixes CVEs. They forced us to learn the details. Our first-choice GPU instance never appeared despite approved quota. Another shape was refused in multiple availability zones. Some model responses streamed valid tokens and returned HTTP 200 even though the answer had been cut off. Credentials that worked for a short human interaction didn’t necessarily survive a long-running agent session. A dashboard could say a resource was healthy while the underlying service wasn’t. None of those problems could be solved by picking a different frontier model.

The security agent gave us another useful lesson. When we compared open models with a hosted reference on actual vulnerability-fixing tasks, the models could identify known CVEs, but a plan that looked plausible could still recommend a version that didn’t fix the vulnerability. We had to test the entire workflow, the grader and the infrastructure before deciding whether the model had failed. For coding agents, the hard question became how to grow beyond a small pilot without making the experience worse. Testing the equivalent of a 15-engineer team showed that two GPU nodes weren’t sufficient; four nodes with a different serving configuration could meet the test. Even apparently light users could produce heavy concurrency through subagents.

We didn’t keep those discoveries private. The six-part Drinking Our Own Champagne series records the failures, experiments and trade-offs, and the fixes helped shape upstream work on gateway authentication, configurable timeouts, observability and scaling. The series is complete, but the feedback loop is not. We run the software, find what breaks, publish what we learn and make improvements that other users can build on.

Why a neutral home matters

Inference Day and Project Champagne have pushed us toward the same conclusion from different directions. The ecosystem needs people who are excellent at their own layers, but it also needs a common way to operate those layers together. And it needs shared knowledge about the unglamorous parts - capacity, identity, observability, reliability and cost - that determine whether inference is actually useful. An open fleet-level control plane gives those communities somewhere to connect their work without asking any one project to solve the entire problem.

Modelplane remains available under the Apache 2.0 license. Our submission for consideration as a CNCF Sandbox project is an invitation to build that common layer with a broader community of maintainers, infrastructure providers and enterprise users. Acceptance is not yet decided. In the meantime, we’ll keep doing what Champagne taught us to do: run it, test it, share the results and make the project better in the open.

Explore Modelplane: github.com/modelplaneai/modelplane

Documentation: docs.modelplane.ai

Project Champagne series: Drinking Our Own Champagne, Part 6