The bug that looked like a bad model

In Part 1, I admitted an uncomfortable truth: we had a technology for self-hosting inference, and we weren't running inference ourselves. Seven weeks, hundreds of changes, and five posts later, our engineers' coding agents run on open models we serve ourselves with Modelplane. This is the post where we step back and look at the big picture. 

Let's start with the issue that summarizes our entire journey. For about two weeks, the coding agents our pilot engineers pointed at, GLM-5.3, misbehaved in ways that looked exactly like a model that wasn't ready. They invented enum values that didn't exist, dropped required fields from tool calls, mangled tool names, and burned hundreds of thousands of tokens reasoning in circles. Every report in our pilot slack pointed the same way: open models weren’t ready.

One of our pilots then proposed running the simplest test we could think of. Ask the model to copy a list of 50 names back exactly. On vLLM it got it right 30% of the time. On SGLang, with the same weights on the same GPUs, it got it right 100%. The serving stack had been corrupting its context the whole time, and we're still pinning down exactly where with the vLLM maintainers.

That story is this whole series at a glance. When something looked like a model failure, it was usually the engine, the gateway, the network, the auth, the observability, or the agent harness around the model. Standing up inference was easy. Testing, validation, and Operating it was hard! It's the job that most platform teams are about to inherit.

We've been calling the person who does that job the Inference Engineer. The operator who picks a model, gets it onto the right GPUs, keeps it serving, tunes it, and gets paged when it falls over. It's SRE for a new kind of workload, and almost everything we learned is about what that role really involves were lessons we’d already knew from the past.

Here's what the last post in this series covers:

• Where our problems actually came from.

• A release process against a moving target.

• Governance and who owns what

• How we measured and where things stand for our use cases.

• What we got wrong and the receipts. 

• What changes when everyone is using your inference.

The scoreboard

We uncovered more than just these issues, but because YOLO we found ourselves early on in a rapid-iteration cycle of discovering issues and then just submitting PRs and patching them in realtime. The overall trend is still obvious, the model was very rarely the problem - it was mostly everything around it.

Provisioning and the control plane lead by a wide margin: node groups that reported failed while their instances kept launching, cache disks that quietly disappeared, rollouts that deadlocked on exclusive GPUs, and resources that reported Ready while the cloud said otherwise. Observability, auth, the serving engine, and the gateway follow in a tight pack. Our own evals and test rig were wrong about as often as the gateway was.

The model bar is the shortest, and even there, half of the six were the model needing more time or more context than our infrastructure allowed it. Almost none of these failures threw an error: a truncated stream returned 200, a metrics API reported AVAILABLE against an empty database, and a node group reported Ready while it was failing.

These are our counts on our serving stack, and another team would probably find a different mix but we suspect the trends would be similar. Anyone moving from a frontier model takes on all of these layers at once, which is why "the model is bad" should be your last guess, not the first.

The Inference Engineer runs a release process against a moving target

Everything we know about production and reliability engineering still applies. What's different is the thing being changed: a model, an engine build, and a set of serving flags, a harness & config, the list goes on. They all change constantly and serving can exhibit wildly different behavior when tweaked in subtle ways on any one of these components. The truth is that the version matrix has multiple axes and models will behave differently against any of them coupled with an organization's use cases. The right combination is yours to find and yours to keep finding. 

Here's an example of just what two days of change churn showed us, and this was just with one tweak around speculative decoding: 

• Speculative decoding had been breaking tool calls under load, so we turned it off. 

• A clean eval of 256 tool calls with 32 in flight, zero malformed, let us turn it back on. 

• A load test then showed it shaving first-token time at the 90th percentile from 4.4 to 4.1 seconds while dropping the slowest 10% of streams from about 13.6 to about 10 tokens a second, so we turned it off again. 

• We realized our load replay used placeholder tokens that speculative decoding can't guess well, and turned it back on.

Four flips in 48 hours and it took more time to decide if we should ultimately enable or disable it. That was just one engine parameter.  This churn also shed light on the impact we were having on live users, as our testing ran alongside our pilot’s sessions:

• Your test rig can cause the incident: One night our replayed load left the engine full of abandoned requests, and real users waited in line behind work nobody was reading. Now every request has a cap, a run isn't finished until the engine is back where it started, and candidates run in a staging slot beside the live recipe instead of replacing it.

• Churn costs trust: During a week of daytime experiments, an engineer told us they'd gone back to Frontier Models rather than risk the downtime. Reliability for a tool people use all day includes predictability, so experiments moved to staging slots and announced windows, each with a posted rollback time.

The practice we landed on looks like any good release process. Every serving change is a pull request. A CI job gives each candidate a fixed window, a written bar for latency, errors and evals, and a load test, then promotes it or rolls it back on its own. Alerting watches the live recipe: on its first real test it flagged bad output in about three minutes, before anyone in the pilot channel noticed. Whatever answer we discover today is a snapshot and the process for finding the next answer is the thing that lasts.

Governance is the real work, so here's who owns what

Modelplane served traffic about twenty minutes after install. Getting it serving in our environment with all our governance constraints took much longer: IAM, networking, security, observability, change management, testing & evaluation, productionization, the list goes on.. Getting a serving stack running is easy. Getting it running inside an enterprise, with every governance constraint that comes with it, not so much.

We told the war stories in Part 2 and Part 4: an endpoint that answered the open internet in 74 milliseconds, monitoring that sat pending for six days while reporting healthy, a door left open for two and a half minutes during a teardown, and a revoked key that could still list every model. Here’s what we wish we had on day one:

Every expensive governance failure was silent. The system told us everything was fine while users lost the end of their answers, while an endpoint sat open, and while monitoring had never once run. If there's one thing I'd tell the next platform team, it's that serving the model is the easy part. You'll spend your time on TLS, identity, timeouts, storage classes, IAM, DNS, and observability: ordinary platform problems showing up in unfamiliar combinations.

This is also where a control plane shines. Most of the first column has already moved into the second. Anything that turns these silent failures into loud ones, and keeps the whole lifecycle declarative, is going to save you a lot of sleepless nights.

Baselines, use cases, and the verdict

You can't tell whether self-hosting is working unless you know how your organization uses AI today. Decide which use cases you're serving, write the evals, and measure today's usage before you cut anything over. You will iterate many times and the baseline is what keeps everyone honest.

We started from three use cases and ignored the hardware (Part 1) as each use-case had a different set of constraints and needed a different kind of measurement and evaluation:

• CVE scanning and backporting: Deterministic graders that check the end state, because the scanner either still reports the vulnerability or it doesn't (Part 3). The bar came from history: about 85% of the fixes our engineers merged went in without edits, so that's the number an open model has to match, not 100%.

• Agentic coding: Replayed agent traffic against a bar set relative to the current recipe, plus evals run during the load test, because tool calls that pass one at a time can break under concurrency (Part 5).

• Interactive use: Structured feedback from the people using it every day, plus usage data from the coding agents themselves.

That baseline mattered more than we expected, because we changed models far more often than we planned: Qwen3-Coder and GLM-4.7-Flash, then MiMo-V2.6-Flash and Qwen3.8, then a larger Qwen3, then GLM-5.3 at 4-bit. Along the way, a recall failure that looked like the model turned out to be our serving config. Without a fixed baseline, every one of those swaps would have been a guess. Ultimately, when a score was low, we learned to check the harness, then the infrastructure, then the grader, before we believed it was the model.

Usage data from real sessions is the other half. Early on, only a handful of pilot engineers were sending us data from their coding agents, and that was enough to show us that agents fanning out subagents behave nothing like long single prompts. That's why we're now collecting it across all of engineering before we change anything else.

The pilot also taught us that engineers compare against the whole product, not the model. The feedback we got included tool-call validation errors, an agent editing the wrong file, an agent pushing straight to main, another deleting its own pipeline with rm -rf, and engineers missing client features they rely on every day, like voice dictation. Guardrails belong in the harness and the platform, not only in the model. The reverse is also true: a smaller model with a well-built harness completed a full multi-ticket project with a good harness driving it.

So here's our honest verdict, by use case:

• CVE pipeline: Close to parity. On the steps we can grade, MiMo matched a Frontier Model at 100% on planning fixes and 92% on applying them, at default settings, though it's slower.

• Long-running agent builds: Viable, once both the serving stack and the harness are right. The same build that failed 11 times in a row finished in about six hours once the engine stopped corrupting context.

• Interactive coding: Usable, but not yet at parity. The gap is as much tool-call reliability and the client experience as it is the model.

What we got wrong

We made plenty of bad bets, and they taught us as much as the good ones:

• We thought picking the model would be the hard part: We spent our early planning on model selection. The scoreboard says the model was the root cause about 6% of the time.

• We blamed the model for a serving bug: For about two weeks, invented values and broken tool calls looked like an open model that wasn't ready. A five-minute copy test would have told us otherwise on day one.

• We trusted status fields: Ready, AVAILABLE, and "healthy" were each wrong at least once, and every time they were wrong, they were wrong quietly.

• We trusted the fit math: Our memory math said a model fit with room to spare. It ran out of GPU memory nine times in four days, mid-request, on an engine that had been serving happily for days.

• We deferred observability: Detailed observability went on after serving, and it cost us days of lost metrics and nearly the root cause of a crash, because 99.5% of the remaining engine log was our own scrapes.

• We expected splitting prefill and decode to win: On our network it lost badly, with first replies stuck around 60 seconds. It needs RDMA, and we didn't have it.

• We expected a host-memory cache tier to help: It added nothing we could measure on our traffic.

• We read a fast result as a good result: Capping context made one recipe look fastest, but only because it was quietly turning long conversations away. Now refusals count as errors.

• We experimented during business hours: It got us answers faster but cost us users' trust. Experiments now run beside the live recipe, not in place of it.

When it's everyone's tool

Project Champagne started with a small group of volunteer engineers we called our pilots (#champagne-pilots). We're now rolling it out to all of engineering, and that changes the standard more than it changes the scale. Volunteers tolerate churn because they signed up for it. Everyone else is just trying to get their work done. Here's how we're approaching it:

• Measure before you change anything: We're collecting usage data from coding agents across engineering, so decisions about models and infrastructure come from how people actually work, not from headcount or guesses.

• Promote recipes, not guesses: Nothing reaches everyone without passing the same CI window, evals, and bar as every other change.

• Separate experimentation from serving: Candidates run in staging slots beside the live recipe. Anything that has to touch the live recipe gets an announced window, a posted rollback time, and alerting watching it.

• Keep a fallback: Frontier Models stay one setting away, so nobody loses a day's work to a regression.

• Make the client side boring: One onboarding path for the coding agents we support, web search through MCP, and credentials that last as long as an agent session.

• Define success up front: Automated pipelines like CVE scanning running fully on our own models, broad daily use across engineering, and the North Star we set on day one: Frontier Models used only for testing and validation.

What this means for Modelplane, and for anyone self-hosting

We set one rule on day one: fix every gap upstream and here's some of our findings and where they stand:

Every row there is a problem we hit so that the next platform team doesn't have to. Modelplane 0.6 brings prefill/decode disaggregation with SGLang, the telemetry fixes, multiple models per GPU node, and weights distributed as OCI images. After that, 0.7 is a hardening release, and most of the open rows above are aimed squarely at it.

Pouring for the whole table

In Part 1, I said the champagne was drinkable and we were still finding the corks. We held the bottle up to the light, and learned that the champagne was rarely the problem. The cellar was! The next stage reflects the reality of where the Inference world is today, it’s constantly changing and our work will never complete.  Rather than focusing on getting to a stable point, we’ve introduced a set of components, tooling, and processes that in the true spirit of Continuous Improvement will allow us to to continue to iterate and improve every single day, regardless of how quickly the ecosystem under us moves and how rapidly the industry as a whole evolves. 

If you're about to inherit self-hosted inference, here's the short version:

• Suspect the stack before the model: Run simple tests first and constrain them to minutes.

• Treat serving changes like releases: A pull request, a fixed window, a written bar, and a rollback for every one.

• Plan for governance, not just serving: Serving takes an afternoon. IAM, networking, identity, and observability take longer and is ultimately the work to get things working in your unique environment..

• Baseline before you switch: Know your use cases and how people use Frontier Models today, or you'll never know if you got better or worse.

• Protect your users from your experiments: Predictability is part of reliability.

• What is true today, won’t be true tomorrow: Continuous Improvement of your change management processes, components, tooling, and testing & evaluation is the goal.  What works today never was.

We used frontier models to bootstrap self-hosting of our own models, but we’re not at the point where we don’t need to depend on the frontier models anymore, and will use our self-hosted models going forward to continue to iterate and improve. Classic bootstrapping and who compiles the compiler.

The champagne is ready for the whole table, and we're pouring it for all of engineering now. If you run your own inference, star Modelplane on GitHub and tell us what breaks.

Further reading

• Part 1: What it takes to own your intelligence

• Part 2: Finding the corks

• Part 3: Use case #1, the build security agent

• Part 4: Candling the bottles

• Part 5: What does it take to grow a self-hosted coding model from pilot to team?