Dalorion Johnson
Senior Platform Engineer
Upbound
Parts 2 and 3 told you what broke and how we fixed it. This one is about how we found out, why it often took longer than it should have, and what we're building so we find out sooner.
Winemakers have a trick called candling: you hold a bottle up to a light to see the sediment you can't see from the outside. Running our own models on Modelplane has felt a lot like that. The bottle looks fine. Every status says healthy. And then someone holds it up to the light.
In Part 2 we listed the corks we found. In Part 3 we put the platform to work. This post goes back through what we built, in the order we built it, and asks the same question at every step: what could we actually see at this point, and what did we miss? The stories here are new. Where Parts 2 and 3 already told one, we link to it instead of telling it again.
1. Before the harvest
Monitoring depends on there being a cluster to monitor. So the first thing you watch is the build itself, and before your own monitoring exists, the platform's status is your monitoring.
The cache disk that kept disappearing
If you've brought up a Kubernetes cluster, you know the wait. Depending on its size, it can take a while: nodes boot and join, images pull, volumes get provisioned and bound, and pods schedule onto all of it. Most of the time you just let it run. But the wait is also where you learn what normal looks like: which statuses move first, which take longest, and which ones should never stay put.
We brought up GLM-5.3 on a GPU cloud that was new to us, so we went one step at a time and waited for each to finish before starting the next. First the cluster: the runbook gave it 45 minutes, because a 1-GPU smoke test had taken 23. Then the model cache. GLM-5.3's weights are 432 GiB, and Modelplane copies them once onto a shared disk on each node via a resource called a ModelCache, so every new replica doesn't pull them from the internet again. Only once the cache was full would we start the model itself.
A ModelCache reports its progress as STAGED: 0/1 while it fills, 1/1 when it's done. Ours said 0/1. The trouble with 0/1 is that it says the same thing whether the copy is ten minutes from done or never started, and with 432 GiB to move, "still 0/1" is normal for a long while.
But we had a reason to be suspicious: two days earlier, on that smoke-test cluster, two other caches had sat Pending for 23 hours, and nobody had worked out why. So I looked one layer down, at the volume the cache was supposed to fill. A slow copy has a volume that's Bound. Ours was Pending. Nothing had ever provisioned it. And every Modelplane resource reported healthy.
So I checked the storage driver, the thing that should have provisioned it.
It was refusing to write to the node's boot disk, because the cache disk it expected wasn't mounted. The nodes had been built without it. The boot-time mount was marked optional, so nothing else complained.
So we attached the disk by hand. Under Modelplane, every cloud resource is a Kubernetes object you can edit directly, and each node group has one. We patched its template to attach the shared disk under the name the boot script was already looking for. We started with the cheap system nodes, in case we were wrong: changing a node group's template rebuilds its nodes, and on the GPU group that meant our only GPU node. We checked the result in Nebius's own API, not just the Kubernetes object, and it worked. The node rebuilt with the disk attached, and its storage driver came up healthy. Four minutes later our patch was gone. Four minutes after that, the cloud API was told to remove the disk again. Something was undoing our fix.
The timing pointed at Modelplane itself: our patch vanished right after Modelplane re-applied its own version of the node group. Crossplane lets you pause that re-applying, so we paused it and patched again. That stopped the tug-of-war. About 45 minutes after we'd first spotted the missing disk, the patch held, and 35 minutes after that the cache was full. Finding out what had been undoing it took a second look four days later, and three tools:
An offline render of the composition showed it produced the disk correctly. The bug wasn't in what Modelplane asked for.
Two recorders through a controlled unpause: one polling the Kubernetes objects every 10 seconds, one polling the cloud every 30 seconds. Four seconds after the composition applied the node group, the cloud was told to drop the disk, and the object lost the disk list again on every reconcile after that.
managedFields, the part of every Kubernetes object that records who owns each field. It showed the list being released, with no owner left to keep it.
Reading the code then turned up the cause: a known Crossplane bug in how resolved references are saved (crossplane-runtime#694), now tracked in Modelplane as modelplane#517.
The part that stings is what we missed two days earlier. That 23-hour Pending on the smoke-test cluster was almost certainly this same bug, though nobody checked before the cluster was deleted. The smoke test passed anyway, because the model it ran never touched the cache.
Candling it: a green status and a passing test both answer narrower questions than "does this work?". Check the thing itself (is the disk attached, on the object and in the cloud?), and make your smoke test use the path you actually depend on.
Part 2's CREATE_FAILED story is the same lesson from the other side: a node group marked failed while its GPUs were running. Its rule still holds: track instances and node joins, not just node-group status.
After GLM had sat at zero replicas for four days, bringing it back took about 47 minutes. The cloud had a GPU node running 35 seconds after we asked, the node was ready in about 4 minutes, and pulling the engine image took 5 more. Then the engine spent 29.5 minutes loading weights, from the same cache that had loaded them in under 8 minutes the first time. From the outside, "loading" and "stuck" look identical: the pod is running, nothing is ready, and no metric moves, because none of them measures load time. Only the engine log could tell them apart, and only if you knew what normal was. Our best explanation: between the two boots we'd turned off a setting called expert parallel, so every GPU now reads a slice of every shard instead of just its own experts. Restarts on a node that has already loaded the weights still take one to five minutes; a brand-new node takes half an hour or more.
And on our first cluster, a from-scratch rebuild took over four hours. Separately, Helm charts that installed before any nodes existed stayed failed long after their pods ran fine.
2. Keys to the cellar
Once there's a cluster, the next question is who can get in, and who shouldn't.
The door that should have been shut
When you put auth in front of an endpoint, the first test is the obvious one: does my token work? Ours did. Every request we care about goes through a gateway that checks a Google identity token or an API key, and our dashboards show requests per second, latency and errors for all of it. But every one of those numbers is about a request that was allowed. Nothing on them can tell you about a request that should have been refused and wasn't.
So alongside the requests that should succeed, we keep a small probe matrix of requests that are supposed to fail, and run it after auth changes. A few of its rows:
Request | Expect |
no token | redirect to sign-in |
garbage token | 401 |
API key signed by a key that isn't ours | 401 |
expired API key | 401 |
revoked API key | 403 |
someone else's name in the caller header | usage still credited to the real caller |
health check, no token | 200 |
In shell, the core of it is a handful of curls:
The first thing we learned about doors is that Modelplane's starts open. Our first environment ran Modelplane 0.4, which on our setup had no front door of its own: each model sat behind a public, unauthenticated, plain-HTTP endpoint, so we bolted Google sign-in and TLS on in front of it. Modelplane 0.5 added a real gateway, InferenceGateway, whose spec.auth handles API keys only. We wanted people to keep signing in with their Google accounts, which also gets them into Grafana through the same door, so we left spec.auth empty and kept running auth ourselves. Without it, Modelplane adds no security policy at all, so the gateway answers anyone from the moment it gets an address. Our rule became: put our own policy on before the gateway gets an address.
Leaving spec.auth empty had a side effect too. Modelplane normally leaves the health check open, but only when it's handling auth itself. With our policy in charge, the load balancer's health check got a 401 like everyone else. Our own check from outside that afternoon saw that 401 and called it fine: we were asking "is the door locked?", and a locked health check looks exactly like a locked door. Sumbry's PR already had the fix, reopening the route. Since then, the matrix also lists what should stay open. Part 2 tells the rest.
Changing the lock went cleanly. When we replaced our temporary auth with the real thing, we applied the new policies beside the old one first. They reported Conflicted: two policies on one gateway don't stack, and the oldest wins. Deleting the old one handed over, and the first probe afterwards was already on the new lock. No open window. To see which policy is actually in force:
Then, on October 1, the revoked key got its 403 on inference. On /v1/models, the endpoint that lists which models exist, the same key got a 200 and the full list.
The policy said that key was denied, so the answer had to be in the proxy. Its configuration dump showed the order the checks run in: sign-in, then the token's signature, then the AI gateway, and only then the rule that asks whether this key is still allowed. The AI gateway answers /v1/models on its own, so that one request never reached the last check. Every other path did.
It only leaked model names; a revoked key still couldn't run anything. But no dashboard would have shown it, because every one of those requests succeeded. That was the problem.
A week later we found a door that was literally open. On October 6 we tore down our first environment, one step at a time, with a check after each step. One step deleted a folder of manifests that held the only auth policy on Modelplane's own gateway, while both models were still running behind it. The check right after it, a request with no credentials, came back 200. We put the policy back, and the same request got a 401. The models had been open for about two and a half minutes. The gateway's access log for that window held eight requests, and all eight were our own checks. The fix was the order: the policy now comes down last, after the models. What would have caught it the moment it happened is the matrix's first row, run in a loop for the whole change:
Candling it: test what should be refused, not only what should work, and keep testing it during a change, not just after one. A door that's open looks exactly like a door that's working.
3. Installing the hydrometers
A winemaker can't see inside a fermenting tank, so they take readings. Ours come from emitters: exporters on each component, a collector on each cluster, and a Prometheus to keep it all.
We used to run several Prometheuses. Since October 12, every cluster's collector ships its metrics over OTLP to just one, so the collector is the one thing every metric passes through. Which is why this next story matters.
The collector that kept its old config
If you've rolled a Deployment, you know the check: Running, Ready, rollout finished. It tells you the pod started. It doesn't tell you what it started with.
We were adding GPU panels to find out whether GLM's eight GPUs were waiting on each other, which meant teaching the collector four new metric names with a Modelplane MetricMapping. Modelplane rolled the collector on each GPU cluster, and we watched for the new names:
Our AWS cluster showed up within about a minute. The GPU cloud's never did, and nothing logged an error. We went down the chain, and every link checked out: the config had the mappings, the exporter was emitting, the logs were clean, and the cluster's other metrics were 13 seconds old. Then the timestamps:
The pod started at 19:41:00. Its config was updated at 19:41:01. One second. Modelplane applies the collector's Deployment and its config separately. The new pod won the race and read the old file, and the collector never reads it again. Deleting the pod fixed it.
It was the third time in four days. The first, during our switch to OTLP, was loud: an error in the log. Our runbook gained a line to recreate the collector after changing where metrics go. The next two were changes to what they're called, so that line didn't cover them, and the third logged nothing at all. The race is still in Modelplane.
Candling it: something outside the monitoring stack has to check that it's alive and current. Alive means the data is fresh. Current means the newest names arrived, from every cluster. Ours was fresh the whole time.
A quieter version: our GPU cloud's nodes came with no GPU exporter, and we deferred it on install day. Four days later a teammate found GLM slow, and every GPU panel for that cluster was empty. One standing query would have flagged it the minute the model's metrics arrived:
4. Tasting notes
Once readings flow, the hard part is knowing which ones answer the question in front of you.
"Wow, is it slow"
If you've used a coding agent against a hosted model, you know what fast feels like: the first words land in a second or two, then text streams steadily, and tool calls go by quickly. You stop noticing the model at all. That's the bar a self-hosted model gets measured against, whether you meant it to or not.
The evening GLM went live for volunteers, a teammate switched Claude Code over to it and reported back: "wow, is it slow." About a thousand tokens a minute, one turn that ran six minutes, and then API timeouts.
So we opened the dashboards, and they were fine. Time to first token: a few seconds. Time per output token: about 22 milliseconds, call it 45 tokens a second per request. Requests running: never more than four, against a limit of eight. Requests waiting: zero. KV cache: under 15 percent. By every panel we had, GLM was a healthy, lightly loaded model.
One number was out of line: how much each request wrote. Requests averaged around a thousand tokens each, and one in ten wrote over three thousand. That's a lot of text for an agent that was mostly running short shell commands. The engine was talking the whole time. It just wasn't talking to us.
The first cause was hidden thinking. GLM-5.3's chat template defaults to its maximum reasoning effort, and the client never asks for less. Timing one prompt at three levels told the story: maximum ran past 140 seconds without an answer, high took 37 seconds, low took 9. And none of that reasoning reached the user. The engine streams it under one field name, our gateway's translation to Claude Code's API reads another, so every turn sat silent until the thinking stopped. That's also why time to first token looked fine: it counts the first token the engine produced, and that was a reasoning token nobody would ever see.
The second cause was our own client. In its auto-permission mode, Claude Code checks each shell command with a safety classifier, and we had it running on the same model. So every shell command was two requests, and the second was one more chance for GLM to think as hard as it could before saying yes. It wasn't a queue, because there was never more than one or two requests at once. It was a wait, repeated on every step.
How did we find it? A user report, and then pointing Claude Code at the problem and asking it to debug itself. Not a dashboard. Setting effort to high fixed most of it overnight, and we've since made low the default.
Three days later we got the other shape. On a busy morning, requests running hit eight and stayed there, requests waiting climbed to eleven, the KV cache filled to 99 percent, and the time requests spent queued went from seconds to minutes. That one is a ceiling: GLM's cache holds about two full-length contexts, and several long agent sessions at once crowd each other out. Every panel showed it.
Candling it: "slow" has shapes: queued, thinking, or genuinely slow. Each has a panel, but check what the panel actually measures. Our time to first token measured the engine's first token, not the user's.
The questions became dashboards
Two questions came up often enough that we turned them into panels.
Is speculative decoding still paying off? A model writes one token at a time, and each token is a full pass through the model. Most of that pass goes to reading the weights out of GPU memory, so checking four tokens costs about the same as checking one. Speculative decoding takes advantage of that. A small extra layer in GLM guesses the next few tokens, and the full model checks them all in one pass, keeping them up to the first wrong guess. On a good hour, GLM gets close to three tokens out of each pass instead of one. Over its first two days it kept roughly half to three quarters of its guesses, hour by hour, depending on what it was writing. If that collapsed, every pass would still pay for the guesses and get nothing back, and GLM would slow down while every other panel looked normal. That isn't hypothetical: an open vLLM bug reports exactly that on the same GPU family, with a newer kernel library than the one we run. Acceptance now has its own row.
Is GLM waiting on its GPUs talking to each other? GLM is split across eight GPUs, so every layer of every token ends with all eight combining their partial results before the next layer can start. Datacenter GPUs do that over NVLink, a dedicated GPU-to-GPU link. Ours don't have one, so the exchange goes over PCIe, and by default it takes the long way: each GPU copies its piece into the CPU's memory, and the others copy it back out. The communication library calls that path SHM, for shared memory. We added SM activity and PCIe panels to see whether it was the bottleneck. The first busy stretch showed the GPU reporting "compute active" at 99 percent, while SM activity peaked at 52 percent and the tensor cores were busy at most about 10 percent of the time. The chip says busy; most of it is waiting.
So we tested the obvious suspect. The GPUs can send data to each other directly over PCIe, skipping the CPU's memory; the communication library just doesn't by default when they sit on different CPU sockets. We turned it on. The link carried half again as much traffic, and prefill got no faster: no gain in a quick test, and about 13 percent slower over a morning of real traffic. We turned it back off. Whatever GLM is waiting on, it isn't the trip through host memory. That's an answer we wouldn't have trusted without the panels, and one we'd have kept guessing at without them.
Three techniques worth stealing
We've written about these incidents before, but not about how to spot them:
Read the number. In Part 3, a request failed at 239 seconds. That's four 60-second attempts. A failure at a round number is a timeout, and the multiple tells you about retries.
Bypass a layer. In Part 2, the same generation ran 191 seconds cleanly through a port-forward straight to the gateway's proxy, and was cut at 60.7 seconds through the load balancer. Take one layer out and see if the problem moves.
Find the evidence before it's gone. When a model on our first cluster crashed out of GPU memory, the request that killed it had bypassed the gateway, so the gateway log was empty. The engine's own log had already rotated, and 99.5 percent of what was left was our own metric scrapes. The stack trace survived in one place: the crashed container's previous log.
5. The cellar master's rules
So what should actually wake someone up once this goes live? Honest answer first: today, nothing does. There's no alerting on our POC yet. Which makes this the right moment to decide what it should be.
We're not starting from scratch on that question. On our hosted platform we learned the hard way that alerts on causes (a pod not ready, a node's CPU high) are mostly noise, while people end up catching most of the incidents that matter. We don't want to repeat that on Modelplane. So the plan has two layers.
Easy wins: cheap guards from data we already have
These go to a channel, not a pager. Each one is a pattern from this post rather than a single incident.
An engine restarted. It needs nothing new:
resets(modelplane_request_duration_seconds_count[15m]) > 0. Now that engine metrics flow, that alone catches a crash like the out-of-memory one in section 4.The observer went quiet or stale. An expected series is missing, or old, for a cluster: a cluster serving a model but sending no GPU metrics, or a new mapping whose series never arrive. That covers both hydrometer stories in section 3.
A status disagrees with reality. Compare what an object says about itself with the thing itself: is the disk attached, did the instances join?
A door that should be shut is open. Run section 2's probe matrix on a schedule, not just after changes: every refusal still refused, the health check still open. Today we run it by hand.
The north star: SLOs on what users feel
This part is a question we're asking, not something we've built. Observability Engineering argues you should alert on what your users experience, measured per request, and use everything else to debug. What would that look like for a self-hosted model?
Start with the user. Ours is a person in an interactive coding agent, waiting on the front door. An eval harness running overnight is a different user with different tolerances, so it gets its own SLO or none.
Then pick the events. The front door already emits OpenTelemetry histograms for every request, so we can count good requests against all requests:
Responsive: time to the first token the user actually sees, measured at the front door, under some threshold. That's queueing plus prefill, plus any thinking the client never shows: the part a person sits and stares at. Section 4 is the warning. The engine's own first-token clock looked fine while users waited.
Fast enough: time per output token under some threshold. That's decode speed, where the speculative-decoding and PCIe panels live.
Answered: the request completed without an error. The histograms carry an error label on failures, so this is countable today.
And deliberately not total duration. How long a request takes depends on how much the model writes. A 191-second generation can be perfectly healthy. That's the trap behind several of the timeout stories, and it's specific to LLMs.
You don't need a new vendor for this. A histogram already counts events, so "requests with time to first token under 10 seconds, divided by all requests" is a plain Prometheus query:
Open-source tools like Sloth or Pyrra turn a small SLO definition into the burn-rate alert rules, and Grafana draws them. For the questions metrics can't answer, like who is sending all these requests (the question that would have shown every shell command arriving twice), the gateway already writes a per-request access log with the caller in it. Shipped to Loki, next to the Grafana we already run, that's one query. Prometheus can't do it, because a label per caller would explode its storage.
Where it gets hard, and where we'd love to hear how others handle it:
Low traffic breaks naive SLOs, whatever the tool. One of our models handled about 5,400 requests in a day; another about 13. At 99.9 percent over 30 days, the second one's error budget is less than a single request. The usual answers are a fleet-wide SLO, longer windows, or synthetic traffic.
No data means no burn. If metrics stop arriving, an SLO computed from them reads as 100 percent good. So the "observer went quiet" guard above doesn't go away once SLOs exist. The SLOs depend on it.
What's the objective for a POC? Nobody's been promised anything yet. The book's advice is to start from what users actually tolerate and use the error budget to make decisions. We have one ready-made: GLM's reasoning default trades speed for better answers, and an SLO would make that trade explicit instead of a hunch.
And some plumbing first. Modelplane's collector ships metrics only, so a log pipeline would be ours to run, and the gateway's access log has been seen dropping lines, which has to be fixed before it can count anything.
What we (re)learned
A status is a summary. It's one component's view of its own job. Verify against the thing itself, and make your tests use the path you depend on.
Test what should be refused, and what should stay open, during a change as well as after it.
Watch the watcher. Something outside the monitoring stack has to check that it's alive and current: fresh data, and the newest names arriving from every cluster.
"Slow" has shapes. Queued, thinking, or genuinely slow: read the shape before you tune, and check what each panel actually measures.
Read the number, and bypass a layer. A round-number failure is a timeout; taking a layer out tells you which one.
Alert on what users feel. Keep the cause signals as guards, not pages.
Further reading
modelplane#517, the cache-disk bug from section 1, and its root cause in Crossplane, crossplane-runtime#694
Observability Engineering (Majors, Fong-Jones and Miranda, O'Reilly)
If you're running models yourself, star Modelplane on GitHub and tell us what you'd alert on.
About Authors
Dalorion Johnson
Senior Platform Engineer
