This series is about drinking our own champagne: we run Modelplane for our own engineers before we ask anyone else to. About 10 of our engineers pilot GLM-5.3, served with Modelplane on 2 nodes of Nebius GPUs (16 × RTX PRO 6000), behind coding agents like Claude Code, OpenCode and pi. Current usage is intermittent. This part asks what it would take to grow from that pilot to a team of about 15, the next size up, with a mix of heavy and light users. The short answer: 2 nodes couldn’t hold a team of 15 with any change we tried, and 4 nodes running SGLang could, for about what we pay per engineer today. One surprise on the way: light users aren’t light, because their subagents keep more requests in flight than heavy users’ long prompts do. The rest of this post is how we know.

No fix was obvious on paper: a smaller context turns away long conversations, a second copy doubles the GPU bill, and a new engine or a prefill/decode split changes how every request is scheduled.

We measured instead, one change at a time, on the fleet our engineers use, with a test rig that runs outside peak usage.

This post is mostly about the method, since our numbers are specific to our hardware and traffic, and we’d like to hear how others do this.

Our traffic, and the bar

A recipe, for us, is one Modelplane ModelDeployment: the engine, its flags, the GPU layout and the number of copies. A recipe run is the 2 hours a recipe gets on the GPUs; a sweep steps the number of concurrent sessions up until a step misses the bar; a window is an off-peak slot where a candidate takes every node and ends with a rollback. Ours runs a single vLLM engine across 2 nodes of 8 × RTX PRO 6000 (16 GPUs), with a 512K context, room for 64K-token answers, multi-token prediction and prefix caching.

Our engineers’ real traffic: what they send, over the last 2 weeks, and how today’s recipe serves it:

At today’s load, the prefix cache holds most conversations and replies start within a couple of seconds. The risk is growth: a conversation that falls out of the cache comes back with a long history and stalls every other agent while it prefills, and more people means more of those.

It’s also sparse, 10 people for a few hours a day, too little to replay or to see a tail in. We bootstrapped with someone else’s traffic instead: public Claude Code traces from SemiAnalysis’ AgentX, whose typical request is longer than our tail. Capping the context at 128K would turn away more than half of them.

A replay reproduces the load’s shape, not users who wait and react, though its prompts are of a similar length to ours. Its unit is the concurrent session (AgentX’s concurrency), one AgentX agent session replayed in a loop. 10 concurrent sessions keep about as many requests in flight as our peak minute, and keep it up for the whole run, so they stand in for today’s 10 pilot users. We assume one pilot user is about one concurrent session, so the target, a team of about 15, is 15 concurrent sessions, and 20 is our stress test. The replay is closed-loop, so a slow recipe receives fewer requests, and 20 is about twice today’s load only for the half-and-half mix. The traces are hashed placeholders rather than real text, and we didn’t correct for what that does to multi-token prediction: it guesses on synthetic tokens, so its speed-up in our sweeps may differ from what real code gets. Every recipe ran it the same way, so the comparison holds up better than the absolute numbers.

We split AgentX traces into 2 tiers by prompt length, the longest quarter (heavy users) and the shortest (light users):

The 2-tier simplification is deliberate - prompt length decides prefill and prefill is what our nodes run short of first. They also bracket our own traffic: our typical prompt sits near the light tier’s, and our tail between the 2 tiers.

We agreed on the bar and wrote it down before the first run. We measured today’s recipe replaying today’s mix of 10 concurrent sessions, then allowed about the run-to-run spread, so today’s 10 users wouldn’t notice a change. These are replay numbers, higher than the real-traffic table above, because replayed sessions never pause and their prompts run longer:

• TTFT (time to first token): P90 ≤ 5.3 s (today’s 4.8 s plus 10%)

• Interactivity: P90 ≥ 10 tokens/s per user, the speed 90% of streams beat (today’s 12.5 less 20%)

• Errors: under 2%, counting the turns a failed conversation never got to send

• Evals: a smoke test at this sample size, which blocks a recipe only on a confident fail

Among recipes that pass, the cheapest per session-hour wins.

Each part gets a verdict, pass or fail with its margin, from a 95% interval bootstrapped over the run’s minutes. When that interval crosses the bar, the verdict is too close to call, and a fail is low-confidence. Minutes within one conversation aren’t independent, so the bootstrap overstates our confidence, and we treat close calls as unsettled. A missing measurement never counts as a pass, and latency reads as no data when less than 95% of the planned load went out.

7 recipes, one change each from today’s or SGLang’s

Each candidate changes one thing from today’s recipe, recipe 0, or from the SGLang one, recipe 1, in a few lines of YAML:

Our first draft’s SGLang recipe changed 3 things at once, so a win wouldn’t have said which mattered. Recipes 3, 6 and 7 add nodes to see what more capacity buys, and compare with each other at the same cost.

How we measure a recipe

Figure 1: A recipe run, from pre-flight to promote or roll back.

A classic canary sends a slice of real traffic to the new version beside the old, which takes 2 recipes’ worth of GPUs, the thing we’re short of. Our rig instead runs the whole recipe on its own nodes, or in a window outside peak usage, and replays agent traffic against it. Each recipe gets one recipe run, end to end in CI:

Each candidate gets 2 hours on the GPUs, rollout and rollback included, so cheap tests run first and a broken recipe fails in minutes. Production stays on today’s recipe: candidates run in a staging slot beside it, or, when they need every node, in an off-peak window that ends with a rollback.

A recipe holds the target when the step at 15 concurrent sessions passes; 20 is the stress step. Each step runs 5 minutes after a warm-up, and the sweep also stops when the KV cache fills. If a recipe can’t serve even today’s load, the rig ends the sweep itself and rolls back. Light-only and heavy-only ladders are optional, to show how sensitive a recipe is to the kind of team.

Serving signals

The rig decides only on what the server reports. We don’t want to rely on agent harness metrics and logs in a shared serving fleet where we may not have access to them or may not trust them. Thus the rig sends no probes and uses its load generator only for how much load it is meant to send, so it sees what an operator sees. Each run lists the signals the bar wanted and didn’t get, and we close those gaps in Modelplane.

Modelplane collects what the rig needs out of the box with OpenTelemetry, from the gateway and every engine, and the rig saves it with every run, so a slow hour comes with what explains it.

Testing the rig before it spends a window

A window lost to a rig bug costs as much as a good one, and we lost 2 that way, both to flaws a few minutes on a laptop would have caught. Twyman’s law says any figure that looks interesting or different is usually wrong, and our first runs had 2: rejected requests that didn’t count as errors, and a warm-up that quietly ate a whole run.

We test it in levels, cheapest first, and each level runs the same phases on a bigger slice:

• Check, in seconds: unit tests and a dry plan of the run.

• Rehearsal, in minutes on a laptop: every phase against a fake gateway that fails on demand. Each injected fault, such as a crashed engine or missing metrics, must produce the right verdict.

• Smoke, in minutes on the fleet: the pre-flight, the mini evals and one short sweep step, to tell whether a recipe has earned a window.

• Recipe run: the whole window, from pre-flight to a capacity number.

Our first runs, during working hours on purpose, found a serving bug before a recipe. Our evals didn’t set max_tokens, so when their clients gave up, the engine kept generating. The disconnect didn’t reach the engine, and we’re still tracing where it got lost. An hour later abandoned requests held every slot and a colleague waited minutes per reply, while the queue we watched looked merely busy. The missing caps were our mistake. Now every request has a cap, a run isn’t over until the engine is back where it started, the dashboard shows cancellations next to the queue, and Modelplane main counts the requests callers still have open at the gateway.

Replacing recipes

2 recipes failed early and cheaply, so we dropped them rather than run them again. This is a benefit of the tiered rig testing so we can eliminate bad serving recipes quickly and replace them with ones that work.

• Recipe 2, 128K context: looked fastest only because it turned long conversations away and lost their turns. It has no replacement. It taught us to count refusals as errors.

• Recipe 4, prefill/decode: couldn’t hold even a few light-only concurrent sessions. Each request waits to move its KV cache to a decode engine over plain TCP, and splitting prefill from decode needs RDMA or close to it, which these nodes don’t have. Recipe 5 replaces it as the heavy-user bet on SGLang.

Where the recipes landed

Every recipe is swept on today’s mix the same way and reported as one number: the most concurrent sessions it holds at the bar, with an interval.

Figure 2: P90 TTFT as concurrent sessions step up on today’s mix, per recipe; the green band is the bar (today’s TTFT + 10%), dashed ‘?’ is unmeasured, ▲ marks where each recipe crosses the bar.

Figure 3: The same for light-only sessions, for comparison.

Figure 4: P90 interactivity, the tokens/s per user that 90% of streams beat, as concurrent sessions step up on today’s mix, at every load each recipe ran. Higher is better; the green band is the bar, 10 tokens/s per user.

• Recipe 0, today’s vLLM: it is the bar at today’s mix of 10, yet holds fewer concurrent sessions of a single kind. Light-only sessions fan out subagents and keep more requests in flight each. A few heavy-only sessions miss the bar on single long prompts, even though they send fewer requests than the bar’s load.

• Recipe 1, SGLang: more than twice today’s light-only capacity on the same 2 nodes.

• Recipe 3, 2 vLLM copies on 4 nodes (32 GPUs): on today’s mix it held 13.7–15 concurrent sessions, an interval that touches the target, so by our own rule it is too close to call rather than a clear miss. TTFT limited it.

• Recipe 5, SGLang + HiCache: the same light-only capacity as SGLang alone, and about today’s heavy-only capacity. HiCache added nothing we could measure, because the KV cache never filled (it peaked under half full), so the gain over vLLM is SGLang’s own.

• Recipe 6, 2 SGLang + HiCache copies on 4 nodes (32 GPUs): held the target on today’s mix, limited by TTFT, and missed the stress step narrowly.

• Recipe 7, the same 2 SGLang copies without HiCache: the simplest recipe that holds the target, and it held at least as much as recipe 6, so HiCache added nothing we could measure. Per session-hour it costs about the same as today’s 2 nodes; per token it costs more, because vLLM gets more tokens through each GPU.

The bar: P90 TTFT ≤ 5.3 s, P90 interactivity ≥ 10 tokens/s per user, errors < 2%. The target is 15 concurrent sessions.

Today’s-mix values come from one sweep per recipe, in a staging slot or a late-night window, with 95% intervals. Recipe 0 sets the bar at 10. Recipe 7 passed every step it ran, up to 20; 25 wasn’t run. Too close means the interval touches the target. Single-kind capacity runs from the last step that passed to the first that failed, with nothing extrapolated. $ per session-hour is the node price divided by the sessions held: each node is 8 × RTX PRO 6000 at about $14.40 an hour ($1.80 per GPU-hour), so 2 nodes are 16 GPUs at $28.80 an hour and 4 nodes are 32 GPUs at $57.60. Errors count the turns a failed conversation never sent. $ per M tokens counts cached input, uncached input and output tokens at the last step that passed, as InferenceX does. The replay is closed-loop, so a faster recipe gets through more turns in a session-hour: $ per session-hour is the cost per user at the bar, and $ per M tokens the cost per unit of work. At the stress step of 20, every 2-node recipe missed badly, recipes 3 and 6 missed narrowly, and recipe 7 passed. 

Figure 5: Concurrent sessions each recipe holds within the bar, per node and per dollar.

Figure 6: The run dashboard in Grafana during one of recipe 3’s early canary runs: TTFT and interactivity at the gateway, errors, requests by model, and the kill switch’s view of the queue.

2 nodes don’t hold a team of about 15 with any change we tried. Switching vLLM to SGLang is the biggest single gain on the same hardware: it lifts today’s mix a little, roughly doubles the light-only ceiling, and runs out of interactivity before TTFT. 4 nodes do hold the target with SGLang: 2 SGLang copies behind one router held it with room to spare, while 2 vLLM copies came too close to call. Per session-hour, that costs about the same as 2 nodes do today, though more per token. HiCache added nothing: the same 2 SGLang copies without it (recipe 7) held at least as much as with it (recipe 6). Each number is one run with an interval, so close calls stay close.

What held up in Modelplane, and what didn’t

We maintain Modelplane, so read this with that in mind. Engines run on different GPUs and add knobs every few weeks, so we bet on leaving them alone: a ModelDeployment works like a Kubernetes Deployment, where you bring the container and its flags and Modelplane shapes the topology around them. This was the first time we leaned on that bet this hard, and it mostly held. Every recipe was a few lines of YAML; 2 needed fixes that are on main, soon to be released in v0.6.

We made the same bet on metrics, since every engine names and buckets its own. A MetricMapping maps them onto common modelplane_* names, so we added what the rig needed the day it needed it. 

The bets didn’t hold everywhere. 2 engine copies fail together (#557), the router’s default prefix match didn’t do what its setting said (#558), and the first canaries found gaps in the metrics, now fixed on main:

• SGLang and mixed fleets. SGLang’s cache, retraction, TTFT and status metrics now reach the modelplane_* names, so an SGLang recipe is judged on the same signals as a vLLM one (#544, #546). Tokens aggregate as plain totals, and prefill and decode engines carry a phase label so a request isn’t counted twice (#546).

• Abandoned work. The gateway now reports the requests callers still have open (modelplane_frontend_requests_active, #548).

Lessons

Treat the rig like production code, not a one-off experiment. Ours is tested in levels before it spends GPU time, every bad run gets a check written for it, and it runs in CI, so the same harness answers the next questions too: the load sweep and a staging slot beside production.

Fix the GPU budget first. It decides the method, and it limits what you can conclude: with one window per recipe, close calls stay close.

Measure capacity on one mix, not pass or fail at one load. Our fixed-load canary said every 2-node recipe failed but not by how much; a load sweep on the mix we serve gives one comparable number per recipe, made the rig simpler, and avoids guessing, since mixes aren’t additive.

Light users aren’t light. Their subagents keep more requests in flight per concurrent session than heavy users’ long prompts do, so plan capacity in requests in flight as well as in users.

Track silent failures. Our 128K context cut TTFT only because long conversations errored out and stopped sending. Count refusals as errors, and count the turns they stop.

Check correctness under load. Our evals passed one request at a time, while our engineers’ agents saw malformed tool calls under concurrent load on today’s recipe, likely a vLLM bug with pipeline parallel and multi-token prediction (vllm#60897). None of our serving metrics moved. Run the evals during the load sweep, and count the tool calls that don’t match their schema (#565).

Cap every request, and watch cancellations next to the queue. When clients give up on uncapped requests, the engine keeps working on them, and the queue only looks busy.

Test on your own hardware with traffic like yours. Agent prompts are long and mostly cached, so a benchmark of short, fresh prompts measures a different system.

What’s next

• Find recipe 7’s ceiling: it passed every step we ran, so the next run starts at 20 and climbs.

• Promote the winner for the team with one label, then try it as a real canary: a share of live traffic beside today’s recipe.

• Replay our own traffic: the shape of our engineers’ conversations, not their content.

• Run the evals during the sweep, so a recipe that breaks tool calls under load fails, starting with a fix for today’s.

• Make it routine: every recipe PR gets a recipe run that posts each phase’s numbers to the PR and the team’s channel as it goes.

We’ll also upstream what we hit: errors by status (#533), gateway metrics by deployment (#534), router and cluster-gateway high availability (#554), copies that fail together (#557), the router’s prefix-match setting (#558), TTFT buckets past 60 s in the gateway, and tool calls checked against their schema at the gateway (#565).

Our bar is relative to today’s recipe, which is all you have before SLOs. A production service would set SLOs per tier of user, gate promotion on error-budget burn, and send a share of live traffic to the candidate, which takes GPUs for 2 recipes at once.

Growing from pilot to team looks like 4 nodes (32 GPUs) running SGLang, for about what we pay per engineer today. If you serve models to your own agents, star Modelplane on GitHub and tell us how you decide what to change.

Further reading

• Part 1: What it takes to own your intelligence

• Part 2: Finding the corks

• Part 3: Use Case #1, the build security agent

• Part 4: Candling the bottles

• AgentX methodology