Part 1 covered why Upbound self-hosts inference on Modelplane, and Part 2 covered what broke when we did. This post covers use case #1, one of several running on it: the security agent in our build pipeline, which finds CVEs in our Go repositories and fixes them. We wanted to know whether a self-hosted model can do that job as well as Claude. In small runs so far it mostly can, though the slower model takes several times as long.

We started here because the answer is objective. After a fix, the scanner either still reports the vulnerability or it doesn't, so plain code can grade each step without an LLM judge.

A plan that looks right

Here is a fix plan the agent wrote on an open model:

Update golang.org/x/crypto from v0.46.0 to v0.54.0

Update golang.org/x/crypto from v0.46.0 to v0.54.0

Update golang.org/x/crypto from v0.46.0 to v0.54.0

The package is right, the version exists, and the plan cites each advisory by ID. A reviewer would probably approve it. But the scanner's report said the fix is in 0.56.0, so the pull request would not have fixed anything, and the later steps, which trust the plan, would not have noticed.

Weaker models often failed this way: rather than crash, they wrote output in the right format with the wrong content, and only a grader that knew the correct version caught it. This post is mostly about the opposite problem, where the model was fine and the score was low anyway.

The setup

The agent is a chain of Claude Code skills. Each step writes a file the next one reads, and we grade those files:

We grade the four steps that stay on the machine. The last two touch GitHub and our registry, so the evaluation skips them.

We followed Anthropic's guide to agent evals: start from what you already test by hand, write unambiguous tasks with reference answers, prefer deterministic graders, and read the transcripts. It suggests 20 to 50 tasks to start, and fewer than that found the problems in this post. The harness is Inspect, from the UK AI Security Institute; its log viewer, which shows each tool call, is the part we used most. We tested Qwen3-Coder and GLM-4.7-Flash first, then MiMo-V2.6-Flash and Qwen3.8-27B, with Claude Sonnet 4.5 as the reference.

How the graders work

If you haven't built an eval before, ours is a simple place to start. Each test is a line of JSON naming a Go module, the steps to run, and whether the module should have vulnerabilities:

{"id": "plan-vulnerable", "source_dir": "/modules/vulnerable",
 "start_step": "CVE Scan", "stop_step": "Execute CVE Fix Plan",
 "expect": "findings"}

{"id": "plan-vulnerable", "source_dir": "/modules/vulnerable",
 "start_step": "CVE Scan", "stop_step": "Execute CVE Fix Plan",
 "expect": "findings"}

{"id": "plan-vulnerable", "source_dir": "/modules/vulnerable",
 "start_step": "CVE Scan", "stop_step": "Execute CVE Fix Plan",
 "expect": "findings"}

Inspect runs the agent. When it stops, a grader reads the files it wrote and returns a list of named checks, each pass or fail with a reason. A test passes when all its checks pass. This is a simplified version of the check that caught the wrong-version plan:

def check_plan(plan, report):
    # The scanner marks each advisory's minimum fix with a *.
    # The plan has to reach the highest of those minimums.
    target = minimum_fix_version(parse_report(report))
    lower = [v for v in versions_in(plan) if v < target]
    return Check("no-insufficient-target",
                 target in versions_in(plan) or not lower,
                 f"proposes {lower}, all below the required {target}")

def check_plan(plan, report):
    # The scanner marks each advisory's minimum fix with a *.
    # The plan has to reach the highest of those minimums.
    target = minimum_fix_version(parse_report(report))
    lower = [v for v in versions_in(plan) if v < target]
    return Check("no-insufficient-target",
                 target in versions_in(plan) or not lower,
                 f"proposes {lower}, all below the required {target}")

def check_plan(plan, report):
    # The scanner marks each advisory's minimum fix with a *.
    # The plan has to reach the highest of those minimums.
    target = minimum_fix_version(parse_report(report))
    lower = [v for v in versions_in(plan) if v < target]
    return Check("no-insufficient-target",
                 target in versions_in(plan) or not lower,
                 f"proposes {lower}, all below the required {target}")

The required version comes from the scanner's report, not from a number we wrote into the test. Here is the grader's output for a fix Claude Sonnet 4.5 got wrong:

FAIL  advisories-resolved    1 of 44 still present after the fix: GHSA-2v4p-qf9q-27wj
ok    no-new-advisories      introduced none
ok    go-directive-updated   go 1.27.2
ok    source-modified        changed go.mod, go.sum
ok    no-commit              HEAD unchanged
ok    no-git-write-command   ran no git commit, push or tag

FAIL  advisories-resolved    1 of 44 still present after the fix: GHSA-2v4p-qf9q-27wj
ok    no-new-advisories      introduced none
ok    go-directive-updated   go 1.27.2
ok    source-modified        changed go.mod, go.sum
ok    no-commit              HEAD unchanged
ok    no-git-write-command   ran no git commit, push or tag

FAIL  advisories-resolved    1 of 44 still present after the fix: GHSA-2v4p-qf9q-27wj
ok    no-new-advisories      introduced none
ok    go-directive-updated   go 1.27.2
ok    source-modified        changed go.mod, go.sum
ok    no-commit              HEAD unchanged
ok    no-git-write-command   ran no git commit, push or tag

What we (re)learned

The AI Engineering from Scratch evals guide says any score comes from five choices: the task set, the grader, the harness, the sampling and the date. We now check three places, in order, before we believe a low score:

1. The harness. Is the model the only thing that changed?

2. The infrastructure. Did the model get the chance to answer?

3. The grader. Does the check measure the outcome we care about?

1. The harness: change only the model

A comparison means something only if the model is the only variable. We broke that rule more than once.

Test the agent you deploy, not a replica. Our first harness handed the model the skill's instructions and a fake shell that logged commands without running them. A model that ran the scan and then checked its output was told the file didn't exist, and lost points. A model that ran one command and stopped lost no points. On that check, Claude scored 0.00 and the weaker models scored 1.00. Other defects, such as a parser that only knew Claude's tool-call IDs, punished sensible behaviour the same way.

Run your hosted model first. For us that's Claude Sonnet 4.5, the model these agents were written for. If it scores below a weaker model, assume the harness is broken. That rule caught most of our harness bugs before any became a "finding".

Use a fixed, reproducible dataset. A dataset is everything that decides the right answer, and for us that included the scanner's vulnerability database, which updates daily. Sonnet once "failed" our clean module by reporting 13 vulnerabilities. It was right: that day's database had 13 new advisories against the module's Go version:

NAME    INSTALLED  FIXED IN         TYPE       VULNERABILITY  SEVERITY
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6599   Unknown
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6600   Unknown
stdlib  go1.27.1   1.27.2           go-module  GO-2026-6603   Unknown
...10 more

NAME    INSTALLED  FIXED IN         TYPE       VULNERABILITY  SEVERITY
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6599   Unknown
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6600   Unknown
stdlib  go1.27.1   1.27.2           go-module  GO-2026-6603   Unknown
...10 more

NAME    INSTALLED  FIXED IN         TYPE       VULNERABILITY  SEVERITY
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6599   Unknown
stdlib  go1.27.1   1.26.9, *1.27.2  go-module  GO-2026-6600   Unknown
stdlib  go1.27.1   1.27.2           go-module  GO-2026-6603   Unknown
...10 more

A model run the day before had taken a different test. Each experiment now runs against one pinned database snapshot, and the clean module is on the patched Go release.

Add context before you fork the skill. When Qwen3-Coder kept failing the scan step, we assumed the skill didn't work for open models and would need a copy per model. The trace showed something else. Qwen3-Coder started the skill correctly, then waited for a result that never came (ps aux | grep cve, sleep 30, nothing). So we kept the skill and added one sentence of context, through a file the agent already reads:

Before you reply, verify the Output file for this step exists and is non-empty. If it does not exist, do not report success: carry out the work yourself and confirm the file is written.

Qwen3-Coder's scan then passed, and GLM's still did. Per-model fixes are a slippery slope: one skill becomes several that nobody can maintain, and the comparison stops being fair. So any context we add goes to all models. To port a skill properly, Hugging Face's upskill builds one from a frontier model's traces and measures it on each model you care about.

2. The infrastructure: "not measured" is not "failed"

With a hosted API, the provider owns the gateway, the load balancers and the credentials. When you self-host, you own them, and their failures show up in the agent's log as model failures. In one early run, an expired credential failed two working models outright.

Qwen3.8's runs on our two largest planning tests ended without a plan, which looked like a model that couldn't handle big inputs. The agent's log said only:

[Step=Plan Fixes]: API Error: Connection error

[Step=Plan Fixes]: API Error: Connection error

[Step=Plan Fixes]: API Error: Connection error

We put a logging proxy between the agent and the gateway and measured time to first token (TTFT). Qwen3.8 reasons before it answers and sends nothing while it does, so on a large plan its TTFT ran past a minute. After exactly 60 seconds of silence the connection dropped, and so did the retry. MiMo's TTFT is a few seconds, so it never came close. We replayed the same planning request at the gateway several times under each setting:

With a reasoning model, TTFT is a timeout risk as well as a latency number.

Suspect a round number. Run times vary, so a cut at exactly 60 seconds, run after run, is a timeout. Ours were stacked, and each fix exposed the next. First, our Modelplane config: a ModelService's idle timeout, which bounds TTFT, defaults to 60 seconds and its request timeout to 300, while a large Qwen3.8 plan can reason for longer. We set both to 30 minutes (docs). Second, the cloud load balancer in front of the gateway dropped silent connections after 60 seconds. With that raised, a request failed at 239 seconds: four 60-second attempts against a second load balancer, because the gateway retried the whole generation. With all of them raised, the connection errors stopped.

One more stall came from the agent itself. Before each shell command, Claude Code runs a quick check with a small helper model. We had pointed it at the model under test, and with Qwen3.8 that check sometimes hung until the 25-minute trial limit. Self-hosted runs now use MiMo as the helper.

Record why each run ended: the model finished, ran out of time, or the infrastructure cut it off. Only the first two count toward the pass rate. When a self-hosted run fails, check these before you blame the model:

3. The grader: check the outcome, then check the checks

Grade the result, not the paperwork. A format check passes the v0.54.0 plan from the top of this post, and only a grader that computes the required version catches it. Even a correct plan can leave the CVE open. MiMo once planned a gRPC upgrade to 1.82.2, the version the scan named, but the module ended up on 1.83.1, a release line where the fix is 1.83.2. Only a second scan caught it. Sonnet 4.5 and Qwen3.8 made the same mistake, so the trap is in the task, not in open models. The AI Engineering from Scratch guide puts it this way: "Check the state, not the agent's story."

Grade your graders. If the grader is wrong, anything you tune against it learns the error. We run ours on the CVE fix pull requests that people merged, plus copies corrupted three ways: lower versions, a dropped package, and a claim that the CVEs can't be fixed. The originals must pass and the copies must fail. The first run let most of the "can't be fixed" copies through, which exposed a check that had never run. The same history gives the target: about 85% of those fixes merged without edits, so an open model needs to match 85%, not 100%.

Suspect the grader when the trace looks right. MiMo's fix for the vulnerable module failed with "no findings", yet its scan, plan and fix were correct. It had re-run the scan to check its work and saved the result over the original report. Our first fix for that had a bug of its own: it counted any mention of the file as a write. Failures then clustered on that one check, and none was real. Another check read MiMo's "all fixable, not 'unfixable'" as a refusal. The same signs came up each time: the trace shows correct work, failures cluster on the newest check, or the check reads prose rather than structure. So each grader fix starts as a failing test built from the trace that exposed it, next to a test that the check still catches a real mistake. We treat each fix as a new check and suspect it the same way.

Give each test a hidden negative control. If the agent skips the bundled scan script, the scanner reports nothing, which is also what a correct scan of a clean module reports. So each scan test pairs a module with known CVEs, which catches the skipped script, with a clean one, which catches invented findings. The controls also have to be invisible, because models notice when they're being tested. Sonnet once quoted our clean module back to us:

as noted in the go.mod comment stating it's the negative control for the CVE remediation pipeline

MiMo read a similar comment in the vulnerable module and flagged its own fix for review. Both are now ordinary modules, and our notes are in a file the agent can't read.

What the models did

In the first round, the models found the CVEs easily. Across eight modules with 50 known CVEs, Sonnet, GLM-4.7-Flash and Qwen3-Coder each found all 50 and reported no false ones. Planning split them, in opposite directions. GLM wrote the v0.54.0 plan from the top of this post in all three trials. Qwen3-Coder wrote no plan at all on three of four modules. An empty plan trips the pipeline's guardrail and stops the run, while a wrong plan becomes a pull request. So for this agent, a model that stops is safer than one that is confidently wrong, even though it scores lower.

The latest round ran each of the 13 scan, plan and fix tests three times per model:

• Harness: Inspect, running the agent's own container image.

• Scanner: Grype 0.119.0 with one pinned database snapshot (v6.1.10).

• Models: MiMo-V2.6-Flash and Qwen3.8-27B on Modelplane, Claude Sonnet 4.5 through Anthropic's API, each at default sampling settings.

• Serving: vLLM. MiMo is split across two RTX PRO 6000 GPUs, and Qwen3.8 runs in FP8 on one L40S.

• Outages: a run that the gateway interrupted is marked "not measured" and run again, not scored as a model failure.

The gRPC trap above caught all three models: Sonnet 4.5 in all three trials, and MiMo and Qwen3.8 in one each. Sonnet's other miss bumped Go to 1.25.13 when the advisories needed 1.26.9. MiMo's scan miss asked a clarifying question in a run with nobody to answer it. Qwen3.8's other misses all hit our time limit on the two large modules, so its scores measure speed as well as accuracy. With 12 to 15 runs per step, the uncertainty is still wide: at 95% confidence, 12 of 12 means 76% to 100%, and 8 of 12 means 39% to 86%. So these numbers show where each model breaks, not how the models rank.

This suite scores accuracy, meaning whether the agent found, planned and applied the right fix. It counts speed only when a run times out. We measured speed anyway because it decides where a model is usable. We replayed the same large planning request against each model five more times:

Qwen3.8's TTFT varies widely: across all our replays of this request, it ran from 11 to 166 seconds. Its reasoning tokens arrive in a burst with its answer, so its speed is output tokens over the whole request, not over the time after the first token.

Qwen3.8 reasons at length and runs on one L40S. On the two large modules, its planning runs took 6 to 19 minutes when they finished, against 3 to 10 for MiMo and about 2 for Sonnet 4.5. That's fine for a build agent that runs unattended, and not for an engineer waiting on it. We are upgrading the cluster, and we have not yet tuned serving for latency, so we expect these times to come down.

What's next

We add tests as we read failures. Next, mostly from Hamel Husain and Shreya Shankar's evals FAQ:

• Harder tests. They suggest a pass rate around 70%, so the set can still tell models apart. All three models score 93% or more on our scan step.

• pass^3 as the headline. We'll lead with pass^3, the chance that all three runs pass, rather than pass@3, the chance that one does, because a build agent runs with nobody there to retry it. In this round, Sonnet 4.5 and MiMo passed 11 of 13 tests in all three trials, and Qwen3.8 passed 9.

• Error analysis as a habit. Sorting failures into categories showed that Qwen3-Coder's are almost all "no output produced" and GLM's are almost all "fix falls short", which a single accuracy number hides.

• Real work. Replay the merged fixes against MiMo.

• Bigger models, on a bigger cluster. GLM-5.3-Flash alone is 321B. The same suite can tell us whether size closes the gap on large modules.

• Repeatable runs. A batching server can return different tokens for the same request, even at temperature 0. We'll try vLLM's VLLM_BATCH_INVARIANT=1 on Modelplane.

• Judges only where code can't decide, such as release notes, each checked against 50 to 100 hand-labelled outputs first.

Next up: use case #2, the agentic coding harness. We want our engineers to run Claude Code and other coding harnesses on our self-hosted models for their daily work. No scanner can say whether that work came out right, so it's harder to grade.

Moving an agent from a hosted model to a self-hosted one? Start here:

• Write down what "done" means. Pick a few tasks you already check by hand, and give each a pass or fail check on the end state, not the agent's reply.

• Run against your cheap hosted model first. It's your baseline and your harness test. If this model fails a test, fix the test.

• Swap only the model. Keep the agent, prompts and data the same, and pin the data. If the new model needs extra context, give it to every model.

• Raise the timeouts before the first run. Open reasoning models can think for minutes before the first token. Set every timeout between the agent and the model above that, including the agent's own helper calls.

• Read failures before you count them. A cut at a round number of seconds is infrastructure, not the model. Keep those runs out of the pass rate.

• Check your checks. Start with deterministic checks, and test them against answers people already accepted or rejected. Add an LLM judge only where code can't decide, and test it against the same human answers first. Keep the tests where the model can't read them.

• Run each test more than once. One run shows where a model breaks. A few let you say how often.

In these small runs, MiMo did about as well as Claude on the steps we can grade, and most of Qwen3.8's misses came from running out of time. We trust that result more than we would have at the start, because we checked the harness, the infrastructure and the grader before we believed it.

If you run your own inference, star Modelplane on GitHub and tell us what breaks.

Further reading

• Anthropic, Demystifying evals for AI agents. The eight steps, and the advice to read transcripts, which is where most of what we found came from.

• Hamel Husain and Shreya Shankar, AI Evals: Everything You Need to Know. Error analysis before infrastructure, code checks before LLM judges, and pass or fail before 1-to-5 scores.

• Inspect, from the UK AI Security Institute.

• AI Engineering from Scratch: Evals and Benchmarks. How to read a public score, and how to build your own eval, with intervals and judge calibration worked through.

• Hugging Face, upskill. Builds agent skills from a frontier model's traces and evaluates them on other models.