From Benchmark to Router: Cost-Aware Model Routing Across Twelve Lanes

A benchmark is a leaderboard. A router has to act on it.

My field-test benchmark left me with fourteen model lanes and a matrix of how each scores per task type. That matrix answers "which model is best." At dispatch time the question is narrower and harder: this task just arrived, which one of twelve routable lanes runs it, right now, with no human in the loop?

Sending every task to the top of the leaderboard is the obvious answer and the wrong one. It is the most expensive lane on every task, including the trivial ones where six other lanes score within a rounding error.

So I built a cost-aware router that turns the matrix into a per-task choice. This is how it works, how I set it up, and where it sits next to the public LLM model routers that solve a related but different problem.

The cost-aware routing principle: cheapest lane above the capability bar

The rule is one sentence. For each task, take the cheapest lane that clears the measured quality floor for that task type, with enough margin that I am not buying rework.

Rework is the expensive failure mode, not the slightly pricier lane. A lane that costs a fraction of a cent more but lands the task first time is cheaper than a free lane that produces something I throw away and re-run.

So the router never looks at price before capability. It holds a capability bar first, then reaches for the cheapest option that still clears it.

That rule has two halves, and the router implements both literally: a floor that filters, and a cost-aware sort that picks.

What the public routers optimize, and what they skip

This space is not empty. RouteLLM (UC Berkeley and Anyscale, ICLR 2025) trains a classifier on preference data to route between one strong and one weak model, and reports roughly 2x cost savings at 95% of GPT-4 quality by sending only about 14% of queries to the strong model.

The OpenRouter Auto Router, powered by NotDiamond, exposes a single cost-quality dial from 0 to 10. Martian and Unify do their own real-time selection per prompt.

These are good at one thing: estimating query difficulty and trading a strong model for a weak one to save money. Two properties they tend to share, and that did not fit my setup.

They route on difficulty, not task type, and they learn it from a generic preference dataset. My ordering changes with the kind of work, and it comes from my own field tests on real coding tasks, not a public set.

And they are mostly black boxes. The selection happens, you pay the bill, and there is no per-decision record of why this model ran this task. For a governed setup that is a missing audit trail.

The difference in my router is not cleverness in the picking. It is the governance around it: a constraint filter and a receipt on every choice.

How it works

The router lives in smart_router.py. A dispatch arrives as an instruction string plus tags. Five things happen before a model runs.

1. Classify the task

The instruction is matched against seven task classes with heuristic regex, then a role-based fallback, then a default of 01_code_generation. Tags do not feed the classifier; they only re-rank afterwards.

The classes: 01_code_generation, 02_code_review, 03_refactoring, 04_documentation, 05_debugging, 06_design, 07_translation.

"Debug the flaky test in the auth module" lands in 05_debugging. "Review the PR for security issues" lands in 02_code_review. Phrase the first one as "Fix the flaky test" and the regex misses it: it falls through to the default. That is the weak point I come back to below.

The class matters because a lane that tops code generation is not automatically the one I want reviewing a security change. The benchmark showed the ordering genuinely shifts per task type.

2. Load the ranked candidates for that class

Each class has its own ranked candidate list, populated from the field-test results. A candidate carries the numbers the router needs:

The composite_score here is the router's own 0-to-10 scale aggregated from the field tests, not the 0-to-5 composite in the benchmark write-up. Same data, different normalization, so do not read the two side by side.

3. Apply the floor

Two filters, in code.

The hard floor is capability. A model scoring below _CAPABILITY_THRESHOLD (7.0) for a class falls into a sub-bar band. It only runs when nothing in that class clears the bar, however cheap it is.

The soft floor is the quality tier. Each class can declare a min_quality_tier. Code review, design, documentation and translation require tier 3, the premium band. Debugging requires tier 2. Below the declared tier a lane does not qualify for that class even if its average looks acceptable. This is where "with margin so I am not buying rework" becomes a config value instead of a hope.

The tier itself is derived, not hand-set: a score of 7.5 or higher is tier 3, 5.0 or higher is tier 2, and anything local or free is locked to tier 1 regardless of score, a deliberate conservatism about lanes I cannot meter.

4. Pick the cheapest capable lane

Among the lanes that clear the floor, the sort is the whole point:

Lanes that clear the bar rank by cost ascending, score descending as the tiebreaker. A cheap-and-strong lane beats an expensive-and-stronger one; a cheap-and-weak lane can never beat one that clears the bar. An unknown cost sorts last within the band, so an unmeasured lane is never treated as free.

5. Filter the constraints, then write the receipt

A constraint filter runs over the survivors. The router consults provider_constraints.yaml and drops any lane that would violate a blocking rule before it can ever be recommended. Kimi only through its own CLI, GLM only through OpenRouter, DeepSeek only on its own key, never the Anthropic SDK in the code path. These are account-safety and cost-trail rules, and the router treats them as a hard gate, not a preference.

A blocked lane is removed and the constraint code that removed it is recorded on the decision. The filter fails open: if the check itself errors, the candidate is kept rather than silently dropped. And a recommendations file that names a model a blocking rule forbids outright no longer loads at all: the router raises at load time instead of filtering it silently.

The output is a RouteDecision: the chosen primary, a fallback, the reason, the constraints applied, and a cost estimate. It is appended to route_decisions.ndjson, file-locked, with a per-dispatch JSON alongside, wired into provider_dispatch.py and subprocess_dispatch.py behind the opt-in --auto-route flag, and folded into the dispatch receipt.

Months later I can answer "why did this task run on that model" with a record, not a guess. That is the same glass-box principle I hold the agents to, turned back on the router. A routing layer that cannot explain its own choices is a heuristic with good PR.

How I set it up

The router reads its rankings from one file, routing_recommendations.yaml, populated from the field tests. A trimmed entry:

Three setup decisions did most of the work.

The GLM lesson is baked into the runner field. The benchmark's cleanest result was that GLM-5.2 through the full harness is a different model from GLM-5.2 as a flat tool-call. On code generation the harnessed lane scores 8.15 against the flat runner's 5.92. So in the config the harnessed runner carries GLM's ranking and the flat runner sits below it.

The router routes GLM through the harness because routing it any other way is buying the rework I already measured. The recipe for that proxy is its own how-to.

Review and design are floored at tier 3. These are the tasks where a plausible-but-wrong answer costs the most, so I do not let the router save money there.

The min_quality_tier: 3 makes the floor non-negotiable. Debugging sits at tier 2: it needs competence but tolerates a cheaper capable lane.

Launch reliability rides next to the score. The benchmark taught me that capability and reliability are different axes. DeepSeek-flash scores 9.02 on code generation, fourth of twelve, yet launched only 44% of the time on that class in the field tests (47% across all runs).

So launch_success_rate sits in the candidate data, and a low one is the signal that this lane needs a fallback wired, not a lane to trust blind on a critical dispatch.

In practice a code-review dispatch classifies as 02_code_review and the tier-3 floor keeps only lanes above 7.5. Without my own DeepSeek key the constraint filter drops the DeepSeek lanes, and the sort returns GLM-5.2 through the harness as primary with Kimi K2.7 as fallback. With an own DeepSeek key, DeepSeek-flash is the cheapest capable lane and becomes primary.

A code-generation task returns GLM-5.2 through the harness because it is the cheapest lane above the bar, even though Claude Sonnet scores higher. Same router, different answer, because the matrix underneath says the ordering changes with the work.

Read also: Quality Escalation Is Not a Fallback

The limits

This is a real system, so the limits are real.

A regex classifier can misroute, as the "Fix" versus "Debug" example shows. This is the known weak point of routing in general: classifier-based routers often fail to beat the single best model when data is thin, and they overfit to whichever model won the training set.

My answer is the floor. A misclassified task still has to clear a capability bar for whatever class it landed in, so a wrong class degrades the pick, it does not hand the work to an incapable lane.

Silent quality regression is the failure mode that does not show on a dashboard. Route to a cheaper model and the answer can degrade in a way that surfaces as a support ticket days later.

The floor-with-margin is the prevention. The receipts are the detection: when a result looks wrong, the route decision says which lane ran it and why.

The scores come from a high-signal, low-volume benchmark. Some per-class cells rest on a handful of replications, and the Sonnet and older Opus scores come from the previous model generation without a re-run. So the rankings are a strong prior, not gospel. I treat the recommendation file as something to refresh when I re-run the field tests, not a constant.

The version of "self-tuning" I trust is that the benchmark proposes the new ordering and I approve the diff, the same human-in-the-loop gate I keep everywhere else.

And routers themselves are an attack surface. The research on rerouting attacks shows adversarial inputs can manipulate model selection to inflate cost or lower quality. The constraint filter caps the blast radius (a manipulated request still cannot reach a blocked lane) and the receipts make the manipulation visible after the fact. Neither is a full defense, and I treat it as open.

The other half: the gate

Routing is half of quality control. The router picks who does the work. Since August a second resolver decides who checks it: it derives the weight of the review gate from the task class and the paths a change touches, so a heavy adversarial review lands on a feature and a security change rather than on every trivial PR.

An explicit gate on the dispatch spec always wins. The router made it obvious why this belonged next to it: the same field-test data that ranks a lane for doing a task ranks it for reviewing one.

The shape worth copying

The pattern generalizes past my twelve lanes. Classify the task, hold a measured floor per task type, pick the cheapest lane that clears it, enforce your hard constraints as a filter and not a footnote, and write down why.

Skip the floor and you optimize straight into rework. Skip the receipt and you cannot defend a single choice when one looks wrong at two in the morning.

The router, the constraints and the field-test methodology are open source in Vinix24/vnx-orchestration. The benchmark it is built on, with the full matrix and the lane-hardening story, is in one harness, five model families.