Jev vs Laya: Typed Decision Models Tested on Real Work

When TypeSafe AI shipped Jev in September, my feed filled up with the same numbers: 200x faster, 400x cheaper, calibrated confidence on every answer. Simon Willison wrote it up as a new shape of LLM. Within days there were explainers on what Jev is.

Almost none of them ran it on their own work. So I did. And because a local, open-weight model with the same answer shape exists, I ran that one too.

This post is the result: one classification task, a regex baseline, four rounds for Laya, one live run for Jev, and a cost check across the rest of my own projects.

What these models are

Jev and Laya do not generate text. You give them a document and a set of typed questions: pick one of these categories, score this from 1 to 5, yes or no. They return numbers with a probability attached. TypeSafe calls them System One models. Willison prefers "decision models", and so do I.

Jev is TypeSafe's hosted model. I called it through OpenRouter at $0.042 per million input tokens, with output free. There are no open weights.

Laya is an open-weight decision model from Convai Innovations, built on ModernBERT-family encoders. The multilingual checkpoint I used sits on mmBERT. I ran it through unofficial community ports for Apple Silicon, one for the Neural Engine and one for MLX. It runs on my Mac, with no key and no cost per call. The port's author writes that its validation is "not proof of general task accuracy".

Same idea, opposite trade-offs: a vendor in the cloud against a model that never leaves my machine.

The test: 671 findings from my own review gates

My quality gates produce findings on every change an agent makes. I wanted to know how they split into style, correctness, security and policy, and whether my reviews are too strict.

The baseline is a keyword regex, the kind of fixed rule I already use everywhere because it costs nothing.

The ground truth needed care. I had no time to label 671 findings by hand, but one group labels itself: every finding about my 70-line function limit is a style finding by construction. That gave 130 cases, or 116 with a stricter selection. I fixed the pass criterion before running anything: the model has to beat the regex on that group. A tie counts as a fail. In the script I set the bar at 69%, because I believed that was the regex score. It turned out to be 62 to 64%, so the bar was higher than planned.

Laya: four rounds, and it still failed

I gave Laya four chances, because a negative result deserves the same second look as a positive one.

Round 1 measured nothing. Two counting bugs: a nested walrus zeroed the distribution, and a dict collision silently dropped 39 duplicate ids. The label language was also tied to the checkpoint.

Round 2 ran on the multilingual Neural Engine build, now with English labels. It classified 4 of 124 style findings correctly: 3.2%. Almost half of all 657 findings did not fit its 96-token window at all.

Round 3 added the category definitions to the prompt. That cost 21 of the 96 tokens and pushed capacity errors to 88%. Result: 1.6%.

Round 4switched to the MLX build with a 1,024-token window, full definitions and zero capacity errors, at 30 ms per decision. Result:19.4%(24 of 124), against62.1% for the regex on the same set. It labelled 57 of the 124 style findings as security. That is with Dutch labels. With English labels it did worse.

The 96-token window comes from how the port was compiled for the Neural Engine. Give it room and it gets better, but nowhere near a keyword list.

Read also: Multi-AI Code Review at the Merge Gate: 28 Codex runs on 14 PRs, an earlier model test at the same gate

Jev: 88.5%, in one run, for two cents

Jev ran live on 22 September on 671 findings, with five classes including "other" and the same rubric text as Laya's fourth round.

All 671 calls succeeded. The whole run cost $0.0195. It passed the criterion I had set in advance.

The two corpora are not identical: 671 findings and five classes for Jev, 657 and four for Laya, because the store grew between runs. Jev and Laya's fourth round got the same rubric. A gap of 88% against 19% does not come from that difference.

What the 88% does not tell you

This is the part the explainers skip.

It covers one class. The only labels I had were for style. The rubric for style literally mentions "function size (70-line threshold)", which is how I selected the ground truth. That makes it the easiest group in the set. The 88% says nothing about correctness or security findings, which are the ones that matter.

Jev is overconfident. On the ground-truth group the median confidence was 1.000 while 88.5% was correct. For the calibration curve I used all 144 labelled rows, because some findings appear twice in the extract: 87.5% correct, expected calibration error 0.10. TypeSafe promises that higher confidence means higher accuracy. The direction holds, the top bin is the most accurate. But 131 answers at 0.9985 average confidence were 93.9% correct, so near-certainty still hides about one error in sixteen.

But low confidence does point somewhere. Of the 18 errors in those 144 rows, 10 sat below 0.9 confidence. Below 0.9, 10 of 13 answers were wrong. So the number is not useless. It flags some of its mistakes and hides the rest at near-certainty.

I cannot reproduce it from disk. The raw Jev output is gone; only the report with the numbers survives. Anyone who wants to check this has to run it again, and so do I.

I never measured Jev's latency. The vendor claims 70 to 500 ms. I have no number of my own.

Where Jev did help: finding what my rules miss

The second test was on my own writing. I run a deterministic linter that flags "not X but Y" constructions and other tics. I asked both models the same yes/no question on 120 of my own sentences, half with the frame and half without: does this sentence contain that frame?

Laya could not separate them. Its best F1 was 0.678, against 0.667 for flagging everything. In seven of eight setups it was more confident on the clean sentences.

Jev separated them, but only just: a median score of 0.58 on sentences the regex flagged, 0.42 on the rest. Far too narrow to use as a gate.

What made it worth $0.0022 was the other end. Eleven sentences the regex had passed scored above 0.6. Three of them were real misses: a three-part list, a detached "not because", and the English "not just". I turned those into deterministic rules. Across my published blogs and LinkedIn posts they catch 98 extra cases.

That is the role I kept. The model may decide which tool I build. It may never decide what the tool finds.

The cost argument, checked against my own projects

Before any of this I did the math for four systems I build or run.

• My business cockpit: 4,512 small model calls in 28 days cost $11.64. With Jev: $0.36. That is 32x cheaper, not 400x, and it saves $147 a year.
• My sales copilot: 87.2% of the work is already handled by exact triggers and embeddings before any model gets involved. The first layer runs locally, where a hosted model cannot go by definition.
• My SEO tool: Jev would replace a field, not a call. No saving.
• A client project: fixed rules already remove most of the input for free. No case.

The 400x may hold on the vendor's own evaluations, but it is measured against sending everything to a large model. None of my systems do that anymore. The expensive layer was already gone. What is left is paying for a new vendor, a key and an integration to save an amount that disappears in rounding.

When I would use it

Yes: a closed classification with a clear rubric, where an error is cheap and visible. Or as a scout that points a deterministic rule at the variants it misses.

Not yet: anywhere the reason matters. A probability gives you a number and no reason. If a person downstream needs to see why, like a client reading a score, a quote from the source beats 0.73 without context.

Not at all: as the gate itself, as the thing that writes to a record, or where nobody will ever read the probability.

I would also rather have more errors that I can see than fewer errors that I cannot. My regex fails loudly and in the same place every time. Jev fails less often, at 0.9985 confidence. Finding those is the expensive part, the same trap as monitoring that watches the wrong thing.

Still untested: the other classes, Dutch business text, latency, and a guardrail that predicts whether a task will fail before it runs. That last one is the case I think this model shape was made for.