Jev vs 25 Lines of Python: Typed Decisions Tested on 450 Dutch Texts

I tested Jev earlier this week on my own code review findings and it scored 88.5% against 63.9% for a keyword rule. A few days later an Instagram reel made a sharper claim: Jev is 25 lines of Python. The idea traces back to a post that calls itself a parody, but the method is real. Take any language model, make it answer with one letter, and read the probabilities from the logprobs.

That claim is testable, so I tested it. This time on three tasks, with 450 Dutch texts, seven approaches and the analysis fixed before I ran anything.

The short version: on the hard cases I could not show a difference between Jev and the 25-line version. What I could show is where both of them failed, and that says more about the questions than about the models.

Read also: Jev vs Laya: Typed Decision Models Tested on Real Work, the first round on my code review findings

What "25 lines of Python" means

Jev by TypeSafe is a decision model. You give it a text and typed questions: pick a category, give a score, answer yes or no. It returns numbers with probabilities.

The DIY version does the same with a general model. You list the options as letters, ask for exactly one token, and read the probability of each letter from the logprobs. A score works with digits, yes/no with the words "ja" and "nee". It fits in a short script, and I ran it on two Qwen models through OpenRouter.

There is one practical difference. Jev answers all questions about a text in one call. The logprobs trick reads a single answer token, so every question needs its own call.

The setup

Three tests of 150 texts each, all fictional and anonymous, in Dutch:

• Helpdesk emails to a webshop: category out of six, urgency 1 to 3, and whether a human should take it.
• Search queries for a bike shop: intent out of four, topic cluster out of eight, commercial value 1 to 5, and whether it is a brand search.
• Sales call fragments: buying signal or type of objection out of six, buying readiness 1 to 5, and whether a decision maker is mentioned.

Each test has three tiers of 50. Easy: the signal word is literally there. Medium: the same meaning without that word. Hard: constructions that trip a rule, like negation ("not broken, the invoice is wrong"), two topics, sarcasm, typos, implicit intent and long texts with the point at the end.

Seven approaches:

Fixed in advance: the classes and definitions, the metric (macro-F1, which weighs every class equally so a model cannot win by overusing one), the thresholds, and a budget of $2. The whole run cost $0.94.

The result on the hard tier

Macro-F1 on the category question, hard tier, 50 texts per test (for search: the average of intent and topic cluster):

Every general language model beats the rule by a wide margin once the signal word is gone. Laya is the exception: on the hard tier it scores below the rule in all three tests. On the easy tier the rule still holds up: 0.93 on helpdesk emails. On the medium tier it drops to 0.15.

Between Jev and Qwen3.7 Flash I found no demonstrable difference. The gaps are +0.00, +0.02 and +0.02 in Jev's favour, and none of the bootstrap intervals excludes zero. With 50 texts per tier that is all this data can say. The hard helpdesk tier sits at the ceiling: four models score 1.00. A test that everyone passes cannot separate them.

Jev was better at scores. The error on urgency was 0.07 points against 0.14 for Qwen3.7 Flash, and 0.25 against 0.34 on buying readiness. I did not set that as a threshold in advance. Read it as a description.

Jev was faster per text, because one call answers everything. The median was about 300 ms per text with all questions, against about 2 seconds for Qwen with three calls in a row. Sent in parallel, the Qwen calls would take an estimated 0.8 seconds. I did not measure that.

Cost was not the argument. Jev came to $0.011 to $0.016 per thousand decisions, Qwen3.7 Flash to $0.006 to $0.010.

Calibration: no difference either

A decision model promises that its probabilities mean something. On these tasks they did, for Jev and for Qwen: an expected calibration error of 0.015 and 0.018. When either said 99%, it was right 98 to 99% of the time.

That is better than what I measured for Jev on my code review findings last week (0.10). Calibration depends on the task, and I would not carry one number over to another job.

Laya is the outlier. Its calibration error is 0.278. In the bucket where Laya claims 98% confidence, 56% is correct. On the helpdesk task it mostly chose "return" and "delivery" and never "account". On the ten long texts it got zero right, because it cuts the text at its 1,024-token window and the point was at the end.

Two things that did not help

Thinking. The same Qwen model with thinking on scored +0.00, +0.05 and -0.03 against thinking off. It was 6 to 8 times slower and 11 to 15 times more expensive per decision. Eight texts got no answer at all, because the model was still thinking when it hit the token limit.

Rule first, model as fallback. This sounds like the thrifty design, and I run versions of it myself. Here it lost to the model alone by 8 to 37 points on the hard tier. The reason is in the rule. On the hard tier it finds something in two out of three questions, because the misleading texts contain the signal words, and then it is wrong one time in three. The model never gets to see those texts.

A rule that fails silently in front of a model is worse than no rule. A rule works as a filter only where you know it is right: literal signals in text you control. That is also how I run the deterministic gates in front of my agents.

Where both of them failed

This is the part that changed how I read the result.

Jev, Qwen3.7 Flash and Qwen3.6 35B-A3B made 40, 44 and 48 mistakes on the category questions. On 21 texts all three were wrong. And 31 of the 33 intent mistakes on the search queries were on the same class: navigational.

Look at what those queries were. "forum voor stroomfietsers" (forum for e-bike riders). "morrow damesfiets met lage instap prijzenpagina" (a brand's price page for a step-through bike). By my definition that is navigational: the searcher wants to reach a specific page. Three models read it as informational or commercial. So would many people.

When three different models make the same mistake on the same item, I look at the question first. My definition of navigational was less clear than I thought when I wrote it.

The leverage is in the questions

A working note on Jev engineering has been going around. It is an independent synthesis of design notes attributed to TypeSafe's founder, not an official document. The line that stayed with me: in a coding agent, "the loop is not where the leverage is". The note argues that every useful feature is a typed question asked at a decision point: which context should the next step see, should this command run, which tool fits?

My benchmark is a small version of the same point. Among the general models, who answered mattered less than I expected. Jev and a general model through logprobs landed within noise of each other. What decided the score was which questions I asked and how sharply I defined the answers. All three hit the same wall on the one class I had defined loosely.

So when I choose between these tools, I now start somewhere else:

• Write the question and its definitions first, and test them on people or on a second model before you blame the model.
• Keep a rule where the signal is literal. It is free, instant and fails in the same place every time.
• Use a cheap general model through logprobs if you already have a provider and ask one or two questions per text.
• Consider Jev if you ask several questions per text, want scores with probabilities out of the box, and prefer one call over wiring up a parser.
• Skip Laya for Dutch text like this, at least in the configuration I tested.

What this test does not prove

• One model family wrote and checked the texts. Subagents of one model wrote all 450, the same model reviewed them, and my orchestrating session corrected 22 labels. A second model (Kimi) labelled a sample of 60 blind and agreed on 48. None of the tested models came from that family, but texts written by one model can suit some readers better than others.
• The navigational label is contested, as shown above. Part of what looks like model error is probably my definition.
• The hard tier was not hard enough. On the helpdesk task it fooled the rule and none of the general language models. Laya, the exception, failed across all tiers. A difference of five points does not show up with 50 texts per tier.
• One run per approach, at temperature 0 on one pinned provider each. Provider quantization can shift results slightly on a rerun.
• Latency was measured sequentially from one machine.
• Three Dutch tasks. This is not a general ranking of these models.