Vincent van Deth 

Your monitoring watches the wrong thing more often than you would like, and here is how I found out. I dispatched four workers in parallel and set a monitor to wait for the processes to exit. An hour later I found out that three of the four had finished roughly thirty minutes earlier. The pull requests were open. The work was done.

The processes had not exited. These workers do not terminate when the work is finished; they sit idle. Process death was the wrong completion signal, and it arrives late or never.

I lost an hour to a monitor that was working exactly as designed.

That is the first of five, all from the same eight-week window, all in different systems, all with the same shape: **the monitor watched something adjacent to what I cared about, and the adjacent thing was fine.**

## 1. Liveness instead of completion

The parallel workers case above. The fix is instructive because the correct signal was not a better process check.

The signal that means "done" for this system is a pull request appearing. That is the artifact the work produces. Process disappearance became a secondary confirmation, and a fifteen-minute stretch without CPU activity became a backstop for the case where neither fires.

Three signals, ordered by how directly they mean the thing. The one that is easiest to check (`kill -0`) turned out to be the weakest, which is why it was there in the first place.

**The general form: your completion detection has to sit on the signal that actually means finished, not the signal that is easiest to observe.** Those are rarely the same, and the gap between them is invisible until the day the process does not exit.

## 2. Presence instead of count

An activation script silently doubled an entire timer fleet. 54 timers firing twice: duplicate scrapes, duplicate syncs, duplicate scoring runs. Everything downstream got each input twice.

Not one monitor fired.

The monitoring checked whether each job was running and whether its data was fresh. Both were true. More true than usual, in fact, since everything ran twice as often.

"Is it running?" and "is it running exactly once?" are different questions, and I had only ever asked the first. Deduplication brought 147 timers back to 92.

This one generalizes further than it looks. Any monitor built on presence has this hole: it cannot distinguish healthy from too much. Queue consumers, cron jobs, webhook handlers, retry loops. If your check is "at least one is alive", a runaway multiplication reads as excellent health.

Five monitoring failures and the signal each one should have been watching

## 3. The database returning NULL as the maximum

A freshness monitor sat red for weeks. Last run timestamp: null. The pipeline it was watching was running fine, 58% of records tagged, most recent tag fresh.

The freshness query did `ORDER BY tagged_at DESC` without `NULLS LAST`. Postgres defaults to `NULLS FIRST` on a descending sort. With roughly 3.900 untagged rows carrying a null timestamp, null came back as the maximum.

The monitor was correct about what it measured. It measured the wrong thing, because SQL's default ordering semantics did not match the intuition of whoever wrote the query, which was me.

This is a false red rather than a false green, so it is the cheap version of the failure. It still cost weeks, because a monitor that is red for weeks trains everyone to stop looking at it. **A permanently red alarm is functionally identical to a disabled one**, and the disabled one is at least honest about it.

Fix: `nullsfirst=False` and a regression test. PR #601.

## 4. The tripwire on the monitor that also failed silently

This is the one that still bothers me.

I have a producer freshness monitor. It stopped running for 68 hours, status `stale`, with 11 open findings from its last run sitting there unread. Its expected interval is 24 hours.

There is a tripwire whose entire job is to fire when that monitor goes quiet. It works. I ran it by hand and it produced exactly the right alert.

It was wired only as a session-start hook. So it fires when I start a session and never otherwise, and the symlink it depended on was dead anyway. Beyond that: **there was no scheduler for the monitor itself.** No timer, no cron entry. Nothing was responsible for running the thing that was supposed to run.

The header comment of that tripwire file reads, verbatim: *a monitor that detects silent failure can itself fail silently.*

I wrote that line. I understood the failure mode well enough to document it at the top of the file. Then I wired the detector to a trigger that only fires when a human shows up, which is precisely the condition under which you do not need an automated detector.

The lesson is not "monitor your monitors", because that recurses forever. The lesson is narrower and actionable: **a detector that depends on the same infrastructure as the thing it watches is not independent.** The dead symlink took out both. If the monitor and its tripwire share a failure mode, you have one monitor with extra steps.

## 5. Freshness checked by the writer instead of the reader

The most recent one, and the one that had not shipped yet, because a review caught it.

I had a plan written for monitoring that measures whether systems do their work rather than whether they answer. The review found four failure modes in the plan, all of which the plan itself was supposed to eliminate.

**The freshness check lived in the writer.** Each run wrote its own timestamp and its own status. Which means a dead timer leaves the last "ok" in place, slowly aging, and nothing notices. Freshness has to be evaluated by the reader, at read time, against the clock. Otherwise you are asking a process that is not running to report that it is not running.

**The ingest metric aggregated across both mailboxes.** One dead connector stays invisible as long as the other one is green. Aggregation across independent sources hides the failure of any single source, which is the entire point of splitting them.

**The escalation clock lived in the file that gets overwritten every run.** So the escalation state reset on every cycle and nothing ever escalated. State that must survive a run cannot live in the run's own output.

**Five of the seven metrics were red on day one**, because the systems being measured were not fully bootstrapped yet.

That last one is the most human failure of the five. An alarm that goes off from day one teaches you within a week to ignore it, and then it is dead for the specific outage it was built to catch. The fix was a per-metric bootstrap acknowledgement with an explicit end date, so the noise is deliberate, bounded and visible instead of permanent.

## The test that catches all five

There is one question that would have caught every failure above, and it takes ten seconds per check.

**What would this monitor report if the thing it watches were completely broken?**

Run it honestly for each one:

- Process-death monitor, workers hung forever: reports "still running". ❌
- Presence monitor, timers duplicated: reports "healthy". ❌
- Freshness query, pipeline dead: reports null, which reads as red. ✓ by accident
- Tripwire, monitor dead and symlink broken: reports nothing at all. ❌
- Writer-side freshness, timer dead: reports the last known "ok". ❌

Four out of five stay silent or report health during the exact failure they exist to catch.

If the answer to that question is "the same thing it reports now", you do not have a monitor. You have a status indicator with no input.

## What I changed

Three rules, applied to every check I have written since.

**Measure the artifact, not the actor.** A pull request appearing, a row landing in a table, a file with the right size. Something the work produces, not something the worker does.

**Evaluate freshness at read time.** The reader compares the last recorded timestamp against the clock. Never let the process report on its own aliveness, because the failure you care about is the one where it cannot report anything.

**Split aggregates by source.** Any metric that sums across independent inputs gets one series per input. The aggregate is for a dashboard; the alert is per source.

And a fourth that is more of a discipline than a rule: **every check gets a case that must pass and a case that must fail.** If both come back with the same answer, the check is broken and you know it in seconds instead of weeks.

That last one has caught more of my own mistakes than the other three combined, because it catches the category where the monitor is fine and my test of the monitor is not.

## The uncomfortable part

I did not learn any of this from a book. All five of these are systems I built, in a framework whose entire premise is that you should be able to prove what happened.

The gap was never in the design. It was in the substitution I made without noticing: measuring what was easy to measure and treating it as if it were what I meant.

That substitution is invisible while everything works. It is only visible during the failure, which is the one moment you were counting on the monitor.

> **Lees ook:** [July build log: merged is not deployed, and green is not verified](/blog/july-build-log)

Previous entries in the build log series: [July](/blog/july-build-log), [June](/blog/june-build-log). The orchestration framework is open source: [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration).

Building checks that watch the thing rather than its proxy is part of what I do as [AI-architect](/ai-consultancy).

your monitoring watches the wrong thing cover

large_your-monitoring-watches-the-wrong-thing_cover

medium_your-monitoring-watches-the-wrong-thing_cover

small_your-monitoring-watches-the-wrong-thing_cover

thumbnail_your-monitoring-watches-the-wrong-thing_cover

your-monitoring-watches-the-wrong-thing_cover

Your monitoring watches the wrong thing

I once told an agent, in plain language, that it had my full authorization to run a production mutation. It refused. I said it again, more explicitly. It refused again.

That was correct behavior, and it took me a minute to appreciate it. The safety layer sits in the harness, not in the conversation. A chat message does not change a permission decision, because if it could, the boundary would be worth exactly as much as the model's willingness to be talked out of it.

This is the useful mental model for Claude Code security: **the things you say are input, the things you configure are enforcement.** Everything below follows from that split.

## Three layers, and what each one actually stops

There are three places a boundary can live, and they fail differently.

| Layer | Where it lives | What it stops | Failure mode |
|---|---|---|---|
| Permission rules | `settings.json` | Tool calls matching a pattern | Too broad, quietly approves neighbors |
| Hooks | `settings.json`, external command | Anything you can express in code | Wrong event; observes instead of blocks |
| Sandbox | OS / container boundary | Everything outside the box | Expensive to set up, easy to leave open |

Most teams have the first one, half-configured. Almost nobody has the third.

The reason to care about the ordering is that each layer catches what the one above it missed, and the cost goes up as you descend. Permission rules are free and coarse. Hooks are cheap and precise. A real sandbox is the only one that holds when the other two are wrong, and it is the one that takes an afternoon.

The three enforcement layers around an agent: permission rules, hooks, and the sandbox boundary

## Standing mandate versus per-action consent

The single most useful distinction I have found, and the one that took me longest to name.

A **standing mandate** is "go ahead, work through the backlog." It covers the ordinary shape of the work: read the repo, write code, run tests, open a branch.

**Per-action consent** is required for a specific class of actions that a standing mandate cannot reasonably cover, no matter how enthusiastically it was given.

In one session my orchestrator was blocked three times in an afternoon. Sending keystrokes into another agent session's trust dialog. Instructing a worker to change its own settings. Merging after a review that had not come back clean.

All three blocks were correct. All three were resolved by asking me, per action, and all three together cost under ten minutes.

The pattern in those three is worth stating explicitly, because it generalizes past this tool:

- **Self-modification.** An agent changing the rules it runs under is not covered by permission to do the work. If it were, every boundary would be one instruction away from removal.
- **Answering someone else's security prompt.** A trust dialog exists to ask a human. An agent that can click through it has defeated the dialog for every future case, not just this one.
- **Bypassing a gate that did not pass.** A merge after a failed review is not a faster version of the same action. It is a different action.

If you configure nothing else, configure these three as things that always come back to a person. They occur rarely enough that the friction is negligible and they are exactly the ones where an automatic yes is unrecoverable.

## What to put in the permission rules

Permission patterns are the cheapest layer and the one people over-invest in. Two rules keep it sane.

**Deny by exception, not by enumeration.** You cannot list every dangerous command. Anyone who has tried has a rule set with forty entries that still misses `python -c`. Deny the small set of genuinely irreversible things (force push, destructive database commands, anything that writes outside the project root) and let the rest go through the normal flow.

**Allow narrowly and specifically.** The point of an allow rule is to remove a prompt you are tired of, not to open a category. `Bash(npm test)` removes one prompt. `Bash(npm *)` removes that prompt and quietly hands over `npm publish`.

That second failure is silent, which is what makes it worth stating. Nothing breaks when an allow rule is too broad. It works better than before. You find out at the moment the broad half gets used.

## Where hooks belong in the security picture

Permission rules match on tool names and command patterns. Hooks run code, so they can decide on things a pattern cannot express.

For access control the shape is a `PreToolUse` hook, because that is the event that fires before the tool executes and is allowed to block it. `PostToolUse` fires after success and cannot block; it is a record, not a gate.

The one detail worth carrying over from the file-access case: a hook on `Read` and `Grep` does nothing about `cat .env`, because that is a single `Bash` call. If you write a hook that guards file reads, write the second one that inspects Bash command strings, or you have locked the door and left the window.

I go through the full configuration in [Claude Code hooks: stop the agent from reading your .env](/blog/claude-code-hooks-stop-the-agent-reading-env).

## The failure mode nobody plans for: the guard that breaks quietly

In June I added a restriction that was entirely correct: a read-only review process should not be able to commit or push. Good change. It shipped.

It also broke the review process for three weeks, and nobody noticed.

The guard set a deny flag on the lane and did not turn on the corresponding scope flag that the caller needed. The lane refused fail-closed, which is the right default. But the error output was being discarded, so the failure surfaced as **silence**: the review simply produced no verdict from that seat, round after round.

The panel still returned results, because it had a quorum rule that tolerates a single missing participant. The fail-safe worked so well that it hid the outage.

Two lessons, and I would trade a lot of clever configuration for either one.

**The dangerous regression is not bad code. It is correct code that a caller was not migrated onto.** New guard added, existing consumer left behind. Every security tightening should be followed by the question: who called this before, and do they still work?

**A discarded error message turns a failure into a mystery.** I spent a day on the wrong diagnosis because the real reason was being swallowed. Surfacing that output was a smaller fix than the one I actually needed, and it would have made the real fix obvious three weeks earlier.

If your security layer can fail closed, make sure it fails **loudly**. Robustness and observability pull against each other here, and robustness usually wins by default because it is the one you designed for.

## The sandbox question

Everything above is enforcement inside the tool. A sandbox is enforcement outside it, and it is the only layer whose correctness does not depend on your configuration being right.

The honest position: for a project on a machine that also holds your credentials, your SSH keys and your other clients' code, the permission layer is doing more work than it should. It is not designed to be the boundary between an agent and your entire filesystem. It is designed to be the boundary between an agent and mistakes within a project.

The practical middle ground I use:

- Agents that touch client code run in a dedicated worktree, not the main checkout
- Secrets are not on disk in any path reachable from the project root
- Anything that writes outside the project goes through one place I can point at

That last one matters more than the first two. The value of a single exit point is not that it is safer in itself, it is that "what did this system send outside" becomes a question with an answer instead of a search.

## What I would do this week

If you run agents on real code and have configured nothing, four steps, roughly two hours total.

1. **Deny the irreversible.** Force push, destructive database commands, writes outside the project root. Five entries, not forty.
2. **Audit your allow rules for wildcards.** Every `*` is a category you opened. Narrow the ones you cannot justify out loud.
3. **Add a PreToolUse hook on credential paths**, plus the Bash companion. Thirty minutes.
4. **Decide your three per-action items.** Self-modification, third-party security prompts, and bypassing a failed gate are a good default set.

Then the harder one, which is not configuration: **check that your guards still fire.** Pick one and try to trip it deliberately. A guard nobody has tested since the day it was written is a guard you believe in rather than one you have.

Frequently asked questions

## The broader point

Every one of these controls exists because agreeing to something is not the same as enforcing it. That is not a Claude Code observation. It is the same thing I write about when a client asks why their AI review process approves everything, and the answer is that a check which never blocks anything is not a check.

> **Lees ook:** [Claude Code hooks: stop the agent from reading your .env](/blog/claude-code-hooks-stop-the-agent-reading-env)

The orchestration layer I run this on is open source: [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration). The permission and hook configuration is in the repository if you would rather read it than take my word for it.

Designing that boundary for a system that is going into production is what I do as [AI-architect](/ai-consultancy).

claude code security what the agent may touch cover

large_claude-code-security-what-the-agent-may-touch_cover

medium_claude-code-security-what-the-agent-may-touch_cover

small_claude-code-security-what-the-agent-may-touch_cover

thumbnail_claude-code-security-what-the-agent-may-touch_cover

claude-code-security-what-the-agent-may-touch_cover

Claude Code security: what the agent may touch

This build log covers July. I log every failure worth remembering. Not the typos, the ones where I believed something that was not true. July produced 678 entries across five projects: 265 in the orchestration framework, 174 in the sales copilot, 163 in the business layer, 42 in the cockpit, 27 in the SEO tool. 255 of them are marked as fully shareable.

Reading them back, four categories dominate. Locking and concurrency, roughly 101 hits. Migrations and schema drift, 86. Audits and audit trails, 73. Deploy versus merge, 67.

They look unrelated. They are the same failure. In every case I was watching a signal that stood in for the thing I actually cared about, and the signal was healthy while the thing was broken.

That is the whole month in one sentence. Here are the four, with the specific way each one lies.

## 1. Concurrency: shared state you did not know was shared

The pattern that cost the most calendar time, and the one I was most confident about beforehand.

I run background agents in parallel, each in its own git worktree. Isolated, obviously. That is what a worktree is for.

Then my own commit landed on an agent's branch, and the branch I intended stayed empty.

Worktree-isolated agents share the object store and the refs with the main checkout. A concurrent branch creation inside an agent raced with my checkout-and-commit in the main directory. The isolation is real for files and not for refs, and I had generalized from the first to the second.

The fix was surgical (move the branch pointer, fast-forward push, no force needed). The lesson was broader: **isolate your own work too, not just the agents'.** Either serialize, or work in a worktree yourself. Do not be the one unisolated writer in a system built for parallelism.

The same shape showed up elsewhere in a much larger form. An activation script silently doubled an entire timer fleet: 54 timers firing twice, so duplicate scrapes, duplicate syncs, duplicate scoring. No monitor fired. The script checked for the canonical name, saw the already-running legacy-named timers as MISSING, and installed duplicates alongside them.

The monitoring checked whether a job was running and whether the data was fresh. It did not check whether the job was running **once**. Presence, not count.

Deduplication brought 147 timers back to 92.

Four recurring failure patterns from July, each showing a proxy signal that stayed healthy while the real thing broke

## 2. Schema drift: two ledgers that disagree

Activating a merged deploy batch, I found that the database's own migration history and the repository's numbered migrations both had entries 186 and 187, with entirely different contents. One said anon-revoke, the other said taste weights.

Two independent tracking systems for the same thing, drifted apart. On top of that, handover notes claiming "youtube not deployed, finance waiting" while most of it was already in production.

There is no clean way to reconcile two ledgers that disagree, because the disagreement is the evidence that neither is authoritative. So I stopped trying.

**When two migration ledgers disagree, the schema itself is the only ground truth.** Not the log, not the handover note. Does the table exist, does the column exist, what type is it. Introspect the objects, derive the actual applied state from that, and apply only the genuine delta.

What felt like a 35-migration tangle turned out to be a small delta. The panic came entirely from trusting logs that had no reason to be right.

## 3. The audit that degrades to green

This is the one I keep writing about, and July gave me the cleanest example I have had.

I built a new fabric audit: checks for split-brain state, per-project ledgers, hash-chain integrity. It went through three rounds of an automated review gate. Rounds two and three caught the same thing, which is why the gate exists.

The audit reported clean while it could not read its own input. A malformed project registry silently became an empty project list, so the per-project check iterated over nothing, found no problems, and returned green.

Graceful degradation in exactly the wrong place. An audit tool that degrades to green when its input is broken is worse than no audit at all, because it manufactures confidence. There was also an `except: pass` on a marker read, which is the same mistake in miniature.

The fix was to fail loud everywhere: an unreadable-but-present registry returns RED instead of quietly empty, a failed chain-verification import returns WARN instead of a silent skip, and project identifiers get validated as safe path components before use. PR #1045, 17 tests.

**The expensive bug in a control system is not a false RED. It is a false GREEN.**

A related discovery, less dramatic and more structural. I wanted to pull fleet-wide tool failures out of the governed audit trail. They are not in there. Of 15.407 receipts at the time, zero had a tool-call field.

The receipts are a provenance ledger: which dispatch produced which commit, which pull request, which gates passed. That is a complete record of what happened and a blind spot for how the system struggled getting there. **Provenance is not observability**, and I had been treating one as if it covered the other for months.

The third variant caught me from the other direction. I was testing whether a cost calculation recognized the model names that actually get stored. Everything came back empty, including the names I was certain were in there. I nearly concluded the whole calculation was broken.

It was my test. I passed an object where a string was expected, so the lookup failed on everything, including my own control case.

That the control case failed too was the signal that the fault was mine. **Always include a case in your measurement that you know must pass.** If it fails alongside the rest, you are measuring yourself.

Once corrected, there was a real finding underneath: the lookup knows the names it expects and none of the names actually stored. Different spelling, different prefixes. 1.377 records with no cost attribution, silently.

## 4. Merged is not deployed

The last category is the most embarrassing, because it is the one I would have said I already knew.

After a merge I considered a bug fixed. The consumer that reported it kept hitting it. The fix was on main; the consumer was running a channel.

The fleet runs a centrally installed, versioned copy behind a symlink channel. Merging to main does not touch that channel. The channel has to be updated explicitly: pull main, flip the symlink, migrate the stores. Until that happens, "merged" and "deployed" are two different states and only one of them matters to the user.

Done meant done only after running the channel update, verifying the symlink, and testing end-to-end from the consumer's own directory with no workaround flags.

A cousin of this one, from the same week. After five squash merges I went to clean up orphaned worktree branches. Git reported them as NOT merged, for branches whose contents I had merged minutes earlier.

Squash merge creates a new commit with a different hash. The original branch commits are not ancestors of main even though their content is. The ancestry check measures lineage, not content.

**The tooling meant to keep you safe raises a false alarm at precisely the safest action.** Know your merge strategy before you believe a data-loss warning, or you will either panic or, worse, learn to ignore the warning.

## What this build log changed

Three things came out of this month that are not just lessons.

**Enforcement moved to the door.** The audit trail showed 13 merged tracks that had been built and merged without their planning gate ever passing. The gate existed. It guarded the bookkeeping at close time and nothing consulted it at dispatch or merge, so enforcement lived only in soft instruction text and any autonomous run walked straight past it.

A governance rule you only observe is a suggestion. It is now enforced at the dispatch door, the one place everything passes through, advisory-first with a signed and audited operator override.

**Fail-loud became the default in every control path.** Not just the audit. Anywhere a check could return "nothing to see" because it failed to look, it now returns an error instead.

**I pointed my own review panel at my own framework.** The question was whether the codebase is more complex than what it does. The answer was uncomfortable and useful: a solid orchestration core, plus a governance layer running ahead of its own demand, with half of it in an undeclared state. Built and tested, neither deliberately on nor deliberately parked.

That last category is the one worth naming, because I suspect it is common. Not dead code and not live code. Code in a state nobody decided on. Every subsystem now has a decision attached: keep, prune, deliberately park, or activate and measure.

## The one-line version

Every failure above has the same shape. Process alive is not work finished. Merged is not deployed. Green is not verified. The ledger is not the schema. Running is not running once.

In each case I picked the signal that was easiest to measure and treated it as the signal that mattered. That substitution is invisible while everything works, which is exactly why it survives long enough to cost you something.

The question I now ask before trusting any check: **what would this report if the thing it measures were completely broken?** If the answer is "the same as now", it is not a check.

> **Lees ook:** [June build log: the month my systems lied to me](/blog/june-build-log)

Previous entries: [June](/blog/june-build-log), [May](/blog/2026-05-31-may-build-log-metrics-launch), [April](/blog/april-build-log-metrics-failures-learnings).

If you are putting agents into production and want this kind of record without building it yourself, that is the work I do as [AI-architect](/ai-consultancy).

The orchestration framework is open source: [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration).

large_july-build-log_cover

medium_july-build-log_cover

small_july-build-log_cover

thumbnail_july-build-log_cover

july-build-log_cover

July Build Log: Merged Is Not Deployed, and Green Is Not Verified

Claude Code hooks are the layer where you decide what an agent may touch. Get the event wrong and you get a smoke detector installed after the house burns down: a PostToolUse hook that logs "the agent just read `.env`". The secret is already in the context window. It is already on its way to a model endpoint. Logging it does not unread it.

That is the entire argument for hooks, and it is the one thing the documentation states clearly and everyone still gets wrong: **only some hooks can block, and the blocking ones fire before the thing you are worried about.**

I run agents against my own repositories every working day, and the hook layer is what makes that acceptable rather than reckless. Here is how the events actually behave, which one you want for secrets, and what breaks when you pick the wrong one.

## Which Claude Code hooks can actually block

Claude Code fires hooks at a lot of points in a session. For file access, three matter.

| Event | When it fires | Can it block? |
|---|---|---|
| `PermissionRequest` | When a tool call needs a permission decision | Yes |
| `PreToolUse` | Before a tool call executes | Yes |
| `PostToolUse` | After a tool call succeeds | No |

`PostToolUse` cannot block. That is not a limitation you can work around with a clever exit code. The tool has already run, the result is already in the transcript, and the hook is an observer.

So for secrets, the answer is `PreToolUse`. But the event is only half the configuration. The other half is the matcher, and that is where most setups leak.

Where PreToolUse and PostToolUse fire relative to the tool call, and which one can block execution

## The matcher people get wrong

The instinct is to guard `Write` and `Edit`. That protects the file from being changed.

That is not the threat. Nobody is worried that the agent will rewrite `.env`. The worry is that it will **read** it, put the contents into the context window, and ship them to an inference endpoint that is not yours.

So the tools to match are the ones that get content out of a file:

- `Read`, the direct path
- `Grep`, which returns matching lines. For a `.env` file that is the entire value you were protecting

`Grep` is the one that gets forgotten. A hook that only matches `Read` looks correct in review and leaks on the first `grep -r "API_KEY"`. The agent was not being sneaky; it was doing exactly what you asked, through a different door.

Matcher syntax is worth knowing precisely here, because it changes evaluation:

- If the string contains only `[a-zA-Z0-9_-, |]`, it is treated as an exact match or a pipe-separated list. `Read|Grep` is a list.
- If it contains anything else, it is evaluated as an **unanchored JavaScript regex**. `^Notebook` is a regex. So is `mcp__memory__.*`.

That distinction bites when someone writes `Read|Grep|Bash(cat *)` and expects the list semantics to keep holding. It will not; the parentheses push it into regex mode and the pattern no longer means what it reads like.

## Bash is the hole in every file-access hook

Here is the part that gets left out of most write-ups, and it is the reason my own configuration is longer than three lines.

Blocking `Read` and `Grep` stops the two tools whose job is reading files. It does nothing about:

All of that is one tool call: `Bash`. If your hook does not inspect Bash commands, you have guarded the front door and left the side gate open.

The practical shape of a working configuration is two entries against the same event:

1. `PreToolUse` matching `Read|Grep`, checking the file path against a deny list.
2. `PreToolUse` matching `Bash`, checking the command string for the same paths.

The second one is imperfect, because a shell command can be obfuscated in more ways than you can pattern-match. I accept that. The goal is not to defeat an adversary; the agent is not adversarial. The goal is to stop the ordinary case where a model reaches for the fastest route to an answer and the fastest route happens to run through your credentials.

If you want something stronger than pattern-matching, the answer is not a better regex. It is to not have the secret on disk in a path the agent can reach at all.

## Exit codes and the JSON contract

Two ways to return a decision, and they do not compose.

**Exit codes.** Exit `0` means success and your stdout JSON gets processed for decisions. Exit `2` is a blocking error: stderr is fed back to Claude, and **stdout is ignored entirely**. Any other exit code is non-blocking; the action proceeds and stderr shows up in the transcript.

That middle rule is the one that costs people an afternoon. If you emit a carefully constructed JSON decision object and then exit `2` for good measure, your JSON is discarded. The block still happens, but your reason message comes from stderr instead, and any structure you built is gone.

**JSON output.** Exit `0` and write to stdout:

`permissionDecision` takes four values: `deny`, `allow`, `ask`, and `defer`. That last one matters more than it looks. `defer` means "I have no opinion, run the normal permission flow." `allow` means "skip the permission prompt entirely."

Using `allow` where you meant `defer` is how a hook written to block one thing quietly becomes an auto-approver for everything else it touches. I have done this. The hook worked, the tests passed, and it had silently removed a permission prompt I still wanted.

Rule of thumb: return `deny` for the cases you are guarding, and `defer` for everything else. Only return `allow` when you have genuinely decided that this specific call needs no human in the loop.

## Prevent, then observe

`PostToolUse` is not useless. It is just not a control.

I use it for the thing it is actually good at: writing a record. Every tool call that completes appends a line to a ledger with what ran, what it touched, and what came back. That ledger sits at 23.292 entries and it is the reason I can answer a question about something that happened three weeks ago without guessing.

The split is worth stating plainly, because it is the same split that shows up in every governance conversation I have with clients who have never heard of a hook:

- **Prevention** is a gate. It runs before, and it can say no.
- **Observation** is a record. It runs after, and it can only tell you what happened.

You need both. You cannot substitute one for the other. A team that has only observation finds out about problems accurately and late. A team that has only prevention blocks the known cases and learns nothing about the unknown ones.

## What I would set up first

If you have no hooks at all, three, in this order.

**One: `PreToolUse` on `Read|Grep`,** denying paths that match `.env`, `.env.*`, `*.pem`, `credentials*`, and whatever your stack calls its secret files. Fifteen minutes.

**Two: `PreToolUse` on `Bash`,** denying commands whose text contains those same paths. Another fifteen minutes, and it catches the majority of the side-gate cases.

**Three: `PostToolUse` on everything,** appending a line to a file. Not for control. So that when something surprising happens, the answer is a lookup instead of a reconstruction.

Everything after that is refinement. `SessionStart` to inject project context, `PreCompact` to checkpoint state before the window is squeezed, `SubagentStop` to catch work that finished outside the main loop. Useful, none of it urgent.

## Where this goes next

Hooks are the smallest useful piece of agent governance: a rule that runs whether or not anyone remembers it. The follow-up question is broader, and it is the one that comes up in every conversation once the .env case is handled: what should the agent be allowed to touch at all, and how do you enforce that rather than agree to it.

I write about that in [MCP vs CLI: when to replace a server with a command](/blog/mcp-vs-cli-when-to-replace), and the record-keeping side sits in [the audit trail I run for AI agents](/blog/audit-trail-ai-agents-ndjson).

If you want the same boundary logic applied to a system you are putting into production rather than a repo you are experimenting in, that is what I do as [AI-architect](/ai-consultancy).

The orchestration layer I use is open source. [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration) if you want to read the hook configuration rather than take my word for it.

claude code hooks stop the agent reading env cover

large_claude-code-hooks-stop-the-agent-reading-env_cover

medium_claude-code-hooks-stop-the-agent-reading-env_cover

small_claude-code-hooks-stop-the-agent-reading-env_cover

thumbnail_claude-code-hooks-stop-the-agent-reading-env_cover

claude-code-hooks-stop-the-agent-reading-env_cover

Claude Code hooks: stop the agent from reading your .env

Over the last few months I have quietly removed more MCP servers than I have added. Most of the work they did moved to a plain CLI call, and the agents got faster, cheaper, and easier to audit.

This is not a "MCP is dead" post. The Model Context Protocol solves real problems, and for some of them nothing else comes close. But the MCP vs CLI question deserves a sharper answer than "add every server you can find". The honest framing is a decision rule: which MCP servers earn their place, and which you should replace with a scriptable CLI call. After running both in production, here is where I draw the line.

## The hidden cost of MCP: the context tax

The problem with MCP is not the protocol. It is what it does to your context window.

Every time an MCP server is connected, its full tool definitions get injected into the model's context on every single turn. Tool names, descriptions, complete parameter schemas. You pay for that on every message, whether or not you ever call the tool.

The numbers are blunt. A single MCP server can add roughly 770 tokens of standing schema per turn. That sounds small until you scale it: a 93-tool GitHub MCP server consumes around 55,000 tokens at initialization, Jira adds another 17,000, and a typical enterprise stack of five to ten servers burns 100,000 to 200,000 tokens before the user types a single character. In practice MCP can eat 40 to 50 percent of the available context window before the agent does any real work.

A CLI binary costs zero standing tokens. It sits on disk and consumes nothing until you call it. When you do, you pay for one command, not for a permanent schema dump.

This is why I am not alone in this shift. In early 2026 Perplexity's CTO Denis Yarats said publicly that they were moving away from MCP toward plain APIs and CLI tools, citing exactly these two issues: context consumption and clunky auth flows.

## Where the CLI wins

For most of what an agent does, a CLI call is simply the better tool.

- **Token efficiency.** A CLI command averages a couple of hundred tokens. The equivalent MCP operation, with its schema overhead, runs from thousands of tokens per call up to tens of thousands of standing schema across a full server set. One measured agent-search comparison found the MCP path used about 17 times more tokens per call. Spend those tokens on the actual problem instead.
- **Determinism.** A CLI call is the same every time. Same input, same flags, same output. You can pin it, version it, and diff it. That is the foundation of a system you can trust.
- **Auditability.** A command is a line you can log. I can show you exactly what ran, with which arguments, and when. For governance that is gold.
- **A smaller attack surface.** Fewer always-on connections, fewer tool definitions sitting in context for a prompt injection to hijack.

This is the core of how I run my own orchestration. It is CLI-driven by design, open source on GitHub, and it never depended on OAuth. That choice is also why it survived the OAuth ban that broke other tools. I wrote that story up in [the best OpenClaw alternative after the OAuth ban](/blog/best-openclaw-alternative-cli-subprocess-oauth-ban) and in [how to orchestrate Claude Code without OAuth](/blog/orchestrate-claude-code-without-oauth-cli-patterns).

## Where MCP still earns its place

Now the fair half. There are jobs where a CLI call is the wrong answer, and MCP is exactly right.

- **The service has no CLI.** If a vendor only ships an API and you do not want to hand-roll a client, an MCP server is the clean wrapper.
- **OAuth and dynamic, multi-user auth.** If you need per-user tokens, refresh flows, and scopes across many users, MCP gives you that without building it yourself.
- **Stateful, multi-step sessions.** When a workflow holds a connection open and carries state across steps, the protocol earns its overhead.
- **Built-in schemas, RBAC, and audit logs.** For a multi-tenant enterprise tool, the standardization MCP provides is worth the tokens.

The pattern is clear. MCP wins when you need auth you do not want to build, state you cannot easily script, or an integration that has no command line of its own.

## The decision table

Strip away the debate and it comes down to matching the task to the tool. This is the table I actually use.

Decision table: keep MCP or replace with a CLI call, per task type

| Task type | Keep MCP | Replace with CLI |
|---|---|---|
| Cloud ops (AWS, GCP, GitHub) | | CLI: mature vendor tools exist |
| Local scripts and file work | | CLI: deterministic, zero schema cost |
| Database queries | | CLI: scriptable, auditable |
| OAuth across many users | MCP: auth you don't want to build | |
| Stateful multi-step session | MCP: holds state across steps | |
| Service with no CLI | MCP: clean API wrapper | |
| Multi-tenant with RBAC + audit | MCP: standardized schemas | |

The rough split I land on in practice: CLI for the roughly 80 percent of tasks built on tools the model already knows, and MCP for the 20 percent where you genuinely need auth, state, or a missing command line.

## The token gap closes if you build the server right

One honest caveat. The token tax is not a law of nature, it is a default. A well-built MCP server that uses code execution and on-demand tool discovery, instead of dumping every schema upfront, can match a CLI on token efficiency while keeping the auth, the audit trail, and the multi-tenant support a raw CLI cannot give you.

So the rule is not "always CLI". It is "do not pay the context tax for nothing". If a server gives you auth and state you would otherwise have to build, it earns its tokens. If it is wrapping a tool that already has a perfectly good command line, it does not.

## The security angle you should not skip

There is a reason I weigh the attack surface, not just the token count. MCP's always-on tool definitions are a target. Security researchers have documented prompt injection through tool descriptions, lookalike tools that silently replace trusted ones, and tool combinations that can exfiltrate files. Adding more tokens to context does not just cost money, it dilutes your instructions and widens the surface an attacker can manipulate.

A CLI call does not remove that risk, but it shrinks it. Fewer standing definitions, deterministic commands, and a log of exactly what ran. I went deeper on this in [why I don't install most MCP servers](/blog/mcp-servers-niet-installeren-security).

## What I would do

Audit what you have connected right now. For each MCP server, ask one question: does this give me auth or state I cannot easily script? If yes, keep it. If it is wrapping a tool with a real CLI, replace it and reclaim the context.

Then measure. Look at how many tokens are spent before the agent does anything useful. That number is your context tax, and most teams are paying far more of it than they realize.

The architecture beats the protocol, the same lesson I drew in [architecture beats models](/blog/architecture-beats-models-ai-agent-dispatches). Match the task to the tool, keep the line auditable, and stop loading schemas you never call. This is [how I design AI systems for production](/ai-consultancy), and the orchestration behind it is open source if you want to see how: [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration).

Frequently Asked Questions

large_mcp-vs-cli-when-to-replace_cover

medium_mcp-vs-cli-when-to-replace_cover

small_mcp-vs-cli-when-to-replace_cover

thumbnail_mcp-vs-cli-when-to-replace_cover

mcp-vs-cli-when-to-replace_cover

MCP or CLI? Which servers to keep, and which to replace with a CLI call

In June, four of my systems told me everything was fine. All four were lying.

A scan reported "completed" with zero pages behind it. A quality gate passed a test suite it should have blocked. A finished feature shipped green and then never ran once. A default engine that every config file pointed at had never actually been built. Four separate products, four green lights, one lesson underneath all of them: green does not mean working.

This is the honest June 2026 AI build log. Not the "what shipped" highlight reel. The part where the status said done, I believed it, and then I found out. If you followed the **[May build log](/blog/2026-05-31-may-build-log-metrics-launch)**, this is the same solo build with a harder edge, because June was the month I stopped trusting my own dashboards and started building the checks that catch them lying.

June 2026 build log key numbers: four products, 346 VNX commits, version 1.0.0, 34 parallel workers in one overnight run, 14,000+ receipts, and six documented cases of a green status that was lying

## The numbers

Counted from the receipt ledger and `git log` over the month. As of 30 June 2026.

| Metric | June 2026 |
|--------|-----------|
| Products I actively built | 4 |
| Commits in the VNX repo (June only) | 346 |
| VNX version reached | 1.0.0 (public tag held back on purpose) |
| Parallel AI workers in one overnight run | 34 |
| Receipts in the ledger (lifetime) | 14,000+ |
| Sales-copilot test suite | grew to ~816 green |
| Governance pieces published | 7 |
| "Green" statuses that were lying (documented) | 6 |

That last row is the whole story. Six times this month, a system reported success while it had failed or done nothing. None of them crashed. That is exactly why they were dangerous.

## Why "green" is the most expensive word in software

Every engineer reading this already knows silent failures exist. That is not the interesting part. What is rare is admitting your own systems did it to you this month, on record, and what is new is how fast the rate is climbing now that AI writes more of the code. This is not a lesson about a thing you did not know. It is a receipt for a thing everyone knows and almost nobody publishes.

A crash is honest. It pages you, it leaves a stack trace, it demands attention. The failure that costs you is the quiet one: an operation that returns a success code while producing the wrong result, or no result at all.

I am not alone in hitting this. The 2025 Stack Overflow Developer Survey found that 66% of developers say AI answers are "almost right but not quite," and 45% lose significant time debugging AI-generated code, while more of them now distrust its accuracy (46%) than trust it (33%). Adoption is up, trust is down. The gap between the two is filled with code that looks finished.

The felt sense lies too. A [randomised study from METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) put 16 experienced open-source developers on 246 tasks in codebases they knew well. With AI tools they were 19% slower, while they believed they were 20% faster. If your own perception of "faster" is off by that much, your perception of "done" is not safe either.

And the code itself drifts. GitClear's 2025 analysis of 211 million changed lines found copy-pasted code rising from 8.3% to 12.3% between 2020 and 2024, refactored lines dropping by more than half, and more code reworked within two weeks of being committed. More volume, less structure, more churn. Speed without verification is not productivity. It is debt you have not been billed for yet.

So June's real work was not shipping features. It was building the instruments that tell me when a green light is lying. Here is where each product lied, and what I built to catch it.

## VNX Orchestration: the gate that passed itself

The spine of June was collapsing a dozen ad-hoc ways of dispatching AI work into one governed entry point. One door, every task through it, every task logged. To trust a door you have to prove it cannot be walked around.

So I did not just write tests for it. I pointed an independent reviewer model at it, prompted only to break it, not to approve it. It came back red. Over two rounds it found six real ways to slip past the gate that my own tests, and two other review models, had all passed clean. Whitespace tricks, a symlink staging path, a check-then-act timing hole. Things that were green everywhere else.

I escalated the keystone change to a stronger model for a third pass, added edge-probe tests for every bypass it found, and only merged once the same adversary that had broken it twice went green on its own attacks. That is the rule I took out of June: a **[deterministic quality gate](/blog/deterministic-quality-gates)** is only real if something that did not write the code tries to break it. A test you wrote to pass is a description of what you expected, not a defence against what you missed.

The sharper moment came at the very end of the month. I caught my own orchestrator routing some of its builds around the governed lane, which meant those builds produced no **[append-only audit receipt](/blog/audit-trail-ai-agents-ndjson)**. The governance runtime was quietly skipping its own governance. I made it a hard rule: builds go through the governed path, with a receipt, or they do not happen. The audit trail has to apply to the thing that owns the audit trail, or it is theatre. That is the argument behind [prompt engineering is dead, governance engineering is next](/blog/governance-engineering-next), proven on myself.

The upside of all that hardening: VNX reached 1.0.0 in the repo this month, 346 commits deep. I am holding the public release tag back deliberately. That last step stays human.

## My SEO tool: "scan completed", zero pages

This one I found the way you should find these things. By using the product like a customer instead of reading its status field.

I logged in, ran a scan, and the dashboard said "completed." There was no data. Zero pages scanned. I went into the logs and the database and found the ugly truth: the record was marked complete while the pipeline itself had returned not-complete, and nothing had been saved. A routing bug had quietly sent the logged-in scan down a shallower path than the one it was supposed to run. The status field was reporting the intention, not the outcome.

Two things made this a perfect example of the June theme. First, every automated test was green, because the tests checked that a scan finishes, not that a scan produces the pages it claims. Second, it only surfaced because I clicked through it live. If I had trusted the "completed" badge, I would have shipped a product that confidently tells paying users it scanned their site while saving nothing.

The fix was straightforward once I could see it: make the status reflect the artifact, not the attempt. A scan is complete when the pages exist and are stored, not when the function returns. The deeper fix was a decision it forced. I used the live-testing month to move the tool away from selling one-off reports and toward recurring value, where "it actually works every time" is the entire product, not a nice-to-have.

## Sales copilot: the default engine nobody built

I am prototyping a tool that listens during a sales call and flags the questions you missed, live. In June I took it from demo to something closer to sellable, which meant letting several models tear the codebase apart and then grinding through the backlog they produced. Security, consent, retention, tamper-evident logs. My finance-industry reflexes applied to my own product.

The best lie was almost funny. The transcription engine that every config, every preset, and every doc pointed at as the default had never been compiled. Nothing in the install process built its binary. So the moment you pressed "Start Call," it reached for an engine that did not exist and crashed. The default path, the one a first-time user hits, was the one path guaranteed to fail. Green in every setting file, dead on first contact.

The fix was to build the binary as part of install, idempotently, with a graceful fallback and an actionable warning instead of a bare crash. Then I benchmarked engines properly on real call recordings and cut live transcription latency from around six seconds to roughly two.

The harder lesson was not technical. Building the paid tier, I finally admitted something I had been avoiding: you cannot enforce a paid feature in client-side, open-source code. Anyone can read it, patch it, or fork it. Pretending otherwise is its own silent failure, a moat that reports as present while being trivially absent. So I stopped pretending the code is the moat and moved the value to the things a fork genuinely cannot copy. Honest architecture beats a lock that only looks locked.

## Mission Control: the feature that shipped and never ran

Mission Control is my own operations cockpit, the layer I run the rest of the business through. In June it went from personal tool toward something a buyer could lift and run. That meant a lot of parallel AI workers building at once, up to 34 in a single overnight run, every plan and every merge passing a model review gate and then me.

Two lies stood out. The first: I built a batch of new scoring jobs. They existed, their tests passed, the code was correct. And they were never wired into the scheduler. A finished, tested, "done" feature that would have sat there silently doing nothing forever, because nothing ever called it. Correct and useless are not opposites. A dedicated cleanup pass caught the orphans and registered them.

The second was worse, and a review panel caught it, not me. A budget check that was supposed to stop runaway work failed open. If the service it queried had an outage, the check quietly returned "you are fine, proceed," authorising unlimited work during exactly the moment you least want that. A safety limit that disables itself when its dependency hiccups is not a limit. Same panel flagged a timing-unsafe secret comparison and a handful of writes that could fail silently and still report success. All of it green, all of it wrong, all of it fixed to fail closed instead.

That is the pattern across all four products. The failures that hurt do not announce themselves. You have to go looking, and the most reliable way to look is to have something adversarial look for you.

## What I learned

None of these are new as principles. What was new this month was paying for them four times in four products, and building the checks that make the principle operational instead of a poster on the wall.

**Green is a claim, not a fact.** Every "completed," every 200, every passing suite is a hypothesis about the world. Treat it as one. Check the artifact, not the status field. The scan is done when the pages exist, not when the function returns.

**A gate is only real if an adversary tries to break it.** A single reviewer, human or model, tests what they thought of. Point something that did not write the code at it, prompted to break it, and it finds the six bypasses everyone else waved through. This is the same reason my audience keeps posting about not trusting evals nobody stress-tests.

**The dangerous failures are silent.** A crash pages you. A false success ships. Spend your instrumentation budget on detecting silence: the job that never ran, the limit that disabled itself, the success that saved nothing.

**Speed without verification is debt.** The METR result is the whole industry in one number. You feel faster and you are slower, because the time moved downstream into debugging what looked done. The gate is not what slows you down. The gate is what makes the speed real.

And the meta-lesson, from catching my own orchestrator skipping its own audit trail: the governance has to apply to the thing that owns the governance, or it is decoration. That is the whole argument for treating governance as an engineering discipline, and this month I got to prove it against myself.

## The public side of June

Underneath the four bugs, I published the through-line all month: one argument in seven pieces. Do not depend on one model, which is what **[the Fable 5 ban taught me about vendor-independence](/blog/fable-5-ban-vendor-onafhankelijkheid)**. Own a floor no order can switch off, which is why I **[benchmarked eight models on my own hardware](/blog/lokale-llm-benchmark-mkb)**. Keep the records, because the **[EU AI Act will soon expect them](/blog/ai-act-augustus-2026-mkb-checklist)**. And measure quality as a number instead of a feeling, with **[first-pass yield](/blog/first-pass-yield-ai-kwaliteit)**. Same thesis as this log, seen from the other side.

> **Read also:** [July build log: merged is not deployed, and green is not verified](/blog/july-build-log)

## What is next

Three things for July, in order:

- Keep the human merge gate exactly where it is. Everything above was caught by adversarial checks plus a human at the end. I am not automating that last step away.
- Ship VNX 1.0 publicly. The code is frozen at 1.0.0. The public tag is the one gate left, and it stays mine to pull.
- Turn the June governance work into a concrete offer for organisations facing the same problem: not more AI, but proof that the AI did what it said. That is the **[AI architecture](/ai-consultancy)** I build.

Four products, one lesson. Solo build, public feedback loop. See you in the July build log.

## Sources

- METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," 10 July 2025: [metr.org](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) (arXiv 2507.09089)
- Stack Overflow, 2025 Developer Survey, AI section: [survey.stackoverflow.co/2025](https://survey.stackoverflow.co/2025)
- GitClear, "AI Copilot Code Quality 2025" (211M lines analysed): [gitclear.com](https://www.gitclear.com/ai_assistant_code_quality_2025_research)
- VNX Orchestration, open source: [github.com/Vinix24/vnx-orchestration](https://github.com/Vinix24/vnx-orchestration). Commit counts and receipts are reproducible from the repo and the ledger.

large_june-build-log_cover

medium_june-build-log_cover

small_june-build-log_cover

thumbnail_june-build-log_cover

june-build-log_cover

June Build Log: The Month My Systems Lied to Me

If you benchmark frontier models with a single chat completion, you are measuring the wrong thing.

A one-shot call gives the model no tools. It cannot read a file, run a test, or fix its own mistake. So the score you get back is not "how good is this model at coding," it is "how good is this model at writing code blind, in one pass." For agentic coding work that gap is the whole story.

I wanted a fair comparison across Claude (Opus, Sonnet), OpenAI Codex (gpt-5.5), Z.AI GLM-5.x, Moonshot Kimi K2.7 and DeepSeek V4. Same tasks, same scoring, same agentic loop. The hard part was never the scoring. It was getting five different model families to run inside one harness without each one failing for a different infrastructure reason.

This is a peer write-up. If you are building something similar through Claude, the proxy pattern, the hardening notes and the numbers below will save you a few nights.

## The fairness problem, stated plainly

Claude and DeepSeek were easy: both speak the Anthropic Messages protocol, so both drive the `claude` CLI directly. The CLI gives them the full agentic harness for free. Tools, file I/O, sub-agents, iteration.

The other three were the problem.

GLM and Kimi went through a simple chat runner at first. No tools. Their correctness scored zero across every cell, not because the models are weak, but because the harness could not let them produce a deliverable. The harness was the bottleneck, not the model. That is a measurement bug, and it is the kind of bug that quietly poisons a benchmark if you do not catch it.

So I had two options. Build a second agentic harness for the OpenAI protocol, or bridge the OpenAI-shaped models into the Claude harness I already trusted. I did both, and the second one is the interesting one.

## The proxy trick: drive any model through the Claude CLI

The `claude` CLI talks to an Anthropic `/v1/messages` endpoint. It reads `ANTHROPIC_BASE_URL` to decide where. That is the whole hook.

Put a local [litellm](https://github.com/BerriAI/litellm) proxy on `localhost:4141` that exposes an Anthropic-compatible `/v1/messages`, backed by GLM on OpenRouter:

The CLI believes it is talking to Anthropic. Inference flows to GLM-5.2 through OpenRouter. GLM now gets the exact same harness as Claude, the same tools, the same iteration budget. "Same model, full harness versus flat tool-call" becomes a real, measurable comparison instead of a guess.

Two constraints this respects, and they matter if you run a governed setup:

- No Anthropic SDK anywhere. It is the CLI, a subprocess, not `import anthropic`. That keeps the account-safety posture intact.
- No direct vendor account for GLM. Inference still flows through OpenRouter, so the cost trail and the provider policy stay consistent.

Codex is the same idea with a different transport. `codex exec --json` is its own agentic loop, so I drive it directly and parse its JSON event stream rather than proxying it. The principle holds: give every model a real agentic loop, then score what it builds, not what it describes.

If you want only the GLM-5.2 recipe and the cheapest route into a harness, I split that into a focused how-to: [GLM-5.2 through the Claude harness via OpenRouter](/blog/glm-5-2-via-claude-harness-openrouter).

## Then the lanes started failing, one at a time

A benchmark is a brutal stress test of the layer underneath it. Not the models. The execution layer: process spawning, isolation, streams, worktrees, credits. Running five model families on dozens of tasks under concurrency surfaced bugs that never show up in a single happy-path dispatch. Here are the ones worth your time.

### The Codex stderr pipe deadlock

This one was a real production bug, not a test artifact.

`codex exec --json` was spawned with `stderr=PIPE`, but the stream drainer only ever read stdout. Codex runs at high reasoning effort, which emits a lot of stderr. The OS pipe buffer fills at around 64 KB, the write blocks, and Codex deadlocks. The harness saw `rc=1` at a few seconds with no deliverable and recorded a failure. It looked like launch flakiness. It was a classic undrained-pipe deadlock.

The fix is one daemon thread that drains stderr to a log:

Any Codex worker that ever emits more than 64 KB of stderr would hit this. The benchmark just made it happen on every heavy cell, so it was impossible to ignore.

### Isolation leaks into the committed seed

Unsandboxed CLI workers run with the repository as their working directory. On from-scratch tasks they would navigate up and write into the main checkout's committed task seed. One worker's output contaminated the next cell's input.

The fix was to move every worker's git worktree outside the repository root, so repo-relative navigation cannot reach the main checkout. Plus a fail-loud guard that refuses to run a cell in the shared checkout at all, and a file lock to serialize worktree add and remove so parallel git operations stop tripping over each other.

Lesson: "isolated" is not a property you assume, it is one you fail loudly on when it is violated.

### Scoring a deliverable when the process exits non-zero

The GLM-through-harness lane produced correct work but the CLI exited `rc=1`. The receipt validator did not recognize the new provider, so the governance step raised, and the whole dispatch crashed with a non-zero code before a receipt was ever written. For weeks I read that as "the model does not close the loop cleanly." It was a one-line allow-list that did not list the lane.

That fix unlocked a subtler one. Once you score deliverables on a non-zero exit, an immediate-exit cell that produced nothing would get scored against the untouched seed and earn a bogus baseline. So scoring-on-failure has to be gated on real wall-clock: if the cell ran for two seconds and made no model call, it is a DNF, not a near-miss. Three small fixes, one clean scoreboard.

### Quota is the silent killer

Every provider has a different wall, and each one fails in a way that mimics a code bug:

- Codex yields a few cells per window, then immediate-exits at 5 to 16 seconds. Account rate limit, not a crash.
- Kimi gives roughly nine cells, then hard-stops with a 429.
- GLM on OpenRouter returns a 402 the moment the request reserves more `max_tokens` than the key's remaining daily limit can afford. The intermittency is not random: the daily budget drains through the day, so cells that pass at 09:00 fail at 13:00.

The harness has to tell these apart from genuine incapacity. A model that fails for quota gets re-run in a fresh window. A model that runs the full task and still produces nothing is real signal you keep. Conflate the two and your benchmark lies in both directions.

The best debugging moment of the whole effort came from this. I dispatched GLM, Kimi and DeepSeek as agents to inspect the failing lanes and propose fixes. The GLM agent could not finish, and its own failure report contained the exact 402 with the message "you requested 32000 tokens, can only afford 4026, adjust the key's daily limit." The model debugged its own lane by failing in a legible way.

## What the numbers say

The plumbing is the point of this post, but the numbers are why the plumbing matters. The composite below is 0 to 5, weighted: 40% correctness and 20% completeness from a deterministic `verify.py` (pytest, SQL constraints, adversarial matrices), 15% cost and 15% speed, and 10% code-quality from a cross-provider judge panel. 442 canonical cells, 14 lanes, 15 tasks across six tiers. Every cell is the median composite of its replications; `N` is the cell count, and a low `N` means read the direction, not the decimal.

#### The quality matrix (median composite per lane × tier)

| Lane | T1 trivial | T2 medium | T3 complex | T4 frontier | T5 review/design | T6 real review |
|---|---:|---:|---:|---:|---:|---:|
| Claude Opus 4.6 | 4.88 (N=9) | 4.99 (N=9) | 4.47 (N=6) | 4.19 (N=4) | 4.22 (N=4) | 4.38 (N=4) |
| Claude Opus 4.7 | 4.98 (N=9) | 4.97 (N=9) | 4.47 (N=6) | 4.16 (N=4) | 4.39 (N=4) | 4.38 (N=4) |
| Claude Opus 4.8 | 4.97 (N=9) | 4.96 (N=9) | 4.70 (N=6) | 4.12 (N=4) | 4.34 (N=4) | 4.37 (N=4) |
| Claude Sonnet 4.6 | 4.97 (N=9) | 4.97 (N=9) | 4.72 (N=6) | 4.17 (N=4) | 4.21 (N=4) | 4.39 (N=4) |
| Codex GPT-5.4 | 4.45 (N=9) | 0.00 (N=9) | 0.00 (N=6) | 0.00 (N=4) | 0.00 (N=4) | — |
| Codex GPT-5.5 | 4.45 (N=9) | 4.43 (N=9) | 4.45 (N=6) | 4.12 (N=4) | 0.00 (N=4) | 4.40 (N=4) |
| DeepSeek V4 Flash (harness) | 4.71 (N=9) | 4.96 (N=9) | 4.36 (N=6) | 4.60 (N=4) | 4.25 (N=4) | 4.45 (N=4) |
| DeepSeek V4 Pro (harness) | 4.98 (N=9) | 4.97 (N=9) | 4.89 (N=6) | 4.69 (N=4) | 4.29 (N=4) | 4.44 (N=4) |
| GLM-5 | 4.88 (N=9) | 4.86 (N=9) | 4.87 (N=6) | 4.36 (N=4) | 4.17 (N=4) | 4.34 (N=4) |
| GLM-5.1 | 4.86 (N=9) | 4.93 (N=9) | 4.72 (N=6) | 0.72 (N=4) | 4.07 (N=4) | 4.43 (N=4) |
| GLM-5.2 (flat) | 4.94 (N=9) | 4.93 (N=9) | 4.25 (N=6) | 0.69 (N=4) | 4.09 (N=4) | 1.77 (N=4) |
| GLM-5.2 (harness) | 4.49 (N=1) | — | 4.46 (N=4) | 4.15 (N=2) | 4.38 (N=2) | 4.37 (N=4) |
| Kimi K2.6 | 4.75 (N=1) | — | — | — | — | — |
| Kimi K2.7 | 4.98 (N=9) | 4.96 (N=9) | 4.46 (N=6) | 4.03 (N=4) | 4.36 (N=4) | 4.42 (N=4) |

The trivial and medium tiers compress: almost everything that runs lands above 4.8, so they do not discriminate. The work happens from T3 onward, and on T4-T6 the harnessed open lanes (DeepSeek Pro 4.69/4.29/4.44, DeepSeek Flash 4.60/4.25/4.45) sit shoulder to shoulder with the Claude lanes. The two `0.69`/`0.72` cells in the GLM flat and 5.1 columns on T4 are the flat-runner collapse that finding 1 is about, and the same lanes recover to 4.15+ once harnessed.

#### Where models actually separate (lane × discriminating task)

| Lane | State-machine SSE | Path-sandbox (sec) | Mock-introspection | Agent-engine design | Real-review A | Real-review B |
|---|---:|---:|---:|---:|---:|---:|
| Claude Opus 4.6 | 4.89 | 3.90 | 4.47 | 4.42 | 4.38 | 4.38 |
| Claude Opus 4.7 | 4.99 | 3.88 | 4.47 | 4.39 | 4.37 | 4.38 |
| Claude Opus 4.8 | 4.98 | 3.80 | 4.70 | 4.34 | 4.38 | 4.37 |
| Claude Sonnet 4.6 | 5.00 | 3.92 | 4.47 | 4.40 | 4.40 | 4.38 |
| Codex GPT-5.4 | 0.00 | 0.00 | 0.00 | 0.00 | — | — |
| Codex GPT-5.5 | 4.49 | 3.86 | 4.42 | 0.00 | 4.42 | 2.19 |
| DeepSeek V4 Flash (harness) | 4.50 | 4.21 | 2.49 | 4.45 | 4.45 | 4.45 |
| DeepSeek V4 Pro (harness) | 4.95 | 4.43 | 4.71 | 4.42 | 4.44 | 4.44 |
| GLM-5 | 4.89 | 3.97 | 4.95 | 4.29 | 4.34 | 3.18 |
| GLM-5.1 | 4.74 | 1.44 | 4.96 | 2.02 | 4.43 | 3.15 |
| GLM-5.2 (flat) | 2.50 | 2.59 | 4.14 | 2.94 | 3.02 | 1.74 |
| GLM-5.2 (harness) | 4.16 | 3.92 | 4.46 | 4.45 | 4.36 | 4.37 |
| Kimi K2.7 | 4.49 | 3.55 | 4.46 | 4.45 | 4.41 | 4.43 |

Four findings stand out.

### 1. The harness reveals capability the flat runner hides

This is the cleanest result in the whole matrix. The same model, `z-ai/glm-5.2` via OpenRouter, scored two ways: once through a thin agentic runner, once through the full `claude` CLI harness via the proxy above. Each cell is the median composite over its replications.

| Task (discriminating) | GLM-5.2 flat | N | GLM-5.2 harness | N | Δ |
|---|---:|---:|---:|---:|---:|
| State-machine + SSE | 2.50 | 2 | 4.16 | 2 | +1.66 |
| Path-sandbox (security) | 2.59 | 2 | 3.92 | 1 | +1.33 |
| Mock-introspection trap | 4.14 | 2 | 4.46 | 2 | +0.32 |
| Agent-engine system design | 2.94 | 2 | 4.45 | 1 | +1.52 |
| Real codebase review A | 3.02 | 2 | 4.36 | 2 | +1.33 |
| Real codebase review B | 1.74 | 2 | 4.37 | 2 | +2.62 |

Same weights, same endpoint, same tasks. The only thing that changed is the harness, and a mid-pack runner climbed into the top cluster on every discriminating task. The lift is largest exactly where the work is hardest: +2.62 on a real codebase review, +1.66 on a state-machine build. The flat runner's weakness is also a variance story. On real-review A it was bimodal across reps (one run 4.29, one 1.76, median 3.02) while the harness held steady at 4.36. A benchmark that only ran the flat call would have filed GLM-5.2 under "weak at coding" and been confidently wrong. The harness N is thin (one to two reps per cell), so read these as a direction, not a decimal.

### 2. Harnessed open models reach Claude-tier on the hard tasks

The discriminating tasks are where models separate. On agent-engine system design, the top cluster is DeepSeek-flash-harness, GLM-5.2-harness and Kimi K2.7 at 4.45, DeepSeek-pro-harness and Claude Opus 4.6 at 4.42, Claude Sonnet 4.6 at 4.40, and the other Claude Opus lanes between 4.34 and 4.39. On the two real-codebase reviews the same harnessed open lanes plus Kimi K2.7 cluster between 4.34 and 4.45, shoulder to shoulder with Claude.

The frontier gap on review and design work is far smaller than a one-shot leaderboard suggests, as long as the harness is equal. On the trivial tier the picture is different: there the Claude and Kimi lanes lead at 4.97-4.98 while a harnessed open lane like DeepSeek-flash sits lower at 4.71. The lift is concentrated exactly where it should be, on the work that needs read, test and fix, not on a one-file mechanical edit. One honest dent in the open-lane story: DeepSeek-flash cratered to 2.49 on the mock-introspection trap (N=2), the one place a Claude or DeepSeek-pro lane clearly held and it did not.

### 3. Lane reliability is not the same as model capability

The Codex columns carry `0.00` cells, and they are the most misread numbers in the matrix. The split below, counted over every run not just the canonical ones, is the only honest way to read them.

| Lane | Attempts | Completed | Launch-fail | Capability-fail | Launch-rate |
|---|---:|---:|---:|---:|---:|
| Claude Opus 4.6 | 63 | 58 | 4 | 1 | 0.94 |
| Claude Opus 4.7 | 63 | 58 | 4 | 1 | 0.94 |
| Claude Opus 4.8 | 69 | 63 | 4 | 2 | 0.94 |
| Claude Sonnet 4.6 | 66 | 58 | 8 | 0 | 0.88 |
| Codex GPT-5.4 | 54 | 19 | 8 | 27 | 0.85 |
| Codex GPT-5.5 | 114 | 41 | 31 | 42 | 0.73 |
| DeepSeek V4 Flash (harness) | 116 | 55 | 61 | 0 | 0.47 |
| DeepSeek V4 Pro (harness) | 109 | 65 | 44 | 0 | 0.60 |
| GLM-5 | 86 | 64 | 22 | 0 | 0.74 |
| GLM-5.1 | 146 | 95 | 47 | 4 | 0.68 |
| GLM-5.2 (flat) | 96 | 63 | 31 | 2 | 0.68 |
| GLM-5.2 (harness) | 17 | 14 | 2 | 1 | 0.88 |
| Kimi K2.6 | 2 | 1 | 1 | 0 | 0.50 |
| Kimi K2.7 | 129 | 79 | 32 | 18 | 0.75 |

`launch-rate` is the fraction that started cleanly; completion is lower again because some runs launched and then failed. The two Codex lanes split apart here. GPT-5.5 launched 73% of the time and, when it landed, scored with the leaders: 4.42 on real-review A, 4.45 on state-machine. But it completed only 41 of 114 attempts, so a thin matrix easily shows a zero where there simply was no finished run. GPT-5.4 is a harder story: 27 capability-fails and zeros across T2 through T5 mean it genuinely did not complete the harder tiers, not just that the launcher choked. So the Codex zeros are a blend, mostly completion-reliability for 5.5 and real non-completion for 5.4, and neither should be read as "Codex is bad at review" when the cells it did finish landed around 4.4. \[^codex]

The same standard runs through the whole benchmark: a cell is a real score only when a genuine run finished, and the launch table is published next to the quality matrix so a zero can never quietly masquerade as a verdict. DeepSeek-flash makes the point from the other side: a 0.47 launch-rate that still produced top-cluster scores on the cells that finished.

### 4. Cost and speed are a separate axis from quality

Quality clustering and cost clustering are not the same chart.

| Lane | Total $ | Median $/cell (metered) | $/composite-point | Note |
|---|---:|---:|---:|---|
| Claude Opus 4.6 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |
| Claude Opus 4.7 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |
| Claude Opus 4.8 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |
| Claude Sonnet 4.6 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |
| Codex GPT-5.4 | 1.4002 | 1.4002 | 0.02977 | metered (Codex CLI) |
| Codex GPT-5.5 | 1.0529 | 1.0529 | 0.00800 | metered (Codex CLI) |
| DeepSeek V4 Flash (harness) | 0.0162 | 0.0081 | 0.00010 | metered (DeepSeek API) |
| DeepSeek V4 Pro (harness) | 0.0379 | 0.0189 | 0.00023 | metered (DeepSeek API) |
| GLM-5 | 2.4305 | 0.0072 | 0.01529 | metered (OpenRouter) |
| GLM-5.1 | 0.9956 | 0.0038 | 0.00714 | metered (OpenRouter) |
| GLM-5.2 (flat) | 3.3895 | 0.0027 | 0.02482 | metered (OpenRouter) |
| GLM-5.2 (harness) | 0.0000 | — | 0.00000 | capture-gap: metered, not logged |
| Kimi K2.6 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |
| Kimi K2.7 | 0.0000 | — | 0.00000 | subscription: $0 = not metered |

The subscription lanes (Claude, Kimi-via-CLI) report `$0`, which means "not metered," not "free." \[^cost] GLM-5.2-harness is a genuine capture-gap: metered via OpenRouter, but the harness lane did not log it. Among the lanes you can actually see a bill for, the standout is not a frontier name. DeepSeek-flash earns a composite point for `$0.00010` and DeepSeek-pro for `$0.00023`, two orders of magnitude under the Codex lanes at `$0.008` to `$0.030` per point, while landing 4.44-4.45 on the real reviews. The per-cell metered cost runs from fractions of a cent on the DeepSeek and GLM lanes to around a dollar on Codex.

Speed is the thinnest axis, because token-rate capture was intermittent on the subscription and harness lanes.

| Lane | Median tokens/s (N measured) | Median wallclock (s) |
|---|---:|---:|
| Claude Opus 4.8 | 18.6 (N=2) | 269 |
| DeepSeek V4 Pro (harness) | 56.5 (N=1) | 116 |
| GLM-5 | 36.2 (N=26) | 124 |
| GLM-5.1 | 29.9 (N=24) | 193 |
| GLM-5.2 (flat) | 14.5 (N=25) | 251 |
| Claude Opus 4.6 | — (0 measured) | 106 |
| Claude Sonnet 4.6 | — (0 measured) | 137 |
| Codex GPT-5.4 | — (0 measured) | 13 |
| Kimi K2.7 | — (0 measured) | 146 |

Read this one with the launch table open. The only robust tokens/s numbers are the GLM lanes (N=24-26): GLM-5 at 36 tok/s, the flat GLM-5.2 runner a sluggish 14.5. Wallclock is a trap on the unreliable lanes: Codex GPT-5.4's 13-second median is not speed, it is failures exiting early, since most of its cells never finished. The honest takeaway on this axis is narrow: where it was measured, the harnessed open lanes are not slower than the frontier, and the flat GLM runner is both the weakest and the slowest.

The point for anyone choosing a model: the cheapest lane that reaches ~4.4 on real code review is not a Claude lane. If your workload is review and design rather than the hardest agentic builds, a harnessed open model can land in the same quality cluster at a metered cost you can actually see and control.

## The caveats

The caveats are not a disclaimer, they are the credibility. A benchmark that hides its failure modes is worse than no benchmark.

- **Small n.** Tiers run two to three replications. This is a high-signal, low-volume design built to discriminate, not to produce confidence intervals. Read the per-rep spread before turning a cell into a ranking. The GLM-5.2 runner bimodality is a good reminder that variance is itself a finding.
- **The judge is an LLM, for 10% of the score.** Only `code_quality` is model-judged, by an Opus-plus-Kimi panel that averages the two and flags disagreement above 1.5. The other 90% of the composite is deterministic `verify.py`.
- **Codex zeros are non-completions, not a review verdict.** A blend of launch failures and, for GPT-5.4, a genuine capability wall on the harder tiers. The launch table is published so the split is visible. Repeated here because it is the easiest number to misquote.
- **Cost zeros are unmeasured, not free.** Subscription lanes (Claude, Kimi) and the GLM-5.2-harness capture-gap all read `$0` for different reasons. Do not compare cost across metered and unmetered lanes as like-for-like.
- **One task carries a known artifact.** The SSRF async-fetch task has a cross-lane filename-convergence effect, so I treat that single column as lower-confidence wherever it appears. \[^ssrf]
- **t6 is two private codebases, about 300K+ lines of code combined.** Only aggregate scores are published, no code and no project names, labelled real-review A and B. \[^t6]

\[^codex]: Codex `0.00` cells are non-completions. GPT-5.5 launched ~73% of attempts but completed ~36% (41 of 114); GPT-5.4 also hit a capability wall on T2-T5 (27 capability-fails). When a run finished it scored ~4.4 on review. See the launch table for the full split.
\[^cost]: Subscription lanes (Claude, Kimi-via-CLI) report no per-call usage, so cost reads `0`. That is "not measured," never "free."
\[^ssrf]: The `08_ssrf` task has a known cross-lane filename-convergence effect and is treated as lower-confidence.
\[^t6]: t6 task definitions live in a gitignored overlay and are never published. Only anonymized aggregate scores are shared.

## What I would tell a peer building this

A few things generalize beyond my setup.

**Bridge, do not rebuild.** If you already trust one agentic harness, put a local protocol-translating proxy in front of your other models instead of maintaining a second harness. One code path to harden is worth a lot.

**Treat the execution layer as the system under test.** The models are fine. The spawns, the streams, the worktrees and the credit checks are where the bugs live. Drain every pipe. Isolate outside the repo. Fail loud on a broken invariant.

**Separate infrastructure failure from incapacity, in code.** Wall-clock, exit code and error class are enough to bucket "re-run this" versus "record this as a real zero." Without that split, quota noise and model weakness look identical.

**Keep a glass-box trail.** Every dispatch writes a structured report and a receipt with the git ref it ran against. When a number looks wrong at 02:00, the audit trail is the difference between a five-minute check and a re-run of the whole matrix.

## Why a fair benchmark costs a week

This benchmark took almost a week, and that is the point, not an apology. The scoring was never the hard part. Getting five model families to run inside one harness fairly, so a low score means the model could not do the work and never that the plumbing failed, is what consumed the days. A weekend benchmark measures whichever model your plumbing happened to favor. A fair one costs a week, and the week is most of the value.

My proof for that claim is not a screenshot. It is 2.800+ receipts across 1.500+ analyzed sessions in a governance-first runtime, every dispatch traceable to the git ref it ran against.

The harness, the lanes and the proxy bridge are open source. The full methodology, with every fairness mechanism cited down to the `file:line`, is in [`scripts/benchmark/field-tests/METHODOLOGY.md`](https://github.com/Vinix24/vnx-orchestration) in `Vinix24/vnx-orchestration`. Every published number traces from `raw.csv` through `scorer.py` to a task's `verify.py`. If a figure here does not reproduce from those, treat it as wrong.

One follow-up from this work: the [GLM-5.2 OpenRouter recipe](/blog/glm-5-2-via-claude-harness-openrouter) for the cheapest route into a harness.

frontier models one agentic harness cover

large_frontier-models-one-agentic-harness_cover

medium_frontier-models-one-agentic-harness_cover

small_frontier-models-one-agentic-harness_cover

thumbnail_frontier-models-one-agentic-harness_cover

frontier-models-one-agentic-harness_cover

One Harness, Five Model Families: Running Codex, GLM, Kimi and DeepSeek Through Claude's Agentic Loop

GLM-5.2 looks mediocre at coding, if you call it the way most benchmarks do.

Run the exact same model through a real agentic tool-harness instead of a flat call and it jumps to frontier-Claude tier. Same weights, same OpenRouter endpoint, same tasks. The only thing that changed is the harness. If you are evaluating GLM-5.2, or any OpenRouter model, for coding, the harness is the variable you are probably getting wrong.

This is the focused recipe. For the full 14-lane benchmark across five model families and the lane-hardening story behind it, see [one harness, five model families](/blog/frontier-models-one-agentic-harness).

## The one comparison this whole post is about

`z-ai/glm-5.2` via OpenRouter, two ways. The flat runner is a thin read/write/run loop. The harness is the full `claude` CLI, pointed at GLM-5.2 through a local litellm proxy. Composite 0 to 5, median of the replications (`N` is one to two per cell, so read the direction, not the decimal).

| Task | GLM-5.2 flat runner | GLM-5.2 via Claude harness | Δ |
|---|---:|---:|---:|
| State-machine + SSE (complex) | 2.50 | 4.16 | +1.66 |
| Path-sandbox / SSRF-style security | 2.59 | 3.92 | +1.33 |
| Mock-introspection trap (debug) | 4.14 | 4.46 | +0.32 |
| Planted-defect code review | 4.09 | 4.31 | +0.22 |
| Agent-engine system design | 2.94 | 4.45 | +1.52 |
| Real codebase review A (private, 300K+ LOC) | 3.02 | 4.36 | +1.33 |
| Real codebase review B (private) | 1.74 | 4.37 | +2.62 |

Two stories in one table. The harness raises the score on the hard tasks, and it removes the variance. The flat runner's real-review-A result swung between 4.29 and 1.76 across reps (median 3.02); the harness held flat at 4.36. The biggest single lift is +2.62 on a real codebase review, exactly the read-run-fix work a flat one-shot call cannot do. A model you would have filed under "not good enough for real work" lands in the top cluster once it can read, run and fix.

## The how-to: GLM-5.2 in the Claude Code CLI

Three steps. The whole trick is that the `claude` CLI talks to an Anthropic `/v1/messages` endpoint and lets you choose where, so you put a translating proxy in between.

**1. Run a local litellm proxy** that exposes an Anthropic-compatible `/v1/messages` endpoint, backed by GLM-5.2 on OpenRouter.

**2. Point the `claude` CLI at the proxy.** Use a local bearer token, not an OAuth subscription token.

**3. Drive your normal agentic loop.** Inference flows OpenRouter to GLM-5.2; the harness is byte-identical to what Claude itself runs. Tools, file I/O, iteration, sub-agents, all of it.

The same recipe generalizes to any OpenRouter-served model. Swap the `model_name` entry and you have Qwen, Llama or anything else OpenRouter serves running inside the same loop. (In my own benchmark DeepSeek ran on its own key rather than OpenRouter, but the proxy pattern is identical.)

## Why this is safe, and why the routing stays consistent

Two things make this more than a hack.

The routing stays consistent. GLM still goes through OpenRouter, with no direct z.ai or Zhipu account, so cost tracking and the model registry stay aligned with the rest of your stack.

The account stays safe. It is the CLI driving a subprocess, not the Anthropic SDK, and the proxy authenticates with a local key, not an OAuth subscription token. Anthropic has cracked down on subscription OAuth tokens used in third-party tools, including the Agent SDK, as a terms-of-service violation. A CLI-driven, local-key setup avoids that entire class of risk.

## The counter-data

The harness is not free magic, and pretending otherwise would make this post hype instead of useful.

On a trivial yaml refactor the harness actually scored lower than the flat runner, 1.74 against 2.99. On one subtle-bugfix task the harness only reached 2.19. The lift is concentrated on tasks that need a read, test and fix loop. On a one-file mechanical edit there is nothing for the loop to add, and the extra machinery can even get in the way. If your workload is bulk mechanical edits, the harness is not your win. If it is real implementation, security boundaries, design and review, it is a large one.

## Caveats

- Small n. Two replications on the hard tasks. The variance is itself a finding, so read the spread before treating a cell as a ranking.
- One review task runs against private production codebases. Only aggregate scores are published, no code and no names.
- The proxy adds latency and a translation layer between the Anthropic tool format and OpenRouter. A model that handles tool-calling poorly will not benefit from the harness no matter what you put in front of it.
- `drop_params: true` is doing real work. It tolerates Anthropic parameters OpenRouter does not map one to one. Without it, some calls fail at the proxy.

The proxy config, the runners and the scoring are open source in `Vinix24/vnx-orchestration`. The source-referenced methodology, with every fairness mechanism cited down to the `file:line`, lives in [`scripts/benchmark/field-tests/METHODOLOGY.md`](https://github.com/Vinix24/vnx-orchestration). Every number in the table above traces from `raw.csv` through `scorer.py` to a task's `verify.py`.

The takeaway is small and load-bearing. Before you conclude a model is weak at coding, check whether you measured the model or your harness. With GLM-5.2 the difference between those two is the difference between bottom tier and frontier tier.

glm 5 2 via claude harness openrouter cover

large_glm-5-2-via-claude-harness-openrouter_cover

medium_glm-5-2-via-claude-harness-openrouter_cover

small_glm-5-2-via-claude-harness-openrouter_cover

thumbnail_glm-5-2-via-claude-harness-openrouter_cover

glm-5-2-via-claude-harness-openrouter_cover

GLM-5.2 Through the Claude Harness: The OpenRouter Route

In de afgelopen weken schreef ik over [CLAUDE.md](/blog/2026-05-04-claude-md-uitleg-beginners), [rules.md](/blog/2026-05-08-rules-md-claude-code-codeerstandaarden), [hooks](/blog/2026-05-13-claude-code-hooks-automatische-acties), [context engineering](/blog/prompt-engineering-is-dood-context-engineering), en [self-learning loops via intelligence pipelines](/blog/vnx-intelligence-system-autonomous-monitoring). Telkens met voorbeelden uit mijn eigen setup.

Vandaag de samenhang: hoe past dit alles samen? Wat staat waar? Wat doet wat? Voor AI peers en hobbyisten die voorbij de eerste experimenten zijn, een complete walkthrough van mijn werkende productie-setup.

Geen abstracties. Wel: file-paths, command-mappingen, patronen die ik 8 maanden in productie heb getest.

## De getallen

- **38 skills**, herbruikbare instructie-sets, gedeeltelijk in `.claude/skills/`, gedeeltelijk in `~/.claude/skills/`
- **11 sub-agents**, gespecialiseerde agents voor research, code review, architecture, etc.
- **6 hooks**, SessionStart, PreToolUse, PostToolUse, Stop
- **4 memory directories**, always, on-demand, per-task, archive
- **3 rules-bestanden**, tone-of-voice, testing, security
- **1 CLAUDE.md per project**, context per project
- **0 vector databases**, alles markdown + NDJSON

Schaal: middelgroot. Niet enterprise. Wel veel voor een solopreneur.

## De directory-structuur

Eerst een vogelvlucht. Hoe is alles georganiseerd?

### Globaal, `~/.claude/`

Belangrijk: globale skills/memory zijn voor mijn algemene werk. Project-skills/memory zijn voor specifieke projecten. Bij conflict wint project.

> 📖 **Lees ook:** **[Hooks in Claude Code: geautomatiseerde acties na elke prompt](/blog/2026-05-13-claude-code-hooks-automatische-acties)**: hoe SessionStart, PostToolUse en Stop je workflow automatiseren

## De gelaagde architectuur: identiteit, voice, en project-context apart

De directory-structuur hierboven is georganiseerd langs één principe: **wat is identiteit en wat is project-context?** Identiteit verandert zelden, project-context verandert wekelijks. Door ze fysiek te scheiden voorkom ik dat een blog-sessie iets weet over een klant-deadline, en dat een coding-sessie zich druk maakt om marketing-buzzwords.

Twee niveaus dus, scherp gescheiden.

Gelaagde Claude Code architectuur: globaal en project apart

### Niveau 1, globaal (`~/.claude/`)

`profile.md` wordt automatisch geladen via de `@~/.claude/profile.md` regel in CLAUDE.md. Voordeel: één bron voor mijn identiteit, geen herhaling per project.

Wat in `profile.md` staat:

- Wie ik ben (Head of AI Solutions SalesMinds + VNX Digital + SEOcrawler)
- Communicatie-voorkeuren (Nederlands, 1e persoon enkelvoud, geen em-dashes)
- Werk-voorkeuren (architect-mindset, governance-stokpaardje, human-in-the-loop)
- Portfolio-overview (één paragraaf per domein)

Het is bewust statisch. Deadlines en project-status horen niet hier, die wonen in project-memory.

### Niveau 2, per project (`<project>/.claude/`)

Drie subtiele dingen die ik hier doe:

**Path-scoping** via een paar instellingsregels bovenaan een rules-bestand (YAML frontmatter voor wie die term kent). Mijn `tone-of-voice.md` heeft `paths: ["Content/**", "project/vnx_digital/**"]`, wat betekent: alleen actief als Claude in die mappen werkt. In een coding-sessie in `Development/` laadt deze rule niet. Officieel ondersteund door Claude Code, weinig gebruikt.

**Project-memory** in `.claude/memory/` met een `MEMORY.md` index. Hier komen observaties die voor één project gelden, bijvoorbeeld aliassen tussen klantnamen, of specifieke pricing-beslissingen.

**Anti-marketing-buzzwords** als aparte rule. Termen als "revolutionair" of "game-changer" blokkeer ik in content-folders, niet in code. Globale en project-rules vullen elkaar aan zonder elkaar te storen.

### Wat dit oplevert in de praktijk

In een coding-sessie in `~/Development/SEOcrawler_v2/`:

- Globale identiteit + voice + anti-AI-tics laadt (~120 regels)
- Project CLAUDE.md laadt (~80 regels)
- Marketing-rules laden NIET (path-scope past niet)
- Project-memory laadt (~50 regels)

In een blog-sessie in `~/Desktop/BUSINESS/Content/blog/`:

- Alle bovenstaande PLUS marketing-rules PLUS blog-rules
- Tone-of-voice.md activeert via path-scope
- Anti-marketing-buzzwords laden in voor content-werk

Totaal: ~300 regels gerichte context per sessie. Geen ruis. Geen ontbrekende info. Claude weet altijd wie ik ben, en in een blog-folder weet hij óók hoe ik klink.

Dit patroon, gelaagde scheiding tussen identiteit en context, is wat het verschil maakt tussen 38 skills die wel werken en 38 skills die elkaar in de weg zitten.

## De 38 skills (overzicht)

Niet alle 38 individueel, wel de categorieën die je kunt zien:

### Content & Marketing (13 skills)

`blog-orchestrator`, `blog-writer`, `blog-images`, `blog-publisher`, `blog-editor`, `seo-interlinker`, `seo-agent`, `linkedin-writer`, `linkedin-strategist`, `linkedin-image-creator`, `social-content`, `copywriting`, `copy-editing`

Deze coördineren mijn content-pipeline. `blog-orchestrator` is de top, hij roept andere skills aan in volgorde.

### Audience & Lead Intelligence (5 skills)

`audience-intelligence`, `lead-scorer`, `discussion-actor`, `cold-email`, `email-sequence`

Voor mijn LinkedIn intelligence engine en lead-werk.

### Product Marketing & Strategy (5 skills)

`product-marketing-context`, `marketing-psychology`, `marketing-ideas`, `pricing-strategy`, `launch-strategy`

Strategische skills die ik wekelijks aanroep voor branding-werk.

### Development Operations (8 skills)

`research-agent`, `vnx-project-setup`, `excalidraw-diagrammer`, `phantombuster`, `claude-api`, plus de SuperClaude `sc:*` skills (`sc:document`, `sc:implement`, etc.)

Voor technisch werk en architectuur-werk.

### Visual & Diagram (3 skills)

`linkedin-carousel-builder`, `excalidraw-diagrammer`, `linkedin-image-creator`

### Reflection & Process (4 skills)

`competitor-alternatives`, `funnel-orchestrator`, `content-strategy`, `reddit-marketing`

Per maand: ik gebruik typisch 5-8 skills per dag, niet alle 38. Het zit allemaal beschikbaar zodat *als* ik het nodig heb, het er is.

## De 11 sub-agents

Sub-agents zijn aparte. Een skill is een instructie-set. Een sub-agent is een aparte sessie met eigen context. Mijn 11:

| Agent | Wat het doet | Wanneer ik hem aanroep |
|-------|--------------|------------------------|
| `general-purpose` | Open onderzoeksvragen | Onderzoek dat 3+ queries kost |
| `Explore` | Code zoeken in repo | "Waar staat X?" |
| `Plan` | Implementatieplan maken | Voor non-trivial features |
| `backend-architect` | Backend ontwerp | Database design, API design |
| `frontend-architect` | Frontend ontwerp | UI patterns, accessibility |
| `system-architect` | Systeem-niveau | Architectuur-keuzes met long-term impact |
| `security-engineer` | Security review | Voor elke significante PR |
| `quality-engineer` | Test strategy | Edge cases bepalen |
| `performance-engineer` | Performance optimization | Wanneer benchmarks nodig zijn |
| `python-expert` | Python best practices | VNX-werk vooral |
| `claude-code-guide` | Claude Code expertise | Vragen over CC zelf |

Niet alle 11 dagelijks. Wel ze allemaal binnen handbereik via `Agent(subagent_type=...)`.

De `context_monitor.sh` is degene die op #1 in Google staat, auto-rotation bij 65% gevuld geheugen.

## Hoe het samenwerkt, een concrete sessie

Hoe ziet een typische sessie eruit?

**Stap 1, Sessie start.** Ik open Claude Code in mijn blog-project. SessionStart-hook draait `load_memory.sh`. Dit injecteert:

- `~/.claude/CLAUDE.md` (persoonlijke werkstijl, ~40 regels)
- `[project]/CLAUDE.md` (blog-project context, ~60 regels)
- `.claude/rules/tone-of-voice.md` (stem-regels, ~90 regels)
- `.claude/memory/always/MEMORY.md` (project-status, ~50 regels)

Totaal: ~240 regels baseline context. Schoon, gefocust, geen ruis.

**Stap 2, Slash command.** Ik typ `/blog kalender`. Slash command leest de eerstvolgende blog uit mijn kalender en triggert de pipeline.

**Stap 3, Skill orchestratie.** `blog-orchestrator` skill activeert. Roept achter elkaar aan:

- `research-agent` (sub-agent, parallelle research)
- `seo-agent` (skill, keyword analysis)
- `blog-writer` (skill, draft schrijven)
- `seo-interlinker` (skill, interne links)
- `blog-images` (skill, visuals)
- `blog-editor` (skill, kwaliteits-check)

Per stap: receipt naar NDJSON (via PostToolUse hook). Output naar Telegram (via webhook).

**Stap 4, Mijn review.** Bij elke significante stap krijg ik een Telegram-bericht met "concept klaar voor review". Ik open in browser, lees, accepteer of vraag wijziging.

**Stap 5, Sessie eind.** Stop-hook draait `auto_commit_notes.sh`. Werknotities en memory-updates worden vastgelegd.

Geen handmatige coördinatie. Geen "wacht, welke skill moet ik nu aanroepen?". Het systeem weet de volgorde, ik beslis de quality gates.

> 📖 **Lees ook:** **[Agentic OS: de centrale bediening voor je bedrijf](/blog/2026-05-16-agentic-os-centrale-bediening-bedrijf)**: hoe agents, orchestratie en context samen een werkend systeem vormen

## Drie patronen die het verschil maken

Niet alle 38 skills zijn even belangrijk. Drie patronen die het meeste verschil maken:

### Patroon 1, Orchestrator skills

`blog-orchestrator` en `linkedin` zijn meta-skills. Ze coördineren andere skills. Een orchestrator skill heeft drie eigenschappen:

- Roept andere skills aan in vaste volgorde
- Heeft duidelijke quality gates per stap
- Slaat tussenresultaten op zodat je kunt onderbreken

Ik heb 4-5 orchestrator skills. Voor herhalende multi-step workflows zijn ze goud waard.

### Patroon 2, Skills met expliciete ICP

Elke skill weet voor wie hij schrijft/maakt. `blog-writer` heeft drie modes (NL MKB, NL AI peers, EN architects). `linkedin-writer` heeft eigen ICP-targeting. `cold-email` werkt per ICP-segment.

Zonder ICP-awareness zou de output generiek worden. Met ICP-awareness past elke output bij de doelgroep zonder dat ik het hoef te specificeren.

### Patroon 3, Skills met "anti-claims"

Elke skill in productie heeft een sectie "wat dit NIET doet". Voorbeeld in `blog-writer`:

- Schrijft niet over health/medical zonder disclaimer
- Schrijft niet over Vincent's vrouw of privé-leven
- Genereert geen testimonials over klanten
- Schrijft niet in andere persoon dan eerste-persoon

Anti-claims voorkomen dat skills dingen doen die jij niet wilt, vaak gevoeliger dan wat ze wel moeten doen.

## Wat ik zou doen anders bij een nieuwe setup

Drie eerlijke leerpunten van 8 maanden:

1. **Begin niet met 38 skills.** Begin met 3 voor je meest-herhalende taken. Voeg pas een 4e toe als die echt 80% bewezen waarde heeft. Mijn 38 zijn organisch gegroeid, niet ontworpen vooraf.
2. **Memory > skills voor onbeschermde info.** In het begin stopte ik veel context in skills. Hoort vaak in memory. Memory is goedkoper te updaten en blijft persistent.
3. **Hooks eerst, dan skills.** SessionStart-hook met memory-loader is meer waard dan 5 nieuwe skills. Bouw de fundering voor je optopt.

## Wat doe je deze week?

Drie acties, naar gelang waar jij staat:

**Beginner** (1-2 maanden CC):

- Maak één CLAUDE.md, één SessionStart-hook, één slash command voor je vaakste taak.

**Intermediate** (3-6 maanden CC):

- Voeg 2-3 skills toe voor terugkerende workflows. Zet rules.md op met testing- en security-regels.

**Advanced** (6+ maanden CC):

- Bouw een orchestrator skill voor één multi-step workflow. Kijk of memory-tier-systeem werkbaar is. Voeg context-monitor hook toe.

Niet alles tegelijk. De waarde komt uit *consistent gebruiken* van wat je hebt, niet uit het hebben van veel.

Veelgestelde vragen

## Samenvatting

38 skills, 11 sub-agents, 6 hooks, 4 memory directories, opgebouwd in 8 maanden Claude Code in productie. De waarde zit niet in het aantal, maar in de drie patronen: orchestrator skills voor multi-step workflows, ICP-aware output, en anti-claims om te voorkomen dat skills doen wat jij niet wilt.

Begin niet met deze schaal. Begin met CLAUDE.md, één SessionStart-hook, en één slash command. Voeg toe naargelang je workflow het nodig heeft.

In zes maanden heb je iets dat compounding werkt, net als bij elke discipline waar je consistent in investeert. Geen 38 skills nodig. Wel 38 weken trouw werken aan de essentie.

***

**Wil je sparring over jouw eigen Claude Code setup?** [Volg me op LinkedIn](https://www.linkedin.com/in/vincentvandeth/), daar deel ik wekelijks build-in-public updates over hoe mijn setup zich ontwikkelt. Of kijk op [mijn werkwijze-pagina](/ai-consultancy) hoe ik dit soort systemen bouw voor andere bedrijven.

***

> **Lees ook:** [Claude Code hooks: stop the agent from reading your .env](/blog/claude-code-hooks-stop-the-agent-reading-env)

## Lees ook

1. [CLAUDE.md voor beginners](/blog/2026-05-04-claude-md-uitleg-beginners)
2. [Rules.md uitgelegd](/blog/2026-05-08-rules-md-claude-code-codeerstandaarden)
3. [Hooks in Claude Code](/blog/2026-05-13-claude-code-hooks-automatische-acties)
4. [Top 10 Claude Code features](/blog/2026-05-09-top-10-claude-code-features-2026)
5. [Prompt engineering is dood: lang leve context engineering](/blog/prompt-engineering-is-dood-context-engineering)
6. [Intelligence beats memory: self-learning pipeline](/blog/vnx-intelligence-system-autonomous-monitoring)

claude code setup skills agents coordinatie cover

large_claude-code-setup-skills-agents-coordinatie_cover

medium_claude-code-setup-skills-agents-coordinatie_cover

small_claude-code-setup-skills-agents-coordinatie_cover

thumbnail_claude-code-setup-skills-agents-coordinatie_cover

claude-code-setup-skills-agents-coordinatie_cover

Mijn complete Claude Code setup: 38 skills, 11 agents, en hoe ik ze allemaal coördineer

Vorige week schreef ik over de [structurele security-blind-spot in het MCP-ecosysteem](/blog/2026-05-12-mcp-security-transport-niet-security-picnic): MCP is een transport-laag, geen security-laag, en de meeste community-servers vergeten dat. Vandaag de concrete vervolglijst.

Niet alle MCP-servers zijn veilig. Sommige zijn populair en *toch* structureel onveilig. Anderen zijn gebouwd door enthousiaste community-leden zonder security-achtergrond. En een paar zijn gewoon experimenten die nooit voor productie waren bedoeld maar toch zo worden gebruikt.

Op deze 2e Pinksterdag, een dag voor reflectie en sortering, een eerlijke lijst van 7 MCP-server-patronen die ik vandaag niet zou installeren. Met telkens het waarom en, waar mogelijk, een veiliger alternatief.

> **Disclaimer:** dit is geen persoonlijke aanval op individuele bouwers. Veel van deze servers zijn met goede bedoelingen gemaakt en functioneren voor hun beperkte use-case. De waarschuwing is voor wie ze zonder eigen security-review in productie wil zetten.

Zeven MCP-patronen die je in 2026 niet zou moeten installeren

## 1. MCP-servers die credentials in environment variables verwachten

**Het patroon:** server-README zegt "set `USERNAME` en `PASSWORD` als environment variabelen." De server gebruikt die credentials direct, zonder authenticatie van de aanroepende AI-client.

**Waarom risicovol:** Iedereen die toegang heeft tot het OS-process (denk: collega's die je laptop gebruiken, een corrupt dependency, een leak via debug-logs) kan via de MCP-server acties uitvoeren. Geen client-auth = open backdoor in jouw bedrijfssystemen.

**Voorbeelden in het wild:** veel community-servers voor SaaS-diensten (boekhouding, CRM, e-commerce platforms).

**Wat dan wel:** kies servers die OAuth 2.1 ondersteunen, of bouw zelf een wrapper die token-auth toevoegt voor jouw deployment. Voor lokaal-only experimenten op één machine: acceptabel, maar ban van productie.

## 2. MCP-servers zonder TLS / die plain HTTP toestaan

**Het patroon:** server draait op `http://localhost:port` zonder TLS, of erger: op `http://0.0.0.0:port` accessible voor het hele lokale netwerk.

**Waarom risicovol:** Iedereen op hetzelfde wifi-netwerk (denk: koffieshop, kantoorruimte, conference) kan de MCP-server aanspreken en data lezen. De `localhost` aanname is op gedeelde infrastructuur niet veilig.

**Wat dan wel:** voor STDIO-transports (lokale subprocess): minder kritiek, maar wel: zorg dat de subprocess onder dezelfde user als de client draait. Voor HTTP-transports: **altijd TLS 1.2+ plus client-auth**. Geen plain HTTP, ook niet "alleen tijdelijk".

## 3. MCP-servers met "iedereen kan alles" tool-access

**Het patroon:** een geauthenticeerde client krijgt automatisch toegang tot **alle** tools die de server aanbiedt. Geen scope, geen rate-limiting, geen per-tool autorisatie.

**Waarom risicovol:** een server die "lees-orders" en "plaats-order-tot-€10K" beide aanbiedt zonder onderscheid is een ramp wachtend om te gebeuren. Een gehackte client krijgt instant toegang tot de zwaarste actie.

**Voorbeelden:** sommige early e-commerce MCP-servers (waaronder versies van de [Picnic-server](https://github.com/ivo-toby/mcp-picnic) die ik vorige week noemde, niet als kritiek op de bouwer, wel als waarschuwing voor het patroon).

**Wat dan wel:** scope-per-tool. Lees-only scopes voor read-acties. Aparte schrijf-scopes met aparte tokens voor schrijf-acties. Dit verdubbelt het bouwwerk maar tien-voudigt de veiligheid.

> 📖 **Lees ook:** **[MCP is een transport-laag, geen security-laag](/blog/2026-05-12-mcp-security-transport-niet-security-picnic)**: waarom het MCP-protocol zelf geen bescherming biedt

## 4. MCP-servers gebouwd door één persoon zonder maintenance

**Het patroon:** een MCP-server op GitHub met laatste commit van 6+ maanden geleden, één maintainer die niet meer reageert op issues, en een dependency-tree die niet meer update.

**Waarom risicovol:** kwetsbaarheden in dependencies worden niet gepatched. Wanneer er een CVE komt voor een library die de server gebruikt, blijft de MCP-server kwetsbaar. Bij Anthropic-policy-veranderingen breekt de integratie zonder dat iemand het oplost.

**Concreet check:** open de GitHub-repo. Wanneer is laatste commit? Hoeveel open issues? Hoeveel hangende PRs? Geen maintenance = security debt.

**Wat dan wel:** kies servers van organisaties (Anthropic, GitHub, grotere community-projecten) of gemaintainde community-projecten met aantoonbaar reactietijd op issues. Of: bouw zelf, voor een specifieke use case, en maintain hem zelf.

## 5. MCP-servers die OAuth-tokens van gebruikers vasthouden

**Het patroon:** server vraagt jou om je OAuth-token van een dienst (Google Drive, Slack, GitHub) te plakken in een config file of environment, en stuurt die token mee bij iedere API-call.

**Waarom risicovol:** OAuth-tokens hebben vaak brede scopes. Een gestolen token van een MCP-server geeft een aanvaller dezelfde toegang als jij. Dit is *exact* het patroon dat in april 2026 bij OpenClaw fataal werd toen Anthropic third-party OAuth verbood.

**Wat dan wel:** servers die de officiële CLI van een dienst spawnen (`gh`, `gcloud`, `aws`) en die hun eigen auth laten regelen. De server houdt geen token vast, de CLI doet het. Veiliger en survivability is hoger bij policy-veranderingen.

## 6. "All-in-one" MCP-servers met 50+ tools

**Het patroon:** server biedt toegang tot 50, 100, of meer tools, bijvoorbeeld "een MCP voor je hele Google Workspace" met Gmail, Drive, Docs, Calendar, Contacts, Tasks, en meer.

**Waarom risicovol:** combinatie van factoren. Hoge complexiteit = hoge kans op bugs. Brede scope = bij compromise grote blast-radius. Eén CVE in één van de 50 tools opent de hele server.

**Wat dan wel:** kleinere, gescoptere MCP-servers per use case. "MCP voor Gmail-only" is betrouwbaarder dan "MCP voor heel Workspace". Composability boven mega-installatie.

## 7. MCP-servers die in production worden gebruikt na alleen "hello world" testing

**Het patroon:** dit is geen specifieke server. Dit is hoe je elke server gebruikt. Iemand installeert een MCP, doet een korte test, het werkt voor één case, en zet hem in productie zonder verdere validatie.

**Waarom risicovol:** zelfs een goed-ontworpen MCP-server faalt onder edge cases (rate-limiting, errors, malformed responses) op manieren die je niet ziet bij een hello-world test. Production-grade is iets anders dan "lijkt te werken".

**Wat dan wel:** drie weken proefdraaien op niet-kritieke use cases. Logging aanzetten van iedere actie. Periodiek manual review van wat de MCP-server heeft gedaan. Pas dan op kritieke business-data.

## De vier checks vóór elke MCP-installatie

Niet abstract, niet theoretisch. Vier directe checks:

**1. Check de README op auth-vermelding.** Geen woord over authentication = run away. Je gaat zelf bouwen of het niet gebruiken.

**2. Check de scope-per-tool.** Heeft de server gescheiden read/write capabilities? Of geeft hij authenticated clients alles?

**3. Check de last-commit-date.** Ouder dan 4 maanden zonder reden = security debt accumulating. Vooral bij community-projecten.

**4. Check de dependency-tree.** Een MCP-server met 200 dependencies is een 200-poorts attack-surface. Lichter is veiliger.

## Wat ik vandaag wel installeer

Voor de balans: drie patronen die ik wel vertrouw.

1. **Officiële MCP-servers van grote organisaties** (Anthropic-onderhouden, GitHub-onderhouden). Niet perfect, wel beter gemaintained.
2. **STDIO-transports voor lokaal-only gebruik** met servers waarvan ik de source heb gelezen. Geen netwerk-exposure = lager risico.
3. **Zelf-gebouwde wrappers** rond officiële CLI's (`gh`, `gcloud`, etc.) zodat de auth in de CLI blijft, niet in mijn MCP-server.

Dat is ~80% van mijn MCP-gebruik. De andere 20%, community-servers, alleen na een security-review en alleen voor niet-kritieke use cases.

## De drie meest gevaarlijke combinaties

Als afsluitende checklist: drie combinaties waarvan ik in 2026 *concrete incidents* heb gezien:

- **HTTP-transport zonder TLS + creds in environment**, netwerk-snifbaar, geen bescherming
- **All-in-one server zonder scope per tool + brede OAuth-token**, één compromise = volle toegang
- **Onmaintained server + production-data**, de combinatie die over 6 maanden in een nieuwsartikel komt

Bij twee uit drie checks: stop met installeren. Bij drie uit drie: nu meteen verwijderen.

## Samenvatting

Niet alle MCP-servers zijn veilig. Zeven patronen om in 2026 niet te installeren: env-creds zonder client-auth, plain HTTP zonder TLS, geen scope-per-tool, onmaintained projecten, OAuth-token-vasthouders, mega-servers, en pas-na-hello-world-getest in productie.

Vier checks vóór elke installatie: README op auth, scope per tool, last-commit-date, dependency-tree.

Voor productie of business-kritieke data: alleen officiële, gescoptere, gemaintainde servers. Voor lokaal experimenten: meer ruimte, met security-bewustzijn.

Op deze rustige 2e Pinksterdag is misschien een goed moment om je eigen MCP-installaties door te lopen. Vier checks, vijf minuten per server. Dat is goedkoper dan een security-incident over zes maanden.

***

**Wil je een audit of advies over jouw MCP-stack?** [Plan een gesprek](https://vincentvandeth.nl/contact), geen verkooppraatje, wel een eerlijk gesprek over wat veilig is in jouw situatie.

***

## Lees ook

> 📖 **Lees ook:** **[MCP is een transport-laag, geen security-laag](/blog/2026-05-12-mcp-security-transport-niet-security-picnic)**: het fundament van deze serie

> 📖 **Lees ook:** **[MCP servers en GDPR/AVG: verborgen risico's voor ondernemers](/blog/mcp-servers-gdpr-avg-privacy-risicos-voor-marketeers-ondernemers)**: privacy-kant van MCP-installaties

> 📖 **Lees ook:** **[De 10 beste MCP servers voor Claude Code in 2026](/blog/beste-mcp-servers-claude-code-2026)**: welke servers wél de veiligheidscheck doorstaan

> 📖 **Lees ook:** **[OpenClaw OAuth-ban: wat je kunt gebruiken in plaats daarvan](/blog/best-openclaw-alternative-cli-subprocess-oauth-ban)**: alternatieven die de OAuth-eis overleven

mcp servers niet installeren security cover

large_mcp-servers-niet-installeren-security_cover

medium_mcp-servers-niet-installeren-security_cover

small_mcp-servers-niet-installeren-security_cover

thumbnail_mcp-servers-niet-installeren-security_cover

mcp-servers-niet-installeren-security_cover

De 7 MCP-servers die je vandaag NIET zou moeten installeren (en waarom)

Een AI die suggesties doet die je consistent negeert, is alleen maar ruis in je inbox. Een AI die patronen herkent en suggesties doet die je vaker wel dan niet aanneemt, dat is een leerloop.

Het verschil is groot. En de meeste AI-systemen bevinden zich in de eerste categorie. Niet omdat ze technisch slecht zijn, maar omdat de feedback-mechanismen ontbreken. Het systeem leert nooit *welke suggesties* jij waardeert en welke je weggooit.

Vandaag voor AI peers en hobbyisten: hoe bouw je een self-learning loop die werkt? Geen autonomous AI die zelf beslist, wel een AI-systeem dat steeds beter wordt in *het soort suggesties* dat jij echt wil ontvangen. Concreet, met patronen uit mijn eigen [VNX-systeem](https://github.com/Vinix24/vnx-orchestration).

## Het verschil tussen automatisch en autonoom

Eerst de framing. Self-learning loops worden vaak verward met autonome AI ("het systeem leert en doet alles zelf"). Dat is niet waar ik het over heb.

**Wat ik bedoel met self-learning loop:**

- Het systeem **detecteert patronen** in jouw werk en stelt acties voor
- Een **mens beslist altijd** of de actie wordt uitgevoerd
- Het systeem **leert van wat jij accepteert of afwijst**
- Volgende suggesties zijn beter afgestemd op wat jij waardeert

**Wat ik NIET bedoel:**

- Het systeem doet zelf acties zonder jouw goedkeuring (= autonomous)
- Het systeem update zichzelf zonder review (= self-modifying)
- Het systeem trekt eigen conclusies over de business (= unsupervised)

Dat zijn drie hele andere systemen, met heel andere risico's. Autonome AI bestaat niet in een MKB-context op een betrouwbare manier. Self-learning loops met menselijke checks bestaan wel, en zijn productief.

Self-learning loop architectuur, patroonherkenning + suggestie + mens-beslist + feedback

## De vier componenten

Iedere werkende self-learning loop heeft vier onderdelen.

### Component 1, Patroon-extractie

Het systeem moet patronen kunnen extraheren uit jouw eerdere acties. Welke beslissingen heb je herhaald? Welke configuraties heb je consistent gekozen? Welke fouten heb je twee keer gemaakt?

In mijn praktijk komen patronen uit:

- **Receipts** van AI-dispatches, welke instructies leidden tot succes vs falen?
- **Edit-historie** in mijn projecten, welke patronen herstel ik herhaaldelijk?
- **Memory updates**, welke regels heb ik na correcties toegevoegd aan rules.md?

Deze drie bronnen samen geven een gestructureerde dataset. Geen vector-database nodig, geen RAG. Gewoon NDJSON of markdown, gestructureerd met audit-kolommen.

### Component 2, Suggestie-generatie

Patroon herkend? Vertaal het naar een concrete suggestie aan jou. **Belangrijk:** geen vage adviezen. Concrete acties.

Slechte suggestie:

> "Je doet dit vaak verkeerd, misschien moet je hier op letten."

Goede suggestie:

> "Patroon: in 4 van de laatste 5 PRs heb je `console.log` toegevoegd voor debug, en daarna handmatig verwijderd voor commit. Voorstel: voeg `console.log` toe aan rules.md als verboden in committed code, met uitzondering voor tests/scripts. \[Accept] \[Wijzig] \[Negeer]"

Drie verschillen: concreet patroon (4 van 5 PRs), concrete actie (rules.md update), concrete keuze (drie buttons).

### Component 3, Mens beslist

Hier is waar de meeste AI-systemen falen, ze suggereren te veel, of ze geven geen makkelijke manier om te accepteren of af te wijzen.

In mijn Mission Control: een suggestie komt als Telegram-bericht met drie buttons. Eén tap = beslissing genomen. Geen lange dialoog, geen bevestigingen, geen "weet je het zeker?".

Cruciaal: **de beslissing wordt vastgelegd**. Niet alleen accepteren/afwijzen, ook context. Als ik 'wijzig' kies, vraagt de bot kort waarom. Dat antwoord gaat terug in de loop.

### Component 4, Feedback in het systeem

De afsluitende stap. Wat doet het systeem met jouw beslissing?

**Bij accept:**

- Patroon wordt versterkt, vaker triggeren bij vergelijkbare situaties
- Eventueel direct toegepast (bijv. rules.md update)

**Bij wijzig + reden:**

- Patroon wordt aangepast op basis van jouw feedback
- Volgende keer wordt suggestie iets anders geformuleerd

**Bij negeer:**

- Patroon wordt verzwakt, minder snel suggereren
- Na 3x negeren: stop met dit type suggestie

Geen complexe ML. Gewoon een teller in een database, met regels voor wanneer iets wel of niet gesuggereerd wordt.

## Concreet voorbeeld uit mijn praktijk

Een self-learning loop die ik voor mezelf bouwde, voor mijn blog-pipeline:

**Periode 1 (eerste maand)**, bot leert mijn editing-patronen. Welke woorden vervang ik vaak? Welke alinea-structuren hertyp ik? Welke H2-titels keur ik af?

**Periode 2 (tweede maand)**, bot stelt voor: "Ik zie dat je in 6 van de laatste 8 blogs de term 'leveraging' hebt vervangen door 'gebruiken'. Voorstel: voeg 'leveraging' toe aan vermijden-lijst in tone-of-voice.md. \[Accept] \[Wijzig] \[Negeer]"

Ik klik Accept. Bot updates `tone-of-voice.md`. Volgende blogs: term niet meer gegenereerd.

**Periode 3 (derde maand)**, bot stelt voor: "Patroon in laatste 4 blogs: je hebt iedere keer een H2 'Conclusie' veranderd in 'Samenvatting'. Voorstel: vervang in blog-writer skill. \[Accept] \[Wijzig] \[Negeer]"

Ik klik Accept. Skill update.

**Resultaat na 3 maanden:** mijn blog-pipeline produceert iets dat *steeds dichter bij wat ik zelf zou schrijven* zit. Niet omdat AI mijn stem heeft geleerd. Wel omdat het systeem heeft geleerd welke aanpassingen ik consistent maak, en die preventief in de output stopt.

> 📖 **Lees ook:** **[Mijn AI-manager werd slechter naarmate ik langer met hem praatte](/blog/ai-agent-context-drift-headless-orchestrator)**: waarom patroon-extractie context-drift voorkomt

## De drie valkuilen die ik ben tegengekomen

Eerlijk over de fouten.

### Valkuil 1, Te veel suggesties

In mijn eerste versie kreeg ik 10-15 suggesties per dag. Verdronk in ze. Negeerde uit zelfbescherming.

Fix: **threshold-systeem**. Een suggestie wordt pas getriggerd na een patroon van 3+ herhalingen. En max 2-3 suggesties per dag, gerangschikt op confidence. Resultaat: ik kijk naar elk voorstel echt.

### Valkuil 2, Suggesties die jezelf bewijzen

Slechte timing van suggesties: ze komen wanneer ik het niet kan beoordelen. Bijvoorbeeld een rules.md-suggestie tijdens een focus-sessie.

Fix: **deferred suggestions**. Suggesties komen niet realtime, maar in een dagelijkse samenvatting (bij mijn `/status` command). Ik kies een rustig moment om ze door te lopen.

### Valkuil 3, Niet bijhouden waarom je negeert

Als ik een suggestie negeerde zonder reden, leerde de bot er niets van, de teller ging gewoon omhoog. Drie keer negeren = patroon verzwakt, maar het *waarom* miste.

Fix: **light-weight reden**. Bij negeren toont de bot drie korte redenen ("niet relevant", "te triviaal", "verkeerde context") plus "anders". Eén tap, twee seconden, drastisch betere learning-loop.

## Wat dit niet kan

Eerlijk over grenzen. Self-learning loops in deze vorm:

- **Kunnen geen creatieve patronen extraheren.** Als jij voor elke nieuwe blog een unieke hook bedenkt, ziet het systeem geen herhaalbaar patroon.
- **Kunnen geen domeinkennis vervangen.** Het systeem leert *jouw* patronen, niet wat universeel goede aanpak is.
- **Kunnen geen architectuur-niveau beslissingen suggereren.** Patroon-extractie werkt op micro-niveau (regels, woordkeuzes, kleine acties), niet op macro-niveau (welke architectuur kies ik).

Voor die soorten beslissingen blijft mens-mens-overleg of zelf nadenken het werkbaarste mechanisme.

## Wat doe je deze week?

Drie acties:

1. **Identificeer één micro-patroon dat je vaak herhaalt**, bijvoorbeeld een woord-vervanging, een opmaak-correctie, of een terugkerende rules.md-update na correcties.
2. **Bouw een minimale loop** rond dat patroon. Hoeft geen Telegram-bot te zijn, een markdown-bestand waar je elke week 5 minuten in noteert wat je hebt aangepast, werkt ook.
3. **Reviewen wekelijks of jouw rules/CLAUDE.md/skills** een update krijgen op basis van wat je in stap 2 hebt verzameld.

Dat is een handmatige self-learning loop. Geautomatiseerd is het mooier; handmatig is het al waardevol. Begin handmatig, automatiseer later.

## Samenvatting

Een self-learning loop is geen autonomous AI. Het is een mens-beslist-systeem dat patronen extraheert uit jouw werk en suggesties doet die jij accepteert, wijzigt, of negeert. De feedback gaat terug in het systeem voor de volgende ronde.

Vier componenten: patroon-extractie, suggestie-generatie, mens-beslist, feedback. Drie valkuilen: te veel suggesties, slechte timing, geen reden bij negeren.

Begin met één micro-patroon, handmatig. Automatiseer als de loop bewezen waarde toevoegt. In 12 maanden heb je een AI-systeem dat steeds beter is in suggesties die jij echt wil ontvangen, zonder ooit autonomie weg te geven.

***

**Vragen over zelfbouwen of automatiseren van een loop voor jouw workflow?** [Volg me op LinkedIn](https://www.linkedin.com/in/vincentvandeth/) of [neem contact op](https://vincentvandeth.nl/contact). Ik werk vaak aan vergelijkbare patronen met klanten. Meer over hoe ik dit bouw: [AI-architectuur & agents](/ai-consultancy).

***

## Lees ook

> 📖 **Lees ook:** **[Agentic OS: waarom AI de centrale bediening voor je bedrijf wordt](/blog/2026-05-16-agentic-os-centrale-bediening-bedrijf)**: hoe self-learning loops passen in een bredere agentic architectuur

> 📖 **Lees ook:** **[Prompt engineering is dood, lang leve context engineering](/blog/prompt-engineering-is-dood-context-engineering)**: de basis voor betere suggestie-generatie

> 📖 **Lees ook:** **[Waarom 'autonome' AI agents niet bestaan](/blog/autonome-ai-agents-bestaan-niet)**: het onderscheid dat self-learning loops scherp houdt

> 📖 **Lees ook:** **[Human-on-the-Loop: van monitoring naar productie-autonomie](/blog/human-on-the-loop-production-graduation)**: hoe je geleidelijk meer vertrouwen geeft aan je systeem

self learning loops suggesties mens beslist cover

large_self-learning-loops-suggesties-mens-beslist_cover

medium_self-learning-loops-suggesties-mens-beslist_cover

small_self-learning-loops-suggesties-mens-beslist_cover

thumbnail_self-learning-loops-suggesties-mens-beslist_cover

self-learning-loops-suggesties-mens-beslist_cover

Self-learning loops: hoe je AI-systeem suggesties doet die je daadwerkelijk wil opvolgen

Toen ik in 2024 over Claude Code sprak met ondernemers, ging het altijd over "prompt engineering". De juiste woorden, de juiste structuur, de magische instructie die output verbeterde.

In 2026 is dat verhaal voorbij. De grootste hefboom voor AI-output is niet langer de **prompt**, het is de **context**. Wat de AI **al weet** voordat je je prompt geeft. Welke files in zijn werkgeheugen zitten. Welke memory hij heeft geladen. Welke tools beschikbaar zijn. Welke ruis erin zit die hem afleidt.

Dat heet context engineering, een term die ik eerder dit jaar al introduceerde. Vandaag een dieper-praktische post: concrete patronen om je AI-context schoon, gefocust en hoog-signaal te houden.

Voor AI peers en hobbyisten die voorbij hun eerste 6 maanden Claude Code zijn, dit is waar de echte productiviteitswinst zit.

## Het probleem: te veel context wordt ruis

Claude Code heeft (mei 2026) een context-window van enkele honderdduizenden tokens. Dat klinkt als veel. Het is genoeg voor 400-500 pagina's tekst.

Maar wat blijkt in de praktijk: zodra een sessie boven de **65%** van zijn context-capaciteit komt, verslechtert de output meetbaar. In mijn eigen VNX-systeem: kwaliteitsscores onder 65% gemiddeld 0.86. Boven 65%: 0.78. Met grotere variatie.

Het probleem is niet "te weinig context". Het probleem is **te veel context met te veel ruis**.

Een AI met 200 pagina's irrelevant materiaal in zijn context werkt slechter dan een AI met 50 pagina's geconcentreerd materiaal. Dat is het hele inzicht achter context engineering.

## Patroon 1, Selectief laden, niet alles laden

De grootste fout van beginners (en eerlijk: ook van mij in 2024): **alle relevante files openen in één sessie**, voor het geval Claude er iets van nodig heeft.

Dat verspilt context-budget op materiaal dat 90% niet relevant is voor de huidige taak.

**Beter patroon:** geef Claude de tools om files **on-demand** te lezen, en vertrouw erop dat hij ze opent als ze nodig zijn.

Concreet: in plaats van bij sessie-start 12 files te openen, vertel je Claude bij start: "we werken aan feature X. De relevante files staan in `src/features/x/`. Lees ze als je ze nodig hebt."

Claude leest dan precies wat hij gebruikt, en de rest blijft buiten zijn context. Resultaat: 50-70% reductie in geladen context, hetzelfde takenresultaat.

## Patroon 2, Memory tier, niet memory dump

In mijn post over [de drie configuratielagen die Claude Code een geheugen geven](/blog/claude-code-rules-memory-setup) schreef ik over memory directories. Wat ik daar niet uitlegde: niet alle memory hoort bij elke sessie geladen.

**Slecht patroon:** SessionStart-hook laadt alle 8 memory-bestanden bij iedere sessie. Resulteert in 5K-15K tokens gevuld met info die voor 90% niet bij deze taak past.

**Beter patroon:** memory-tier-systeem.

Claude Code