My LLM Council: Four Modes, Five Lenses and the Questions I Give It

I call my LLM council from any folder through a small skill, and that skill keeps a log.

From 16 July to 17 September it holds 29 runs. Sixteen were strategy questions. One was a code sweep.

Runs started inside a code repo go to that repo's own store and aren't in this count.

The idea is simple. One question goes to five models from five labs, and each seat reads it through a different lens.

Four steps then build on each other. The last step may not invent agreement.

Why one model is an opinion

Research on LLM as a judge has measured the problem.

In MT-Bench, GPT-4 gave its own answers a 10% higher win rate than human judges did. Claude-v1 gave itself 25%.

The authors call that evidence inconclusive (Zheng et al. 2023).

Panickssery, Bowman and Feng (2024) found that the better a model recognizes its own text, the more it prefers it.

A jury of model families did better than one large judge.

Command R, Claude 3 Haiku and GPT-3.5 together reached a Cohen's kappa of 0.763 with human labels on Natural Questions.

GPT-4 alone got 0.627 and cost seven to eight times as much (Verga et al. 2024).

Mixing helps in review too. In a preprint from October 2026, one same-model plus one cross-model review found 56.7% of seeded errors (Song 2026).

Two same-model reviews found 42.7%.

Two reviews by the strong cross-model reviewer did about as well as the mix. The author can't separate a different model from a more capable one.

How my LLM council differs from Karpathy's

Andrej Karpathy made the term popular with llm-council, a Saturday hack he calls "99% vibe coded".

Models answer, rank each other anonymously, and a chairman writes the final response.

Mine has lived in vnx-orchestration since PR #1101 on 10 July 2026. The default roster has five seats from five labs.

Those are Codex (OpenAI), Kimi (Moonshot), Claude Sonnet (Anthropic), GLM via OpenRouter (Zhipu) and DeepSeek.

It differs from Karpathy's in three places. Every seat reads the question through its own lens.

One seat attacks the consensus. One step checks the strongest claims against the code or the sources.

1. Diverge. Each seat answers through its own lens.
2. Contrarian. One seat attacks the emerging consensus. What it attacks depends on the mode.
3. Verify. The top five claims are checked against the code or the sources. Each one gets CONFIRMED, REFUTED or UNVERIFIABLE, with the evidence.
4. Synthesis. One report with consensus, surviving dissent, verified claims and open questions.

Two prompt lines carry most of the design. From scripts/lib/deliberation_panel.py:

Four modes, five lenses each

A mode sets four things: the five lenses, the contrarian's target, what the verify step checks against and what the synthesis must deliver.

Every seat gets one lens, by its position in the seat list. Run fewer seats and the lenses at the end of the list drop out.

Sweep: a codebase or a set of documents

• Lenses: security and unsafe patterns, correctness and edge cases, dead code and unused surface, refactoring and simplification, performance and resources.
• Contrarian: the severity ranking and every "this is fine". Which flagged issue is a non-issue, and which unflagged area is the real risk?
• Verify: reads each cited file:line and checks that it supports the claim.
• Delivers: a ranked, deduped findings list and the single fix with the most leverage.

I used it to check all PRs merged over four days for collisions, contribution to the product and AI slop.

It complements multi-AI review at the merge gate, which looks at one diff at a time.

Architecture: a design and its trade-offs

• Lenses: the clean design and its data model, feasibility and effort, operational reality (failure, rollback, observability), alternatives nobody proposed, maintainability and coupling.
• Contrarian: where the design breaks under load or over time, and the simpler alternative it dismisses.
• Verify: checks each feasibility and effort claim against the code.
• Delivers: a decision doc with the rejected alternatives and a phased rollout with rollback.

Questions I gave it: was a process for commercial blueprints too heavy, and what is the lightest version with the same effect?

What refactor plan for file-size debt could a cheaper model execute on its own? How do I measure tokens and time per PR without adding overhead?

Strategy: a business or product call

• Lenses: the opportunity, the execution path and resources, the risk and what could kill it, the market and customer reality, sequencing and what comes first.
• Contrarian: the core assumption. What has to be true for this to work, and why might it not hold?
• Verify: flags every assumption presented as a fact.
• Delivers: a one-page call with the recommendation, the bet it rests on, the surviving risks and the first move.

This is where most logged runs go: 16 of the 29. Open core or fully open for a product whose moat is the service?

Does a fixed-price project fit its budget and six weeks, given what I can reuse? Which lead source should a client's lead machine get first?

My LinkedIn system collected well and delivered nothing: was that the technology, the design or the operator?

Research: a market or a source question

• Lenses: market size and demand signals, competitors and incumbents, trends and timing, risks and headwinds, the underserved wedge.
• Contrarian: the optimistic consensus. What is the strongest case that this thesis is wrong or already lost?
• Verify: tries to refute each top claim against the sources.
• Delivers: a cited briefing with confidence per claim and the three decisions it should inform.

One run asked for the best source for a starting list of Dutch companies and their decision-makers, given that Sales Navigator has no native export.

Plans before code

The same idea runs as a gate. Before a feature gets built, a panel of several model families reviews its plan.

Each seat votes PASS, REVISE or BLOCK. The gate turns that into PASS, REVISE or INFRA_FAIL. A single BLOCK makes the round a REVISE.

INFRA_FAIL means no seat returned a readable verdict. It is an infrastructure outcome, never a judgment on the plan.

The full panel gets at most two rounds. Without a PASS by then, one tiebreaker model decides START or STOP. On STOP the remaining gaps become open items.

Every seat's verdict lands in an append-only, hash-chained ledger. Later I can see whether each seat actually answered.

Seats per call, and who sees the data

The --seats flag runs part of the roster. That serves two purposes.

The first is independence. For a review of AI-written code I can take the writer's lab off the panel.

A reviewer from the same lab is not an independent check. After the Song preprint I treat that as a choice per review, not a rule.

The second is data. Outside the repo the panel runs through a thin skill called business-panel.

It calls the one implementation in the repo, so a second copy can't drift.

The skill offers the full roster, a core of Codex and Claude, or my own list. Before a run it names which providers will see the content.

Three of the five seats run models from Chinese labs: Kimi and DeepSeek at the lab itself, GLM through OpenRouter.

For client data I want to know that before the question leaves my machine.

The dispatch layer enforces the provider rules: Kimi only through its CLI, GLM only through OpenRouter, DeepSeek only on its own key.

A CI check keeps the Anthropic SDK out of the code.

The guardrails

A coverage floor. Below three seats with real content, the synthesis refuses. From configs/plan_gate_panel.yaml:

I can push past it with --allow-degraded. Then it's my decision and it's on record. A run where no seat delivered never exits with code 0.

It fired in October. A code sweep lost two of its four seats, and the report said "refusing synthesis: 2/4 lenses delivered (minimum 3)".

The skill log shows two more refusals, at 1 of 3 and 2 of 5 seats. A council that writes a confident report from two opinions looks like a check and isn't one.

A coverage line. Every report opens with how many lenses actually answered and which ones failed:

The line wasn't always right. In August it showed 4/5 on a run where nothing had delivered, because it counted dispatch records.

Now it counts against the receipts.

Verify against the source, not against the seats. Different makers are not independent judges. Kim et al. (ICML 2025) compared more than 350 models.

On one leaderboard dataset, two models that are both wrong pick the same wrong answer 60% of the time.

Five models with a shared blind spot are still one blind spot.

Choi, Zhu and Li (NeurIPS 2025) show that majority voting explains most of the gain attributed to multi-agent debate.

Defaulting to REFUTED or UNVERIFIABLE when evidence is thin is my way to steer toward correction instead of agreement.

Check the receipt, not the label. One report labelled the DeepSeek seat "Model: sonnet, Provider: claude".

The seat fills in those two fields itself. Inside the Claude Code harness, DeepSeek wrote down the harness's identity instead of its own.

The receipt named the DeepSeek model. The model name is not the model, and the label is not the seat.

📖 Read also: One Harness, Five Model Families: how Codex, GLM, Kimi and DeepSeek run through Claude's agentic loop.

Where it breaks

• The lenses are fixed per mode. One research run in the log carries the note "MODE-MISMATCH": research lenses on a technical question about voice models. My own six angles for a code review once had to go into the question, because they didn't fit the five sweep lenses.
• Seats fail. The log notes a failed, empty or broken seat in 12 of the 29 runs. In two of them not one seat delivered, because a panelist role wasn't registered. Four runs are reruns of a question that had already run. Skip the coverage line and you read one opinion as a panel.
• The log is light. It records that a run happened, not that its report survived. One logged report is nowhere on disk.
• Seats don't always answer the question. In one sweep DeepSeek gave one score where six were asked. Codex once delivered only a summary, because my panel prompt passed no report path.
• Remove a lab and another one writes the last word. Every later step picks a seat in order of preference. Codex runs the verify step by default. With Claude off the panel, it wrote the synthesis too.
• The cost is lopsided. In one sweep round the receipts book Codex at $70.04 over four steps and the DeepSeek seat at $0.035. The Codex figure is list price kept for bookkeeping: it runs on a ChatGPT login.

If you build one

• Pick lenses per type of question, not one generic prompt.
• Give the contrarian a target per mode.
• Verify against the artifact: code, source or receipt.
• Put the coverage floor in code and make the refusal loud.
• Name the providers before the data leaves.

Building a council into your own decisions

The council doesn't make the decision. It shows where five labs agree, where they don't and which claims survive a check against the source.

It also refuses to answer when too few seats delivered. That is what I want from a second opinion.

I design setups like this for teams that let AI write code or prepare decisions, and want the check arranged before the verdict. More on how I work as an AI consultant.