October Build Log: Unknown Is Not Zero

This build log covers October. Same method as before: I log every failure worth remembering, the ones where I believed something that was not true. Once a month I read them back.

October in numbers, counted from git and the receipt ledgers:

Around 11,100 receipts landed in the project ledgers in October. 30 blogs went live.

Reading the month back, one shape dominates. A check asks a question, and most of those questions have three honest answers: yes, no, and I could not look.

Almost every expensive failure in October came from a check that only had two. The third answer existed. It just came out as yes.

Unknown is not zero. Here are four places where it was.

1. The SEO tool: a missing field became a measurement

The first end-to-end scan on a real site wrote 38 findings. Most of them were false. "Very thin content" on a page of 2,352 words. "No internal links" on every page, while each page had between 17 and 41.

The normalizer read six fields under names the crawler no longer used. Every field it could not find, it turned into a 0.

The detector downstream read that 0 as a measurement and reported it with full confidence. The chain had been switched off since May, so nobody noticed the names had drifted apart.

The fix is small. The normalizer reads the real fields, and a missing source now produces no fact at all instead of a zero.

I bumped the versions of the detector, the prioritizer and the scoring so the old false rows age out. The same scan now writes 11 findings, each one in line with the crawl data.

A default of 0 is not neutral. It turns every measurement you failed to take into a confident, wrong statement.

2. The business layer: 59 nights red, then green with nothing

A nightly job in the business layer turns published blogs into concepts for other channels. In October I found it had exited with code 1 every night since it was installed on 23 August.

That is 59 nights. The job never loaded the environment file with the database URL, so the client refused to start. Nothing read the exit code.

I fixed the environment and the job went green. It still produced nothing: zero concepts in production since the day it was built.

Three faults were stacked on top of each other. Handing work to the queue failed because the queue app was not open in that standalone process, and the failure was caught as a warning.

The task then ran on a worker on another machine and read a blog folder that only exists on my Mac. The file-not-found error was booked as success. And the event emitter looked for a key under a different name than the environment provided.

Fixing the top layer alone would have given me a green run with 15 silent failures underneath. So the job now runs locally, takes at most two blogs a night from the last 60 days, and exits with 1 for every blog that fails. The first run made 9 concepts.

A job that ends green proves it ended. Count what it made.

3. The orchestration framework: the code under review became the diagnosis

My review gates send each change to reviewers running other vendors' models. In October two of those reviews were booked as "quota exhausted" after they timed out at 900 seconds.

The error line was the string form of the timeout exception. That string is the whole start command, including the diff under review: 164,000 characters.

The quota detector scanned that text and found "insufficient balance" and "reached your usage limit". It concluded the lane was out of credit. Both phrases were test fixtures in the code being reviewed.

So the code under review decided why its own review failed.

A timeout is now called a timeout, on an explicit signal from the lane itself. No error line carries the command or the prompt anymore.

The deadline scales with the size of the diff, from 900 to 3,600 seconds, and the next long review ran on for 1,126 seconds where it used to stop at 900.

The fix had a sequel. With the longer deadline, a review cut off at its deadline could still count as a pass, because the gate read a half-written verdict.

The reviewer flagged that as blocking in the next round, and the fix went into the same pull request. The last edge case closed in a follow-up pull request.

If a diagnosis scans free text, you have to know whose text it is.

4. The orchestration framework: an allowlist is only as complete as its list

Before a worktree is deleted, a salvage step saves what is in it, and a verify() call confirms nothing was lost. Two pull requests for this failed three review rounds each.

Every round, the reviewer found a new case where verify() said clean while something got lost: staged work, a reused folder, a bundle that borrowed objects, a dirty submodule.

When a reviewer finds a new hole every round, look at the shape of the check before you look at the cases. verify() was a denylist. It looked for known problems and said clean to everything else, so every new kind of state was a new hole.

I rebuilt it as an allowlist. Only state the salvage provably preserved may come back empty. Every other line is a named problem.

The reviewer still found a new hole every round. A populated submodule. A backup inside the worktree. And in round three, ignored files that the status command never showed.

The allowlist checked what the status command listed, and that command left out ignored files. An allowlist over an incomplete listing is a denylist with extra steps.

The rebuild is now split in two. The first part fails closed on every status line, ignored files included. Only the second part adds an explicit list of paths that are safe to throw away.

"Only allow what you know" works only if you can see everything first. The listing is the real contract.

The reviewer, measured

Most of the holes above were found by the same reviewer, so I checked whether it is simply strict. I had my orchestrator read its 50 most recent findings and sort them.

Of the 11 it marked as blocking, 9 were serious, 2 were small and none were noise. For 29 of the 50 it reproduced the failure in the checkout before reporting it.

The full comparison, every reviewer on the same commits and counted in both directions, gets its own post on 1 December.

August and September in three lines

I did not write build logs for August and September. Three entries from those months belong here, because they have the same shape:

• 609 of 11,089 tests in the business layer carried no marker, so no gate job ever selected them. The gate reported nearly ten thousand green and looked exhaustive.
• A converter read the model choice from a key that existed in 1 of 765 routing decisions. In every other case it fell back to what the worker said about itself, so the receipts recorded the worker's claim instead of the router's decision.
• I asked where the lost time in 35 orchestrator sessions went. The 18 fixes I had shipped would have prevented, fully or in part, about 2% of it. 74% of the minutes was waiting and escalation.

The first two are checks that could not see and said fine. The third is the same thing one level up: I fixed what irritated me before I measured where the time went.

📖 Read also: July build log: merged is not deployed, and green is not verified: the earlier lessons this section corrects.

What earlier build logs got wrong

July said fail-loud became the default in every control path. October showed that loud is half of it.

The nightly job failed loudly for 59 nights, and that changed nothing because nothing read the exit code. A loud failure needs a reader. A quiet success needs a count of what it produced.

"Merged is not deployed" came back at least five times. Once in my own words: I said a fix would reach every project at once, and an hour later another project still failed. Only one project runs the repository directly. The others run a pinned release.

June said: keep the human merge gate exactly where it is. That is no longer true.

Between 16 and 28 October, 258 framework merges went through without me. 162 needed both reviewers, 85 only the first, and 11 left no record of which. The SEO tool's orchestrator merges on its own as well.

The one-line version

A check has three answers: yes, no, and I could not look. Every failure above had only two, and the third came out as yes.

July's question was what a check reports when the thing it measures is broken. October adds a narrower one: what does it return when it cannot look? If the answer is "the same as when everything is fine", you have a default with a check's name on it.

Previous entries: July, June, May, April.

If you are putting agents into production and want this kind of record without building it yourself, that is the work I do as AI architect.

The orchestration framework is open source: github.com/Vinix24/vnx-orchestration.