This build log covers July. I log every failure worth remembering. Not the typos, the ones where I believed something that was not true. July produced 678 entries across five projects: 265 in the orchestration framework, 174 in the sales copilot, 163 in the business layer, 42 in the cockpit, 27 in the SEO tool. 255 of them are marked as fully shareable.
Reading them back, four categories dominate. Locking and concurrency, roughly 101 hits. Migrations and schema drift, 86. Audits and audit trails, 73. Deploy versus merge, 67.
They look unrelated. They are the same failure. In every case I was watching a signal that stood in for the thing I actually cared about, and the signal was healthy while the thing was broken.
That is the whole month in one sentence. Here are the four, with the specific way each one lies.
1. Concurrency: shared state you did not know was shared
The pattern that cost the most calendar time, and the one I was most confident about beforehand.
I run background agents in parallel, each in its own git worktree. Isolated, obviously. That is what a worktree is for.
Then my own commit landed on an agent's branch, and the branch I intended stayed empty.
Worktree-isolated agents share the object store and the refs with the main checkout. A concurrent branch creation inside an agent raced with my checkout-and-commit in the main directory. The isolation is real for files and not for refs, and I had generalized from the first to the second.
The fix was surgical (move the branch pointer, fast-forward push, no force needed). The lesson was broader: isolate your own work too, not just the agents'. Either serialize, or work in a worktree yourself. Do not be the one unisolated writer in a system built for parallelism.
The same shape showed up elsewhere in a much larger form. An activation script silently doubled an entire timer fleet: 54 timers firing twice, so duplicate scrapes, duplicate syncs, duplicate scoring. No monitor fired. The script checked for the canonical name, saw the already-running legacy-named timers as MISSING, and installed duplicates alongside them.
The monitoring checked whether a job was running and whether the data was fresh. It did not check whether the job was running once. Presence, not count.
Deduplication brought 147 timers back to 92.
2. Schema drift: two ledgers that disagree
Activating a merged deploy batch, I found that the database's own migration history and the repository's numbered migrations both had entries 186 and 187, with entirely different contents. One said anon-revoke, the other said taste weights.
Two independent tracking systems for the same thing, drifted apart. On top of that, handover notes claiming "youtube not deployed, finance waiting" while most of it was already in production.
There is no clean way to reconcile two ledgers that disagree, because the disagreement is the evidence that neither is authoritative. So I stopped trying.
When two migration ledgers disagree, the schema itself is the only ground truth. Not the log, not the handover note. Does the table exist, does the column exist, what type is it. Introspect the objects, derive the actual applied state from that, and apply only the genuine delta.
What felt like a 35-migration tangle turned out to be a small delta. The panic came entirely from trusting logs that had no reason to be right.
3. The audit that degrades to green
This is the one I keep writing about, and July gave me the cleanest example I have had.
I built a new fabric audit: checks for split-brain state, per-project ledgers, hash-chain integrity. It went through three rounds of an automated review gate. Rounds two and three caught the same thing, which is why the gate exists.
The audit reported clean while it could not read its own input. A malformed project registry silently became an empty project list, so the per-project check iterated over nothing, found no problems, and returned green.
Graceful degradation in exactly the wrong place. An audit tool that degrades to green when its input is broken is worse than no audit at all, because it manufactures confidence. There was also an except: pass on a marker read, which is the same mistake in miniature.
The fix was to fail loud everywhere: an unreadable-but-present registry returns RED instead of quietly empty, a failed chain-verification import returns WARN instead of a silent skip, and project identifiers get validated as safe path components before use. PR #1045, 17 tests.
The expensive bug in a control system is not a false RED. It is a false GREEN.
A related discovery, less dramatic and more structural. I wanted to pull fleet-wide tool failures out of the governed audit trail. They are not in there. Of 15.407 receipts at the time, zero had a tool-call field.
The receipts are a provenance ledger: which dispatch produced which commit, which pull request, which gates passed. That is a complete record of what happened and a blind spot for how the system struggled getting there. Provenance is not observability, and I had been treating one as if it covered the other for months.
The third variant caught me from the other direction. I was testing whether a cost calculation recognized the model names that actually get stored. Everything came back empty, including the names I was certain were in there. I nearly concluded the whole calculation was broken.
It was my test. I passed an object where a string was expected, so the lookup failed on everything, including my own control case.
That the control case failed too was the signal that the fault was mine. Always include a case in your measurement that you know must pass. If it fails alongside the rest, you are measuring yourself.
Once corrected, there was a real finding underneath: the lookup knows the names it expects and none of the names actually stored. Different spelling, different prefixes. 1.377 records with no cost attribution, silently.
4. Merged is not deployed
The last category is the most embarrassing, because it is the one I would have said I already knew.
After a merge I considered a bug fixed. The consumer that reported it kept hitting it. The fix was on main; the consumer was running a channel.
The fleet runs a centrally installed, versioned copy behind a symlink channel. Merging to main does not touch that channel. The channel has to be updated explicitly: pull main, flip the symlink, migrate the stores. Until that happens, "merged" and "deployed" are two different states and only one of them matters to the user.
Done meant done only after running the channel update, verifying the symlink, and testing end-to-end from the consumer's own directory with no workaround flags.
A cousin of this one, from the same week. After five squash merges I went to clean up orphaned worktree branches. Git reported them as NOT merged, for branches whose contents I had merged minutes earlier.
Squash merge creates a new commit with a different hash. The original branch commits are not ancestors of main even though their content is. The ancestry check measures lineage, not content.
The tooling meant to keep you safe raises a false alarm at precisely the safest action. Know your merge strategy before you believe a data-loss warning, or you will either panic or, worse, learn to ignore the warning.
What this build log changed
Three things came out of this month that are not just lessons.
Enforcement moved to the door. The audit trail showed 13 merged tracks that had been built and merged without their planning gate ever passing. The gate existed. It guarded the bookkeeping at close time and nothing consulted it at dispatch or merge, so enforcement lived only in soft instruction text and any autonomous run walked straight past it.
A governance rule you only observe is a suggestion. It is now enforced at the dispatch door, the one place everything passes through, advisory-first with a signed and audited operator override.
Fail-loud became the default in every control path. Not just the audit. Anywhere a check could return "nothing to see" because it failed to look, it now returns an error instead.
I pointed my own review panel at my own framework. The question was whether the codebase is more complex than what it does. The answer was uncomfortable and useful: a solid orchestration core, plus a governance layer running ahead of its own demand, with half of it in an undeclared state. Built and tested, neither deliberately on nor deliberately parked.
That last category is the one worth naming, because I suspect it is common. Not dead code and not live code. Code in a state nobody decided on. Every subsystem now has a decision attached: keep, prune, deliberately park, or activate and measure.
The one-line version
Every failure above has the same shape. Process alive is not work finished. Merged is not deployed. Green is not verified. The ledger is not the schema. Running is not running once.
In each case I picked the signal that was easiest to measure and treated it as the signal that mattered. That substitution is invisible while everything works, which is exactly why it survives long enough to cost you something.
The question I now ask before trusting any check: what would this report if the thing it measures were completely broken? If the answer is "the same as now", it is not a check.
Previous entries: June, May, April.
If you are putting agents into production and want this kind of record without building it yourself, that is the work I do as AI-architect.
The orchestration framework is open source: github.com/Vinix24/vnx-orchestration.
Vincent van Deth
AI Strategy & Architecture
I build production systems with AI — and I've spent the last six months figuring out what it actually takes to run them safely at scale.
My focus is AI Strategy & Architecture: designing multi-agent workflows, building governance infrastructure, and helping organisations move from AI experiments to auditable, production-grade systems. I'm the creator of VNX, an open-source governance layer for multi-agent AI that enforces human approval gates, append-only audit trails, and evidence-based task closure.
Based in the Netherlands. I write about what I build — including the failures.