Building a Real-Time Sales Copilot Solo in 30 Days: 8,040 LOC, One MLX Crash, and Two Different Fixes

TL;DR

I built a real-time sales copilot solo over 30 days. AGPL-3.0, 8,040 lines of Python, 110 commits, 52 test files, 247 tests in the suite. Runs entirely on Apple Silicon. It captures mic and system audio in parallel, transcribes both streams with Whisper, classifies each utterance against twelve Dutch B2B pain-point categories, and pre-loads matching case slides into the live screen-share. v1.0.0 was tagged on day 13. On day 24 MLX threw two distinct Metal-level SIGABRTs during real continuous inference. The system kept shipping because on day 9 I had merged a whisper.cpp backend adapter behind the same TranscriptionBackend Protocol. Same-day swap. v1.0 shipped on time. Five days later (day 29, PR-102) I replaced the dual-engine architecture with a single InferenceWorker fed by a priority queue, that eliminated the entire class of Metal command-buffer races at the source.

This is an honest write-up of what works, what failed, and what I would do differently. Code is at github.com/Vinix24/sales-copilot (AGPL-3.0).

Why I built this

There is no open-source real-time sales copilot. Gong, Chorus, and Fireflies all sit on the post-call analytics side, all cloud-first, all priced €75–€200/seat/month. For Dutch B2B sales teams, that pricing is the smaller problem. The bigger problem is procurement: AVG/GDPR plus the EU AI Act plus 2024 Dutch DPA enforcement makes "we record your sales calls into a US cloud" a six-week conversation with legal and IT, every time.

Apple Silicon changed the math. Whisper-large-v3-turbo runs at ~0.1–0.3× real-time factor on an M4 with about 1.6 GB of GPU memory. Silero VAD does its job in under 10 ms a frame on CPU. semantic-router with a local embedding model handles classification on every transcript chunk without network. The only thing missing was someone willing to wire it together as an OSS-first product instead of a closed SaaS.

I built it because I wanted it. I run my own sales calls. I do not want to ship audio of a prospect to a US transcription provider to find out which case study to pull up next. The €0 self-hosted price tag is a side effect, the real reason is the data never leaves the laptop.

The first commit (da4085f, 2026-04-08) is "Initial commit: docs, config, scaffolding (no source code)." The second commit set up VNX governance for the build. The third commit was the first WebSocket hub. Day 1 was a PRD, a TTD, and a research report, not code.

📖 Read also: Glass Box Governance: Receipts, Not Chat Logs: the governance architecture that made this build possible

Architecture decisions

The system splits into four modules, each independently runnable:

1. Talk-time tracker, Silero VAD on two audio streams, breathing-bar dashboard, monologue alert at 76 s. Zero AI dependency. Shipped first because it provides standalone value with no model risk.
2. Live transcriber, Whisper through a pluggable backend layer. Originally WhisperLiveKit, then mlx-whisper direct, then whisper.cpp behind the same Protocol.
3. Pain detector, semantic-router (embedding-based classification) does fast routing on >90% of cases. Only uncertain matches escalate to an LLM via instructor. The LLM is provider-agnostic across Gemini, Groq, OpenAI, and Ollama; nothing in the code knows which one is configured.
4. Reports, async event consumer that writes JSON + Markdown post-call reports.

Inter-module communication is WebSocket on localhost only. Channels: /ws/transcript, /ws/talk-time, /ws/pain-points, /ws/slide-control, /ws/coaching, plus later /ws/objections, /ws/suggestions, /ws/summary, /ws/config. No external network calls except the optional cloud LLM, and that one is opt-in per route.

A few calls I would repeat in every project:

• Localhost-only hub. No NAT, no auth, no TLS termination, no observability platform. The system runs in front of a single human operator. Simplicity is a security property.
• Vanilla HTML/CSS/JS dashboard. Explicit rule in CLAUDE.md: no build step. The dashboard is one HTML file plus three JS files served from file:// and pointed at http://localhost:8760. There is no React, no bundler, no Vercel deploy. It will still work in 2032.
• Dual-stream audio instead of diarization. Mic is self, system audio is prospect. Diarization-quality bugs are a class I refused to inherit. I built three audio backends to make sure I always have a way to capture both: AudioTee (Swift, Core Audio Taps), BlackHole (loopback), and Loopback (commercial). Each platform/macOS version had different constraints.
• Pre-loaded hidden slides instead of dynamic DOM injection during screen-share. Reveal.js renders the case-study deck on call start; the orchestrator only flips visibility on a matching pain detection. Anything that mutates the DOM under a live screen-share is a class of bug I did not want to debug live.

The architectural call I would repeat in every project: the transcription backend was a Protocol from day one, not from day twenty. That decision saved me on day 24. From src/sales_copilot/modules/transcriber/backends/base.py:

Adapters emit raw payloads to a callback. TranscriberEngine is the single place that converts raw backend output into the canonical transcript event published to /ws/transcript. Swapping mlx-whisper for whisper.cpp meant pointing one env var at a different adapter. The downstream consumers, dashboard, detector, reports, never saw it happen.

Timeline

I did not write a line of source code on day 1. By day 6 I had a certified four-module system because day 1 was spent on the PRD. The governance-first cost up front bought me the right boundaries to refactor, swap backends, and cut dead code without breaking downstream consumers. The Glass Box pattern earns its keep on weeks two and three, not week one.

The hardest engineering problems

Dual-stream audio without diarization

Three audio backends in parallel was the correct call and also more work than I budgeted for. AudioTee is a small Swift binary I built around Core Audio Taps; it works on Sonoma+ but needs entitlements I had to learn. BlackHole is the most permissive loopback option but requires the user to route their meeting app through it. Loopback (the commercial Rogue Amoeba product) is the easiest UX but I cannot ship it.

The architectural decision that made all three viable was treating mic and system as physically separate streams from capture through publication. There is never a "merged audio" representation anywhere in the pipeline. Two MicStream instances, two transcribers, two VAD loops, two transcript channels with speaker: "self" and speaker: "prospect" baked into the canonical event. No diarizer to misattribute "yeah" to the wrong party. No speaker-embedding model to retrain. Every problem class that diarization brings, speaker confusion, overlap handling, voiceprint drift, got eliminated by accepting a hardware constraint instead of a software solution.

WebSocket race conditions (PR-95.1)

This is the one I am most embarrassed about. The orchestrator broadcasted call_started on /ws/config at __main__.py:242, then awaited _run_call(...), which then spawned the transcriber, talk-time, and detector modules. Each module subscribed to /ws/config on startup. The hub was fire-and-forget. By the time the modules connected, call_started was long gone. Transcriber blocked at whisper_direct.py:63 on start_event.wait() indefinitely. No transcription. Empty transcript pane. Talk-time logged "Speech event:" because the audio loop ran VAD unconditionally but _HeartbeatController._call_active stayed False, so nothing got broadcast.

I worked through four hypotheses in a deep-debug session, A (BlackHole capture conflict), B (subscribe-after-broadcast race), C (warmup raises silently), D (single-stream divergence). A, C, and D were refuted by direct observation in the logs. B was confirmed by tracing the subscribe order. The fix landed in commit db35e68: sticky-replay on the hub. When a client subscribes to /ws/config while _call_active is True, the hub immediately sends {"type": "call_started", "config": <latest_config>} to that single subscriber. Live broadcasts unchanged.

The lesson is older than WebSockets. In any async pub/sub system, joins-after-broadcast is the default failure mode, not the edge case. Two follow-up patches (PR-96.3 sticky transcripts, PR-98 null-speaker handling) applied the same pattern in the same week. The broader fix, replacing fire-and-forget config events with a small sticky current_call state plus a GET /api/call endpoint, is filed under refactor opportunities and not yet done.

Whisper hallucinations on silence

Whisper-large-v3 is excellent at Dutch business speech and abysmal at silence. It does not output empty strings. It outputs the most plausible string given the prior, and the prior was trained on roughly the entire YouTube-plus-public-broadcasting corpus. So during a quiet moment in a sales call, the model confidently produces:

Every entry on that list is a real outro the model memorized. tv gelderland and omroep gelderland are Dutch regional broadcaster sign-offs. vertaald door and geredigeerd door are subtitle-team credits. thanks for watching and subscribe to are YouTube outros. Whisper learned these as silence-priors because the original training data had millions of audio clips ending in exactly that pattern.

I considered three fixes: lower temperature (helps a little, hurts accuracy on real speech), VAD-gating before inference (helps but my VAD already runs and the buffer flush is independent), and post-filtering. Post-filtering won on every axis. The blocklist lives in whisper_direct.py, takes ten lines, and runs in microseconds. Pragmatic beats correct here. Retraining the model to not produce broadcast outros on silence would cost me a month and would still leak some other hallucination class into production.

The MLX disaster

On day 24, the system threw two distinct Metal-level errors during real continuous inference on M4:

Both map onto the active MLX thread-safety tracking issue ml-explore/mlx#2133. The default stream and StreamContext were not safe to share across threads at the version I had pinned, and a separate failure mode bubbles GPU-completion errors out of a libdispatch callback as an uncatchable abort() from Python. Recent MLX work has added per-thread default streams behind explicit stream APIs, but adopting that fix is a multi-day refactor of how the backend creates and pins streams, not a drop-in upgrade.

My setup hits this cleanly. Two parallel transcribers, mic and system, both calling into mlx-whisper, both sharing the default MLX stream, race condition by construction. For a production system that needs to ship today, reworking the MLX integration around per-thread streams is a multi-day refactor I do not want to do under pressure.

I did not have to. On day 9, fifteen days before I needed it, I had merged the whisper.cpp backend adapter (483c594) and written docs/WHISPER_CPP_DESIGN.md describing it as "a viable backend addition, but not a one-file swap. The correct implementation is to treat it as a new local inference adapter under the existing transcript event pipeline." On day 12 (c3676df, 2026-04-19) I vendored whisper.cpp as a pinned git submodule. The adapter went into CI. I ran one manual end-to-end test against it and put it on a shelf.

That afternoon the swap was three changes:

1. __main__._eager_warmup_at_startup reads TranscriberConfig.from_env() and routes through create_backend(), which now picks whisper.cpp when WHISPER_BACKEND=whisper.cpp.
2. whisper_cpp_backend.py flag fix for whisper-cli 1.8.4 syntax: --no-context became -mc 0, --temperature became -tp 0.
3. The hallucination filter got extended (the blocklist above) because the new backend tripped on silence in slightly different shapes.

The runtime trade is real. mlx-whisper ran at ~0.1–0.3× RTF on M4. whisper.cpp runs at ~0.5× RTF on the same hardware. End-to-end latency went from 2.5–3 s to roughly 3.5–4 s. Word-error rate on Dutch business speech stayed in line with what I was already getting because the model weights are identical between backends. RAM stayed around 1.6 GB. For a live coaching tool where the human is also reading the prospect's face, that latency increase is acceptable. For a tool that has to ship without crashing, it is the only choice.

The honest takeaway: do not trust a single ML runtime in production. Have a Plan B before you need one. The runtime cost of maintaining the adapter for fifteen days was effectively zero, one extra CI job. The cost of not having it would have been a week of debugging MLX internals on a deadline.

Day 29: from emergency swap to structural fix (PR-102)

Whisper.cpp kept v1.0 alive. It did not kill the bug. The bug was that I had two WhisperDirect engines instantiated in the same process, each calling into mlx-whisper, each pushing work onto a default Metal command queue with no coordination between them. Switching the inference engine sidesteps the race; it does not redesign the concurrency.

Five days after launch I shipped PR-102 (18e277c). The new architecture replaces the dual-engine layout with a single InferenceWorker fed by a SharedInferenceQueue. Mic and system audio still arrive on physically separate streams, the speaker: "self" / speaker: "prospect" invariant is preserved end to end, but they now serialize through one inference path. Two priority levels:

• system (prospect speech) → HIGH priority. Pain detection runs on what the prospect just said; the human operator needs that signal in real time.
• mic (own speech) → LOW priority. My own words are coaching context, not a real-time decision input. They can wait a chunk if the queue is full.

A new audio_bufferer.py accumulates raw audio per stream until either 6.0 s of audio or a 1.5 s silence gap; the bufferer pushes a job onto the shared queue; the single InferenceWorker pulls jobs in priority order and dispatches them through the backend. There is now exactly one MLX caller in the process. The Metal command-buffer race cannot fire because there is no second caller to race with.

A second mode dropped out of the same refactor. TRANSCRIBE_SELF_LIVE=false flips mic transcription off during the call entirely, then runs a batch transcribe over data/sessions/<id>/mic.wav after the call ends. The post-call report still shows the full bilateral transcript (sorted by start_ms), but during the call the inference budget is fully spent on the prospect. For sales calls where I already know what I said, this is the right trade.

The honest second takeaway: an escape route is not the destination. On day 24 the swap was correct, the situation demanded it and I had the material ready. On day 29 the redesign was correct, the situation was calm and I had time to think. Both are real engineering moves. They just answer different questions. Skipping either step leaves you fragile.

26 new tests landed with PR-102; the suite has since grown to 247. Default backend is back to mlx-whisper because it is faster, whisper.cpp remains a one-env-var fallback. Total cost of the structural fix: about a day of focused work, mostly in inference_queue.py, inference_worker.py, and audio_bufferer.py. The bug is gone, the architecture is simpler, and the system runs better than it did before the crash.

📖 Read also: April Build Log: 10,000+ Dispatches: metrics and failure modes from the same month this was built

Open core economics

I did not enjoy picking a license. I read the OSI list twice. The decision tree was:

• MIT, too weak. A competitor could close-source-extend, ship it as their hosted SaaS, and not contribute back. I have watched this happen to other developer tools too many times.
• BSL 1.1, the right shape but not OSI-approved. HackerNews would notice. The point of open-sourcing this is partly the credibility signal; a non-OSI license gives that up.
• AGPL-3.0, the strong-copyleft answer to MIT-extraction. Anyone running a modified version as a network service has to release their modifications. Companies with proprietary-derivative-work policies cannot use it.
• AGPL-3.0 + commercial dual-license, what shipped. The OSS is a real OSS badge. The commercial license is the escape valve for SaaS vendors, regulated industries, and proprietary-derivative concerns. Pricing per organization, contact me directly.

The OSS/Pro split follows four rules:

1. Individual = free, team = paid.
2. Static/default = free, dynamic/AI-generated = paid.
3. Self-hosted = free or Pro, managed/SaaS = Pro or Enterprise.
4. Integrations are the upsell ladder.

What is in OSS, everything I would actually want to use:

• All transcription, all detection (12 Dutch pain categories + 5 objection categories), talk-time, phase detection, slide injection (pre-loaded), suggestions, summary, post-call report.
• The fine-tune pipeline. scripts/finetune/{train_whisper.py, prepare_dataset.py, evaluate.py} are in OSS as of e35f45b.
• Single-user dashboard, local SQLite.

What is Pro (€29/mo single, €49/user/mo team, custom enterprise):

• The pre-trained NL sales Whisper model (the artifact, not the build tools).
• HubSpot/Pipedrive/Salesforce/Teamleader CRM sync.
• Dynamic LLM-generated slides.
• Multi-user team dashboard.
• SSO + audit logs (enterprise).

I give away the gun. I sell the bullets. Anyone can fine-tune Whisper with the OSS pipeline. I have spent a year collecting Dutch sales recordings and PR-97 shipped session audio recording in OSS, every operator running the OSS now collects a corpus of their own data to fine-tune on. What they do not have is my year of training data. The Pro model is the artifact, not the recipe.

What I would do differently

Started with too many backends. AudioTee, BlackHole, and Loopback simultaneously was the wrong sequencing. I should have shipped with BlackHole only, most permissive license, easiest to install, lowest macOS-version risk, and added AudioTee on demand. I lost about a week to integration debugging that could have happened later. The same critique applies to the three transcription backends, but at least there the third one saved my release.

Default-disabled features that should have been default-on. The PR-91b commit message reads: "Flip ENABLE_OBJECTION_DETECTION, ENABLE_SUGGESTIONS, ENABLE_SUMMARY, DYNAMIC_SLIDES to default true so first-time users see AI activity out of the box." I shipped these flags off because each one had a small risk of misbehavior on first run. The cost of that caution was that the first three real demos showed a transcript pane and nothing else. First-time-user experience is a separate testing dimension from "does the feature work" and I did not have a test for it.

Underestimated MLX thread-safety risk earlier. The escape route was the right design. The trigger for building it should have been "I read the MLX 0.31.0 release notes and they mention thread-safety work," not "I have a vague feeling MLX could surprise me." By the time I read those notes carefully, my system was already hitting the bug.

What is next

Concrete ship-list:

• Shipped this week (post-launch). PR-102 shared InferenceWorker + priority queue; PR-103 google-genai 1.72 migration to GenerateContentConfig (five Gemini call sites had been silently failing on TypeError); PR-104 dashboard layout with coaching-priority and chronological transcript ordering.
• v1.x near-term. faster-whisper (CTranslate2) backend behind the same Protocol for Linux/Windows support. Streaming partial outputs (Otter-style typewriter feel) once the dashboard learns to render interim results.
• Pro v2.x. Pre-trained NL sales Whisper model (training data being collected via PR-97 right now). CRM integrations. Optional cloud-LLM polish for post-call written reports.
• VNX Note. A €99 ESP32-S3 BLE meeting recorder that pairs with the OSS for offline meeting capture. Hardware companion in design.

I am building this in the open. PRs welcome. Honest feedback more welcome. The repo is github.com/Vinix24/sales-copilot, the license is AGPL-3.0, and the wait-list for Pro features is on the README.

The single most useful thing I have learned in 30 days: design the escape route fifteen days before you need it, then design the structural fix five days after you need it. If you want to understand how I apply this governance-first approach to client projects, see my work as an AI architect. The escape route gets you through the crisis with nothing dropped. The structural fix is what keeps the bug from coming back.