Inside OnCue: the architecture of a local-first, real-time sales copilot

A live sales copilot has to answer three questions in the time a prospect finishes a sentence. What did they just say? Was that an objection or a buying signal? And what should I say back? Each question has a different latency budget, and only one of them needs the cloud.

This is OnCue, my open-source live sales copilot. It runs local-first on a Mac, the code is on GitHub under AGPL-3.0, and it listens along on a real sales call to transcribe, detect, and coach. This post is the architecture and the measured numbers behind it. No hand-waving, and one asterisk I will get to.

Three jobs, three latency budgets

The whole design falls out of one observation. A live copilot does three things while the prospect is still talking, and each has its own budget.

Transcription and detection run fully local and fast. The generative reframe, the "what should I say right now" step, is the only piece a local LLM cannot serve at conversational latency today. That is the asterisk. The 100 percent local tier ships transcription, detection, and summarization on-device. Only the live reframe reaches out, and even that is your choice, not a default.

The live path can never block on a local LLM. So the embedding router and the pre-authored cards carry the instant path, and the LLM reads the conversation on its own cadence, off the critical path. Measured over the whole chain (audio, transcription, detection, model, render), the p95 sits at roughly 2.9 seconds and no measured segment went over 5 seconds.

Two tracks, not one classifier

Detection is not a single pipeline. Two tracks run side by side over the prospect's speech, and they answer different questions at different costs.

Track one, cards behind an embedding router. Prospects rarely say "that is too expensive". They say "we would have to look at the budget for that". The router matches on meaning rather than tokens, in roughly 5 milliseconds, against a library of canonical objections. A hit puts a pre-authored card on screen: a known objection mapped to its strongest rebuttal, zero inference, fully predictable output. Objections and buying signals are single-sourced here, so the curated rebuttal and the tier check are applied in exactly one place.

Track two, the window classifier. One instructor-structured LLM call reads the last five prospect chunks as a whole and returns pain points and doubt. It fires only when the window holds at least three chunks and at most once every five seconds (DETECTOR_MIN_CHUNKS, DETECTOR_DEBOUNCE_S), and only the prospect's speech ever enters it.

What keeps this affordable is not an escalation threshold. It is the window, the debounce, and the prospect-only filter. That also makes the privacy boundary explicit: the classifier is a model call, so whichever tier you picked for the call is the tier that reads those windows. Choose Ollama and nothing leaves the machine; choose a cloud provider and short prospect-only fragments do.

Both tracks land on the same screen, which is where the split becomes visible. The objection card on the right carries a curated rebuttal and fired in milliseconds. The pain point cards next to it came out of the window read, arrived a beat later, and are the ones that reference a matching case.

The design consequence I like: track one is a data problem, not a model problem. Every card and every canonical objection I add takes work off track two, which lowers both latency and cost. That is a much better scaling curve than swapping in a bigger model.

Why the window, and why two debounces

Track two is deliberately not a per-sentence classifier. A sliding window buffers the last N transcript chunks (DETECTOR_WINDOW_SIZE=5), because a thought frequently spans two or three fragments and a per-chunk classifier keeps cutting them in half. The system prompt is phase-aware and carries ten-plus negative examples, which is what stops it firing on backchanneling ("ja", "hmm", "precies").

There are two separate debounces, and conflating them is a good way to build a dashboard that behaves like a slot machine. DETECTOR_DEBOUNCE_S throttles how often the window is re-read at all. DEBOUNCE_SECONDS=45 is a per-category cooldown on what reaches the screen, so the same objection does not re-announce itself every time the window slides by one chunk.

The legacy semantic-router path still exists in the codebase, but it is no longer the default classification route.

Everything talks through a localhost hub

The backend is a locally running Python process with two browser frontends. Every component communicates over WebSocket on 127.0.0.1:8760. No component can reach another over the network without going through the hub. The hub binds to localhost only, so it is not reachable from other machines by default.

The channels are granular: /ws/transcript, /ws/talk-time, /ws/pain-points, /ws/objections, /ws/buying-signals, /ws/coaching, /ws/suggestions, /ws/summary, /ws/slide-control. Each module is independently runnable and testable.

One decision matters more than the rest here. Transcription runs through a single inference worker with a bounded priority queue, not two parallel engines. An earlier dual-engine design crashed with a Metal SIGABRT under concurrent GPU access on Apple Silicon. The single-worker rewrite gives sequential, stable GPU access. The prospect stream gets priority 0 (high), my own mic priority 1 (low). So the person I need to react to is always transcribed first.

Two streams instead of diarization

The audio layer hides every capture route behind one AudioStream protocol: start, stop, read, plus chunks_received and a device label. MicStream reads the microphone at 16 kHz mono and is tagged self. The prospect route is whatever the platform offers: a whole-system Core Audio process tap on macOS, a BlackHole virtual device as fallback, or a WASAPI loopback on Windows.

That split is the point. Because my voice and the prospect's voice arrive on two physically separate streams, speaker attribution is free. No diarization model, no clustering, no confusion when two people talk over each other. Every architectural problem you do not create is one you never have to tune.

It also makes the priority queue meaningful. Two labelled streams give the queue something to prioritise, which is why the prospect is always transcribed ahead of me. Silero VAD gates both streams on CPU in under 10 milliseconds a frame, so silence never reaches the transcription queue at all.

Diarization does ship, using pyannote v3, but in shadow mode and off by default. It is there for the multi-speaker case, not for the two-party call that dual-stream already solves.

A provider-agnostic LLM layer

Every model provider sits behind a Python Protocol, wrapped with instructor for structured output. Swapping a provider is a config change, not a rewrite. OnCue ships with six providers today: Groq, OpenAI, Gemini, Ollama, Vertex AI, and Azure OpenAI. The default is gemini-2.5-flash.

The most useful thing I measured was the reasoning budget of the reframe call. Same structured call, only the thinking budget changed.

128 tokens of thinking took the call from 11 seconds to instant plus a couple of seconds, with cleaner questions, on one knob. That is the difference between a copilot that interrupts your flow and one that keeps up.

Transcription has the same story. The heaviest fixed cost in the chain is transcription, not generation, so I benchmarked the backend on identical audio.

For generation I compared a fast model against a frontier one on the same structured path. Haiku 4.5 landed at p95 ~2.4 s at $1 in and $5 out per million tokens. GPT-4o was not measurably faster and cost roughly twice as much. Frontier is the reference, not the default. Fast and cheap wins the live path.

The numbers that shaped the model choices

I did not pick models on vibes. I measured them on the copilot's own signals.

For detection, my local free-tier model (gemma-3n-e4b) hit a 58 percent category match with zero failed calls. The cloud reference (gemini-2.5-flash) hit 87 percent in 1.4 seconds. Local is good enough for the fast route, and the cloud sharpens it where precision matters.

For transcription accuracy, the quantized large-v3-turbo-q8_0 (834 MB, roughly 55 percent of full size) scored 17.1 percent WER against 17.2 percent for the full turbo model on a clean call, streaming at p50 2.79 seconds. Half the footprint, same accuracy.

The embedding result was the one that changed how I think about this. I tested four multilingual embedders on 16 canonical objections and 18 held-out phrasings. The best model (multilingual-e5-large) got 13 of 18 first-hit, 72 percent. The old small model already in the stack (MiniLM, 384 dimensions, from 2020) got 12 of 18 and beat two larger models, losing by exactly one query. Every per-query embed ran under 15 milliseconds. The lesson: a fuller library of objections and rebuttals beats a better embedding model. Fill the memory, do not chase the model.

The bottleneck is hardware, and I need help there

The open problem, stated plainly: transcription is the heaviest fixed cost in the chain, and I cannot brute-force it on the machine I have.

Locally, whisper large-v3-turbo sits at p50 990 ms on 4 threads. I doubled it to 8 threads and got 986 ms. That is noise, not a speedup. The cost is not thread-bound, it is raw-compute-bound, so throwing more CPU at it does nothing.

Groq runs the same model on the same audio at 209 ms, roughly 5x faster, for $0.0005 a run. That gap is what a genuinely fluent live experience feels like. The catch is that it sends audio off the machine, which is the one thing this design exists to avoid. So it ships as opt-in, never as the default.

But it proves the ceiling is hardware, not the model. A sufficiently strong local GPU should be able to approach that 209 ms figure with the audio staying on-device, which would close the last real gap between the local tier and the cloud one.

I cannot benchmark that alone. I have one Mac, which is one data point. If you run a serious local GPU, a DGX-class box, or a Strix Halo, your numbers are worth more than mine here. The transcription harness is in the repo. Run it, open an issue with your hardware and your p50, and that turns a hunch into a spec. This is the contribution I most want right now.

License enforcement without a server in the hot path

The tiering has to work offline. So the license authority issues Ed25519-signed keys with an SCP- prefix, and verification happens fully on the client with no network call.

The 32-byte Ed25519 public key is baked into the build itself, in a generated _embedded_keys.py that the verifier resolves at startup, so there is no config file or environment variable that can swap it at runtime. An SCP- key is a 91-byte struct (version, tier, rotation epoch, issued-at, expires-at, a 128-bit pseudonymous license id, and a 64-byte signature), and the signature covers the first 27 bytes. The private key lives only on the issuing worker, so there is no way to mint a valid key from the client.

A single FeaturePolicy is the one entitlement source. It reads the key, verifies the signature and expiry offline, and holds the whole capability list in one place. Pro grants eight capability ids today; the four that matter most to a user are in-call slide injection, telephony capture, the central tamper-evident audit, and autostart. Missing, invalid, or expired keys degrade to the free tier. The only phone-home is a revocation check that posts nothing but the 128-bit license_id, at most once every seven days, and it fails open inside a grace window. The server never terminates the app over license status.

One note on status: the verifier, the entitlement policy, and the grace flow ship and are tested. The Cloudflare Worker that issues keys is deployed and live, and the pilot signup flow runs on it end to end: request, double opt-in, key issued, seat counted. The paid checkout path is built but has not been exercised in production yet, because the pilot keys are free.

Privacy is a choice you make per conversation

Audio, full transcripts, and any session recordings never leave the machine. whisper.cpp transcribes on-device, so unlike cloud-Whisper services, no LLM destination ever receives audio. What can optionally leave is short, PII-redacted transcript fragments of one to three sentences, and only the prospect's speech is classified, never my own.

Where those fragments go is a per-call decision across three tiers:

1. Fully local, with Ollama. Nothing leaves the device.
2. Your own cloud tenant. Azure OpenAI in your Azure subscription, Vertex AI in your GCP project, or AWS Bedrock in your account. Fragments stay inside your existing contract, DPA, and chosen EU region, with no new processor added.
3. A third-party API, processed under that provider's terms and region.

The same person can pick tier 1 for a sensitive call and tier 3 for an internal sales training. The own-tenant tier makes compliant processing within your tenant possible, but it does not make an organization compliant by itself. The controller stays responsible for provider, region, contracts, and lawful use.

The asterisks

Three things I will not pretend away. The live reframe cannot run locally at conversational latency today, so that one job leans on the cloud or a stronger machine on your LAN. The mlx-whisper backend is an experimental fallback with a known silent zero-segments bug, which is why whisper.cpp is the default. And the license control plane issues keys in production, but its paid checkout path has not carried a real transaction yet.

None of the four benchmark tracks needs the cloud to reproduce. That was the whole point, the same way I run my agentic-harness benchmarks in the open. You can run the shootout on your own machine.

Read also: Glass Box Governance for multi-agent AI: why I log receipts, not chat transcripts, in production AI systems.

Build on it

The code is on GitHub: github.com/Vinix24/OnCue, AGPL-3.0. The architecture doc and the reproducible benchmark harness (with an NDJSON audit log) are in the repo. Clone it, poke holes in the latency claims, or send a pull request (there is a CLA).

If you break one of the numbers above on your own hardware, open an issue with your setup. The transcription benchmark on fast local silicon is the single most useful contribution right now. What the tool does at the product level lives at vincentvandeth.nl/oncue.

This is the kind of production AI system I design and ship: architecture first, models second.