<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Realtime on Rason Yang</title><link>https://rasonyang.dev/tags/realtime/</link><description>Recent content in Realtime on Rason Yang</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><copyright>© 2026 Rason Yang</copyright><atom:link href="https://rasonyang.dev/tags/realtime/index.xml" rel="self" type="application/rss+xml"/><item><title>A Realtime Voice Bot on One Mac, and Why the Third Call Breaks It</title><link>https://rasonyang.dev/posts/realtime-voice-on-one-mac/</link><pubDate>Mon, 26 Oct 2026 15:03:35 +0800</pubDate><guid>https://rasonyang.dev/posts/realtime-voice-on-one-mac/</guid><description>&lt;p&gt;AICC answers calls with anything that speaks the OpenAI Realtime protocol.&#10;In &lt;a href="https://rasonyang.dev/posts/ai-native-call-center-in-four-projects/#the-model-cascade-realtime-gateway" &gt;the first post in this series&lt;/a&gt; that meant a hosted model, or&#10;Cascade Realtime Gateway, which runs ASR → LLM → TTS behind the protocol but&#10;still calls hosted providers for each stage. The question this post answers:&#10;can one Mac run the whole cascade itself, every model local, and still answer&#10;inside the deadlines AICC&amp;rsquo;s client enforces?&lt;/p&gt;&#10;&lt;p&gt;For one call, yes. On an M3 Max, from the end of the caller&amp;rsquo;s speech to the&#10;first audio sent, &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L21-L25" target="_blank" rel="noreferrer"&gt;P50 is 1.37–1.39 s and P95 1.52–1.54 s&lt;/a&gt;&#10;on turns the turn detector judged complete, and&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L190-L197" target="_blank" rel="noreferrer"&gt;every one of AICC&amp;rsquo;s windows had more than 2 s of margin&lt;/a&gt;. For&#10;three calls, not reliably. &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L172-L177" target="_blank" rel="noreferrer"&gt;The TTS synthesizes one stream at a time&lt;/a&gt;,&#10;and at three sessions one answer&amp;rsquo;s first audio came&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L204-L207" target="_blank" rel="noreferrer"&gt;3.39 s after &lt;code&gt;response.created&lt;/code&gt;&lt;/a&gt;, past AICC&amp;rsquo;s 3 s limit.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;What runs where&#10; &lt;div id="what-runs-where" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-runs-where" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;mlx-realtime &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L3-L7" target="_blank" rel="noreferrer"&gt;speaks a documented subset of the OpenAI Realtime API (GA) over&#10;WebSocket at &lt;code&gt;/v1/realtime&lt;/code&gt;&lt;/a&gt;, and the official &lt;code&gt;openai&lt;/code&gt; Python&#10;SDK connects by changing only &lt;code&gt;base_url&lt;/code&gt;. AICC&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L19-L22" target="_blank" rel="noreferrer"&gt;attaches through its existing &lt;code&gt;gateway&lt;/code&gt; profile, with no AICC change&lt;/a&gt;.&#10;Inside, it is &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L45-L66" target="_blank" rel="noreferrer"&gt;four processes&lt;/a&gt;:&lt;/p&gt;&#10;&lt;p&gt;&lt;figure&gt;&lt;img&#10; class="my-0 rounded-md"&#10; loading="lazy"&#10; decoding="async"&#10; fetchpriority="low"&#10; alt="Architecture: AICC, the openai SDK and the benchmark connect over WebSocket to a Python gateway that runs Silero VAD, Smart Turn, two echo cancellers and G.711 on the CPU; the gateway calls three MLX processes over HTTP on localhost: Qwen3-ASR-1.7B 8-bit (2.5 GB), Qwen3.6-35B-A3B 4-bit under mlx_lm.server (22.9 GB), and Qwen3-TTS-0.6B 8-bit with a 4-bit fallback (3.8 GB), which synthesizes one stream at a time"&#10; src="https://rasonyang.dev/posts/realtime-voice-on-one-mac/architecture.svg"&#10; &gt;&lt;/figure&gt;&#10;&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;The gateway&lt;/strong&gt;, Python 3.12 and asyncio, owns the protocol, sessions,&#10;echo cancellation, Silero VAD and Smart Turn, segmentation and barge-in.&#10;Its environment &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L68-L73" target="_blank" rel="noreferrer"&gt;has no &lt;code&gt;mlx&lt;/code&gt; packages installed&lt;/a&gt;, so it&#10;cannot load an MLX model by construction; Silero and Smart Turn run on&#10;the CPU under onnxruntime.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Three MLX backends&lt;/strong&gt; on localhost, each its own uv project:&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L620-L627" target="_blank" rel="noreferrer"&gt;Qwen3-ASR-1.7B 8-bit, Qwen3.6-35B-A3B 4-bit under &lt;code&gt;mlx_lm.server&lt;/code&gt;, and&#10;Qwen3-TTS-0.6B CustomVoice 8-bit with its 4-bit build as the fallback&lt;/a&gt;,&#10;every one pinned by revision.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Resident after warmup on the real stack:&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L628-L629" target="_blank" rel="noreferrer"&gt;LLM 22.9 GB with 64 cached prompts, TTS 3.8 GB, ASR 2.5 GB, gateway 0.25 GB&lt;/a&gt;,&#10;about 29.5 GB in all, more than the README&amp;rsquo;s &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L39-L48" target="_blank" rel="noreferrer"&gt;&amp;ldquo;about 25 GB&amp;rdquo;&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;Cascade Realtime Gateway is&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L26-L27" target="_blank" rel="noreferrer"&gt;&amp;ldquo;a design reference only, not a client of this service&amp;rdquo;&lt;/a&gt;:&#10;mlx-realtime reuses its protocol decisions and lessons where they are&#10;GA-faithful. The service is&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L39-L48" target="_blank" rel="noreferrer"&gt;sized for development and testing&lt;/a&gt;: at most four sessions,&#10;because a human call AICC transcribes&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L80-L90" target="_blank" rel="noreferrer"&gt;opens two transcription sessions, one per leg&lt;/a&gt;.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The deadlines&#10; &lt;div id="the-deadlines" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-deadlines" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;AICC&amp;rsquo;s client &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L63-L69" target="_blank" rel="noreferrer"&gt;enforces four deadlines&lt;/a&gt;:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;From&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;To&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Limit&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;response.create&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;response.created&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;3 s&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;response.created&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;first audio delta&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;3 s&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;one audio delta&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;the next&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;under 2 s&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;response.created&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;response.done&lt;/code&gt;, for a tool-only reply&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;3 s&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;What happens when one is missed:&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L149" target="_blank" rel="noreferrer"&gt;A missed deadline silently abandons the turn.&lt;/a&gt;&amp;rdquo; No error,&#10;no retry. AICC also &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L156" target="_blank" rel="noreferrer"&gt;streams caller audio continuously, including while the&#10;bot speaks, and has no echo cancellation&lt;/a&gt;. Both facts shape&#10;everything below.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Where 1.4 seconds goes&#10; &lt;div id="where-14-seconds-goes" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-14-seconds-goes" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;The benchmark &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L15-L23" target="_blank" rel="noreferrer"&gt;replays 126 public utterances&lt;/a&gt; (FLEURS Chinese and&#10;English, ASCEND code-switched conversation, 1.6 to 9.9 s each) in 20 ms&#10;frames at real time, with &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L29-L40" target="_blank" rel="noreferrer"&gt;AICC&amp;rsquo;s own session settings&lt;/a&gt;:&#10;&lt;code&gt;server_vad&lt;/code&gt; with 500 ms of silence. The machine was&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L110-L112" target="_blank" rel="noreferrer"&gt;an M3 Max with 128 GB on AC power, High Power mode, nothing else on the GPU&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;figure&gt;&lt;img&#10; class="my-0 rounded-md"&#10; loading="lazy"&#10; decoding="async"&#10; fetchpriority="low"&#10; alt="Stacked bar of the median time per stage for one call on the telephone channel, G.711 μ-law: end of speech detected 525 ms, of which 500 ms is AICC&amp;rsquo;s silence setting; Smart Turn 68 ms; ASR 220 ms; LLM first token 318 ms; first sentence 96 ms; TTS first chunk 76 ms; first audio sent 102 ms; about 1.4 s in all"&#10; src="https://rasonyang.dev/posts/realtime-voice-on-one-mac/stages.svg"&#10; &gt;&lt;/figure&gt;&#10;&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L130-L141" target="_blank" rel="noreferrer"&gt;Per stage, one session&lt;/a&gt;, on the telephone variant (8 kHz G.711&#10;μ-law in):&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;522–525 ms is the caller&amp;rsquo;s own silence.&lt;/strong&gt; The VAD&amp;rsquo;s end-of-speech&#10;decision is &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L59-L67" target="_blank" rel="noreferrer"&gt;&lt;code&gt;silence_duration_ms&lt;/code&gt; plus detection&lt;/a&gt;, and AICC&#10;asks for 500 ms. More than a third of the wait is a setting the client&#10;chose.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;The models take about 610 ms&lt;/strong&gt;: ASR 220 ms, LLM first token 318 ms,&#10;TTS first chunk 76 ms.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;The ~100 ms of &lt;code&gt;first_audio_sent&lt;/code&gt; is silence.&lt;/strong&gt; Qwen3-TTS&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L149-L150" target="_blank" rel="noreferrer"&gt;puts about 0.45 s of silence before every segment&lt;/a&gt;; the TTS&#10;synthesizes it and the trimmer drops it.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L143-L147" target="_blank" rel="noreferrer"&gt;The sum of the stage medians matches the end-to-end median&lt;/a&gt;. The&#10;P95 is another story. Over all turns it is 3.78 to 3.87 s, because&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L143-L147" target="_blank" rel="noreferrer"&gt;Smart Turn held 12–13 turns per run as incomplete&lt;/a&gt; until its 3 s&#10;ceiling. Ten of those are ASCEND excerpts that really do stop mid-sentence&#10;(&amp;ldquo;我认为现在的AI，比如说在。&amp;rdquo;). That is why the headline figures are&#10;quoted on turns that ended on a complete verdict, n = 113–114 per run.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Fitting the LLM into 3 seconds&#10; &lt;div id="fitting-the-llm-into-3-seconds" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#fitting-the-llm-into-3-seconds" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;The median turn is the easy case. The hard ones are AICC&amp;rsquo;s phase changes:&#10;the flow swaps the instructions, then the caller asks to be transferred, and&#10;the model has to emit a &lt;code&gt;transfer_to_agent&lt;/code&gt; call inside the tool-only&#10;window.&lt;/p&gt;&#10;&lt;p&gt;Qwen3.6 is &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L604-L609" target="_blank" rel="noreferrer"&gt;hybrid, Gated DeltaNet plus attention, so its cache cannot be&#10;trimmed back&lt;/a&gt;. &lt;code&gt;mlx_lm.server&lt;/code&gt; can reuse only whole prefixes it&#10;saved, so each of AICC&amp;rsquo;s phase changes re-prefilled about 1650 tokens.&#10;The LLM backend now saves checkpoints where the tool definitions end&#10;and where the system message stops matching recent ones, so a phase change&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L604-L609" target="_blank" rel="noreferrer"&gt;prefills ~190 tokens, not ~1650&lt;/a&gt;. Together with warming the&#10;prefix before the caller&amp;rsquo;s next turn,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1503-L1507" target="_blank" rel="noreferrer"&gt;median prefill fell from 1625 tokens to 47 on a cold call, and from 1711 to 123 on AICC&amp;rsquo;s turn after a tool result&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1476-L1483" target="_blank" rel="noreferrer"&gt;Measured on the LLM alone&lt;/a&gt; (mock ASR and TTS, one call at a time,&#10;20 runs per case, so p95 is the maximum), from &lt;code&gt;response.created&lt;/code&gt; to the&#10;first sentence boundary or, for a tool-only reply, to &lt;code&gt;response.done&lt;/code&gt;:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Case&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Before: no warming, stock cache (p50 / p95 s)&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Checkpoints + warming&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;A cold call, new instructions&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1509-L1521" target="_blank" rel="noreferrer"&gt;1.32 / 1.38&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;0.41 / 0.49&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Transfer after an instructions change (tool-only)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1509-L1521" target="_blank" rel="noreferrer"&gt;2.89 / 3.46&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;1.50 / 1.88&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&amp;rsquo;s reply to a tool result (spoken)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1509-L1521" target="_blank" rel="noreferrer"&gt;1.48 / 1.80&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;0.52 / 0.64&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&amp;rsquo;s next turn is a transfer (tool-only)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1509-L1521" target="_blank" rel="noreferrer"&gt;3.02 / 4.29&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;1.53 / 2.08&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;Without them, transfers missed AICC&amp;rsquo;s 3 s window at p95 even before the ASR&#10;was counted. With both changes,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1569-L1582" target="_blank" rel="noreferrer"&gt;every case is inside it at p95, the tightest at 2.17–2.58 s with ASR&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;The other LLM problem was not latency. In M0,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L742-L745" target="_blank" rel="noreferrer"&gt;11% of realistic transfer turns ended with &lt;code&gt;finish_reason: tool_calls&lt;/code&gt; and no parsed call&lt;/a&gt;:&#10;Qwen3.6 writes &lt;code&gt;&amp;lt;transfer_to_agent&amp;gt;&lt;/code&gt; instead of &lt;code&gt;&amp;lt;function=transfer_to_agent&amp;gt;&lt;/code&gt;.&#10;The LLM backend now&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L600-L603" target="_blank" rel="noreferrer"&gt;rewrites that form for tools the request declared, and never for undeclared names&lt;/a&gt;.&#10;Live, &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1560-L1563" target="_blank" rel="noreferrer"&gt;60 of 60 transfers were delivered, 4 of them recovered that way&lt;/a&gt;.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The third call&#10; &lt;div id="the-third-call" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-third-call" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;The TTS server &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L632-L633" target="_blank" rel="noreferrer"&gt;serves streams strictly one at a time&lt;/a&gt;. In M0,&#10;with two concurrent requests the second waited 2.9 s for its first chunk;&#10;with four, the last waited 8.4 s. The LLM batches and the ASR takes four at&#10;once, so &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L80-L90" target="_blank" rel="noreferrer"&gt;the TTS queue is the capacity limit of bot calls&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;The gateway shares it as well as it can. The queue is&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L416" target="_blank" rel="noreferrer"&gt;earliest-deadline first&lt;/a&gt;: the session that will run out of audio&#10;soonest goes next. An &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L1074-L1080" target="_blank" rel="noreferrer"&gt;audio reserve&lt;/a&gt; holds back up to 1.5 s&#10;of synthesized audio and releases it a delta at a time, to keep AICC&amp;rsquo;s 2 s&#10;gap watchdog fed. That was not enough. In the two-session&#10;run &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L198-L203" target="_blank" rel="noreferrer"&gt;one answer&amp;rsquo;s second segment waited 2.3 s behind another session&amp;rsquo;s long&#10;segment, and the wire went quiet for 2092 ms&lt;/a&gt;: AICC would have&#10;abandoned the turn. Now, when the reserve runs dry and more audio is&#10;coming, the gateway &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/DECISIONS.md#L44" target="_blank" rel="noreferrer"&gt;sends 20 ms of silence every second&lt;/a&gt;. In&#10;the three-session soak, 23 responses were bridged that way and no gap&#10;exceeded 1.0 s.&lt;/p&gt;&#10;&lt;p&gt;Then the first-audio window, the one that does not bend:&lt;/p&gt;&#10;&lt;p&gt;&lt;figure&gt;&lt;img&#10; class="my-0 rounded-md"&#10; loading="lazy"&#10; decoding="async"&#10; fetchpriority="low"&#10; alt="Horizontal bars of AICC&amp;rsquo;s first-audio window, response.created to the first audio delta at the client, for one, two and three sessions, against AICC&amp;rsquo;s 3 s limit: P95 0.94–0.97 s and maximum 1.1 s with one session; P95 1.27–1.96 s and maximum 2.65 s with two; P95 1.94–2.23 s and maximum 3.39 s with three, past the limit. Beside each row, the TTS first chunk P95 (79–80 ms, 0.6–1.0 s, 1.1–1.4 s) and backend_busy failures (0, 1, 7)"&#10; src="https://rasonyang.dev/posts/realtime-voice-on-one-mac/sessions.svg"&#10; &gt;&lt;/figure&gt;&#10;&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L158-L170" target="_blank" rel="noreferrer"&gt;Across three runs per configuration&lt;/a&gt;, the medians barely move.&#10;The tails are the TTS queue:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;TTS first chunk, P95&lt;/strong&gt;: 79–80 ms at one session, 645–1036 ms at two,&#10;1135–1420 ms at three. A segment of N seconds of speech&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L172-L177" target="_blank" rel="noreferrer"&gt;holds the TTS for about N/4 s&lt;/a&gt;, and the second segment of a&#10;one-sentence answer is often 6–9 s of speech.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;End to end, complete verdicts, P95&lt;/strong&gt;: 1.52–1.54 s at one,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L21-L25" target="_blank" rel="noreferrer"&gt;2.53–2.78 s at three&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Failures&lt;/strong&gt;: &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L179-L183" target="_blank" rel="noreferrer"&gt;7 responses at three sessions failed with &lt;code&gt;backend_busy&lt;/code&gt;, 1.8%&lt;/a&gt;:&#10;their segment waited in the queue more than 2 s past its deadline.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;AICC&amp;rsquo;s window&lt;/strong&gt;: &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L204-L207" target="_blank" rel="noreferrer"&gt;one first audio at 3.39 s&lt;/a&gt;, of which 1.92 s&#10;was the TTS queue, and one failed response at 3.22 s. AICC would have&#10;abandoned both.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The bench is also busier than a call:&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L154-L156" target="_blank" rel="noreferrer"&gt;each session talks without a pause&lt;/a&gt;. That makes the three-call&#10;numbers pessimistic, not wrong.&lt;/p&gt;&#10;&lt;p&gt;The shared GPU makes it worse, and merging processes would not help.&#10;While the LLM decodes, the TTS&amp;rsquo;s real-time factor rises&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L264-L275" target="_blank" rel="noreferrer"&gt;from 0.25 to 0.33–0.38&lt;/a&gt;. While the LLM prefilled ~16k&#10;tokens it reached 2.08, half real time, and any session speaking then would&#10;starve. AICC&amp;rsquo;s prompts are ~1.6k tokens, a tenth of that. And merging&#10;the three model processes into one&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/DECISIONS.md#L42" target="_blank" rel="noreferrer"&gt;would fix nothing&lt;/a&gt;: &amp;ldquo;the capacity limit is the TTS stream itself&amp;rdquo;.&lt;/p&gt;&#10;&lt;p&gt;What would help is&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/DECISIONS.md#L47" target="_blank" rel="noreferrer"&gt;more TTS capacity or shorter waits&lt;/a&gt;: a second TTS process (a code&#10;change, since the provider takes one &lt;code&gt;base_url&lt;/code&gt;), mlx-audio&amp;rsquo;s batched&#10;generation, capping continuation segments so none holds the TTS for more&#10;than about a second, or a longer queue timeout now that the keepalive feeds&#10;the watchdog. None is done yet.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The service hears itself&#10; &lt;div id="the-service-hears-itself" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-service-hears-itself" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;Echo on these calls is mlx-realtime&amp;rsquo;s problem.&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L822-L829" target="_blank" rel="noreferrer"&gt;Neither AICC nor FreeSWITCH cancels it&lt;/a&gt;, and the server cancels&#10;its own response when it hears the caller, so it &amp;ldquo;would therefore interrupt&#10;itself on its own echo, and AICC&amp;rsquo;s 800 ms barge guard cannot prevent that&amp;rdquo;.&#10;On simulated telephone echo with no canceller,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L514-L515" target="_blank" rel="noreferrer"&gt;Silero fired on 75% of echo-only windows: 1005 false &lt;code&gt;speech_started&lt;/code&gt; in 96 calls&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;One canceller could not serve both consumers. On simulated G.168 echo paths:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;WebRTC AEC3&lt;/strong&gt; keeps the VAD quiet, but its suppressor removes quiet&#10;caller speech. Offline, in continuous double talk at −15 dB,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L704-L717" target="_blank" rel="noreferrer"&gt;it cost the English ASR +28.4 points of WER&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;SpeexDSP&amp;rsquo;s linear canceller&lt;/strong&gt; keeps the caller intelligible, but&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L704-L717" target="_blank" rel="noreferrer"&gt;its residual triggers the VAD&lt;/a&gt;: &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/SPIKE.md#L517-L526" target="_blank" rel="noreferrer"&gt;0 of 96 calls&lt;/a&gt;&#10;kept the VAD quiet on echo alone.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;So each session &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L831-L836" target="_blank" rel="noreferrer"&gt;runs both&lt;/a&gt;: AEC3 in front of Silero and Smart&#10;Turn, Speex with its preprocessor in front of the ASR. The reference for&#10;both is the bot&amp;rsquo;s own output,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L838-L849" target="_blank" rel="noreferrer"&gt;placed on a model of AICC&amp;rsquo;s playout&lt;/a&gt;. Through the gateway&amp;rsquo;s own&#10;path, still on simulated echo,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L1797-L1816" target="_blank" rel="noreferrer"&gt;the dual path kept double-talk error within its limits in all 8 cells&lt;/a&gt;:&#10;the worst was continuous double talk at −15 dB, +1.8 points in Chinese and&#10;+1.6 in English, against +13.7 and +33.2 for AEC3 alone.&lt;/p&gt;&#10;&lt;p&gt;A line without echo is its own failure. AEC3&amp;rsquo;s suppressor then takes a&#10;caller who talks over the bot for echo, and&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L858-L866" target="_blank" rel="noreferrer"&gt;silenced them by up to 44 dB, so barge-in came 2.5–3.4 s late or not at all&lt;/a&gt;.&#10;AEC3 is now bypassed after 1.5 s of playing with no echo seen. The first&#10;version of that bypass let chance correlation peaks in the caller&amp;rsquo;s own&#10;speech turn it back on; requiring three agreeing windows cut&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/DECISIONS.md#L45" target="_blank" rel="noreferrer"&gt;false resumes from 29 to 2, and the SDK barge-in on the real stack from 1503 ms to 539 ms&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;I have since run real calls through AICC and FreeSWITCH: the bot did not&#10;cut itself off on its own echo, and the ASR kept up. That is an&#10;observation, not a measurement.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;What I am not claiming&#10; &lt;div id="what-i-am-not-claiming" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-am-not-claiming" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;Live echo numbers.&lt;/strong&gt; Every echo figure above is on&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L869-L870" target="_blank" rel="noreferrer"&gt;simulated G.168 echo paths&lt;/a&gt;. The real calls were observed,&#10;not measured: there is no echo path, ERL, delay or double-talk number&#10;from a real line, and&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/PROGRESS.md#L2303" target="_blank" rel="noreferrer"&gt;the repository still lists that measurement as deferred&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Recorded phone calls.&lt;/strong&gt; The latency corpus is public read speech and&#10;conversation excerpts &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L15-L23" target="_blank" rel="noreferrer"&gt;sent through a telephone codec&lt;/a&gt;, and&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L29-L40" target="_blank" rel="noreferrer"&gt;the bench plays nothing back, so it has no echo&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;One code version.&lt;/strong&gt; &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L117-L121" target="_blank" rel="noreferrer"&gt;The one-, two- and three-session runs ran on&#10;&lt;code&gt;bb75826&lt;/code&gt; plus a working tree that gained fixes between runs&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Statistics.&lt;/strong&gt; One machine, one run per audio variant, three variants&#10;per session count, no confidence intervals on latency.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Approved targets.&lt;/strong&gt; The per-stage budget and the latency targets are&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/DECISIONS.md#L41" target="_blank" rel="noreferrer"&gt;proposed, not approved&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Capacity.&lt;/strong&gt; One call meets AICC&amp;rsquo;s deadlines;&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md#L329-L334" target="_blank" rel="noreferrer"&gt;three do not reliably&lt;/a&gt;. Two calls sit in between: one&#10;&lt;code&gt;backend_busy&lt;/code&gt; and the one 2 s gap that led to the keepalive silence.&#10;Two calls have not been rerun since.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;A comparison with Cascade or a hosted model.&lt;/strong&gt; Different datasets,&#10;different networks, different hardware.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Production.&lt;/strong&gt; The service is&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L39-L48" target="_blank" rel="noreferrer"&gt;sized for development and testing&lt;/a&gt;. WebRTC and SIP&#10;transports, Docker, multiple machines and persistence are&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md#L40-L41" target="_blank" rel="noreferrer"&gt;out of scope&lt;/a&gt;.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Run it&#10; &lt;div id="run-it" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#run-it" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;On an Apple Silicon Mac with roughly 30 GB of memory to spare,&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L50-L59" target="_blank" rel="noreferrer"&gt;from the README&lt;/a&gt;:&lt;/p&gt;&#10;&lt;div class="highlight-wrapper"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-sh" data-lang="sh"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;make models-mlx &lt;span class="c1"&gt;# once: the pinned MLX models, ~25 GB of downloads&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;make dev &lt;span class="c1"&gt;# the backends and the gateway, on 127.0.0.1:8000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;make bench &lt;span class="c1"&gt;# the latency and ASR report against the running gateway&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;p&gt;Then point any Realtime client at it, &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L74-L94" target="_blank" rel="noreferrer"&gt;as the README&amp;rsquo;s SDK example does&lt;/a&gt;:&lt;/p&gt;&#10;&lt;div class="highlight-wrapper"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AsyncOpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;http://127.0.0.1:8000/v1&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;local&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/README.md#L133" target="_blank" rel="noreferrer"&gt;&lt;code&gt;make bench&lt;/code&gt;&lt;/a&gt; writes the report whose numbers are above;&#10;&lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md" target="_blank" rel="noreferrer"&gt;LATENCY.md&lt;/a&gt; explains how each one is produced.&lt;/p&gt;&#10;&lt;p&gt;Code: &lt;a href="https://github.com/rasonyang/mlx-realtime" target="_blank" rel="noreferrer"&gt;mlx-realtime&lt;/a&gt; · &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/LATENCY.md" target="_blank" rel="noreferrer"&gt;docs/LATENCY.md&lt;/a&gt; · &lt;a href="https://github.com/rasonyang/mlx-realtime/blob/8234db193cc3e007c0408f29c4b1e1e508f98772/docs/ARCHITECTURE.md" target="_blank" rel="noreferrer"&gt;docs/ARCHITECTURE.md&lt;/a&gt;&lt;/p&gt;&#10;</description><media:content xmlns:media="http://search.yahoo.com/mrss/" url="https://rasonyang.dev/posts/realtime-voice-on-one-mac/featured.jpg"/></item><item><title>An AI-Native Call Center in Four Open-Source Projects</title><link>https://rasonyang.dev/posts/ai-native-call-center-in-four-projects/</link><pubDate>Mon, 12 Oct 2026 10:54:49 +0800</pubDate><guid>https://rasonyang.dev/posts/ai-native-call-center-in-four-projects/</guid><description>&lt;p&gt;Call a support line and the call usually goes like this. A recording reads&#10;you a menu. You press 3, then 1, then 0 because nothing fits. You wait in a&#10;queue. An agent picks up and asks for the order number you already keyed&#10;into the menu.&lt;/p&gt;&#10;&lt;p&gt;Behind that call there is more pain, just not the caller&amp;rsquo;s:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;The agent&amp;rsquo;s desktop softphone has to be installed, configured and kept&#10;registered. When a laptop sleeps, the agent is silently unreachable.&lt;/li&gt;&#10;&lt;li&gt;The PBX is either on the public internet, where it is scanned all night,&#10;or behind a session border controller that costs real money.&lt;/li&gt;&#10;&lt;li&gt;If a voice bot has been bolted onto the front, it is wired to one vendor&amp;rsquo;s&#10;API, and it hands the caller to a person with none of what was said.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Each of those pains sits on a different boundary of the call path: the&#10;public edge, the switch, the model, and the agent&amp;rsquo;s browser. I built one&#10;open-source project per boundary. This post is the map; later posts take&#10;one problem at a time.&lt;/p&gt;&#10;&lt;p&gt;&lt;figure&gt;&lt;img&#10; class="my-0 rounded-md"&#10; loading="lazy"&#10; decoding="async"&#10; fetchpriority="low"&#10; alt="Architecture: callers and agent browsers reach FreeSBC; FreeSWITCH and AICC sit on the private network; AICC talks the Realtime protocol to a hosted model or to Cascade Realtime Gateway"&#10; src="https://rasonyang.dev/posts/ai-native-call-center-in-four-projects/stack.svg"&#10; &gt;&lt;/figure&gt;&#10;&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;One call, end to end&#10; &lt;div id="one-call-end-to-end" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#one-call-end-to-end" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;A caller dials in. The carrier&amp;rsquo;s SIP reaches &lt;strong&gt;FreeSBC&lt;/strong&gt;, the only host&#10;with a public address, and FreeSBC relays it to FreeSWITCH on a private&#10;network.&lt;/li&gt;&#10;&lt;li&gt;FreeSWITCH hands the call to &lt;strong&gt;AICC&lt;/strong&gt;, which answers it with a realtime&#10;voice model. AICC speaks the OpenAI Realtime protocol, either to a hosted&#10;model (OpenAI, Qwen, Doubao, Gemini) or to &lt;strong&gt;Cascade Realtime Gateway&lt;/strong&gt;,&#10;which runs ASR → LLM → TTS behind the same protocol.&lt;/li&gt;&#10;&lt;li&gt;The model talks; a flow decides the phase and when a person is needed.&#10;When one is, AICC moves the caller into a &lt;code&gt;mod_callcenter&lt;/code&gt; queue together&#10;with a summary of the conversation so far.&lt;/li&gt;&#10;&lt;li&gt;An agent&amp;rsquo;s phone rings. The phone is &lt;strong&gt;Web SIP Phone&lt;/strong&gt;, a Chrome&#10;extension whose audio reaches FreeSWITCH through FreeSBC over WSS and&#10;DTLS-SRTP. The agent answers in the AICC web app; the extension has no&#10;answer button.&lt;/li&gt;&#10;&lt;li&gt;When the call ends, the bot phase, the queue wait and the agent&amp;rsquo;s part&#10;are one record.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The edge: FreeSBC&#10; &lt;div id="the-edge-freesbc" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-edge-freesbc" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;&lt;strong&gt;The pain.&lt;/strong&gt; A SIP server with a public address gets found by scanners&#10;and brute-forced, around the clock. Commercial SBCs fix that for a price. The&#10;open-source route usually means assembling a SIP proxy, a separate media&#10;relay and their configuration, which is a lot to own for one switch.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The project.&lt;/strong&gt; FreeSBC&amp;rsquo;s pitch is &amp;ldquo;the Caddy experience:&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L3" target="_blank" rel="noreferrer"&gt;one binary, one YAML file, &lt;code&gt;./freesbc run&lt;/code&gt;&lt;/a&gt;&amp;rdquo;. It is the only&#10;element with a public address; FreeSWITCH or Asterisk&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L8" target="_blank" rel="noreferrer"&gt;stays on a private LAN and only ever talks to FreeSBC&lt;/a&gt;. It is a&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/design.md#L14-L20" target="_blank" rel="noreferrer"&gt;stateful SIP proxy, not a B2BUA&lt;/a&gt;, and it terminates ICE-Lite,&#10;DTLS-SRTP and RTCP-mux for browsers so the switch&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L115" target="_blank" rel="noreferrer"&gt;only sees plain RTP&lt;/a&gt;. The front door has&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L106" target="_blank" rel="noreferrer"&gt;per-IP rate limits and scanner fingerprinting&lt;/a&gt;, and an INVITE&#10;is &lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/edge.md#L47" target="_blank" rel="noreferrer"&gt;admitted only from the exact transport address of a live registration,&#10;or from a carrier&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The boundary.&lt;/strong&gt; A feature belongs in FreeSBC only if&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L144-L156" target="_blank" rel="noreferrer"&gt;only the edge can see or do it&lt;/a&gt;. CPS limits, toll-fraud rules&#10;and CDRs stay on the switch, which already does them well. That rule had a&#10;cost: the whole trunk/B2BUA plane was&#10;&lt;a href="https://github.com/rasonyang/freesbc/commit/c363f6e52afb5ca2ececcccac3a4879b792e8875" target="_blank" rel="noreferrer"&gt;deleted on 2026-10-02&lt;/a&gt; because it failed the test. The same&#10;discipline gives topology hiding by construction:&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L105" target="_blank" rel="noreferrer"&gt;every SDP body is built by FreeSBC, never copied from the other leg&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Not done yet.&lt;/strong&gt; Client legs are&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/edge.md#L218" target="_blank" rel="noreferrer"&gt;not fully topology-hidden&lt;/a&gt; (issues #166 and #167),&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/edge.md#L230" target="_blank" rel="noreferrer"&gt;IPv6 is untested&lt;/a&gt;, and there is&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/README.md#L173" target="_blank" rel="noreferrer"&gt;no TURN or full ICE&lt;/a&gt;.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The switch and the call: AICC&#10; &lt;div id="the-switch-and-the-call-aicc" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-switch-and-the-call-aicc" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;&lt;strong&gt;The pain.&lt;/strong&gt; DTMF menus. Callers repeating themselves after a transfer.&#10;One call split into an IVR log, a queue log and a recording. Dialplans&#10;spread across XML files.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The project.&lt;/strong&gt; AICC is&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L3" target="_blank" rel="noreferrer"&gt;an open-source call center where the AI is the default answer, not an add-on&lt;/a&gt;&amp;rdquo;.&#10;There is &lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/docs/phase1-decisions.md#L12" target="_blank" rel="noreferrer"&gt;no DTMF IVR&lt;/a&gt;: every external number answers with a bot&#10;flow. &amp;ldquo;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L45-L46" target="_blank" rel="noreferrer"&gt;The model owns the dialogue; the flow owns the phase.&lt;/a&gt;&amp;rdquo;&lt;/p&gt;&#10;&lt;p&gt;The handoff is a tool call. &lt;code&gt;transfer_to_agent&lt;/code&gt;&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/internal/flow/builtin.go#L213-L222" target="_blank" rel="noreferrer"&gt;requires a queue, a reason and a summary&lt;/a&gt;, plus any slots&#10;collected so far. The agent&amp;rsquo;s screen-pop arrives as a &lt;code&gt;PARTY_RINGING&lt;/code&gt; event&#10;over SSE, and the bot-phase transcript is&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L55-L59" target="_blank" rel="noreferrer"&gt;already there, because the call id is minted before any leg exists&lt;/a&gt;.&#10;The result is &lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L68-L70" target="_blank" rel="noreferrer"&gt;one CDR with one transcript and one recording&lt;/a&gt;,&#10;not three fragments.&lt;/p&gt;&#10;&lt;p&gt;On the switch side, FreeSWITCH&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L98-L102" target="_blank" rel="noreferrer"&gt;reads its directory and its queues from PostgreSQL through Lua&lt;/a&gt;,&#10;so adding an extension, a queue or a number is a database change. The&#10;whole thing is &lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L12-L13" target="_blank" rel="noreferrer"&gt;one Go binary plus PostgreSQL and FreeSWITCH&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The boundary.&lt;/strong&gt; A deployment runs&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/internal/provider/profile.go#L392-L394" target="_blank" rel="noreferrer"&gt;exactly one provider, chosen at startup&lt;/a&gt;, and a cascaded&#10;ASR + LLM + TTS pipeline&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/README.md#L157-L160" target="_blank" rel="noreferrer"&gt;never enters this repository&lt;/a&gt;. Anything that speaks the Realtime&#10;protocol can be plugged in instead, which is why the next project exists.&lt;/p&gt;&#10;&lt;p&gt;A taste of what this layer has to get right:&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/CLAUDE.md#L124" target="_blank" rel="noreferrer"&gt;Turn end ≠ playback end.&lt;/a&gt;&amp;rdquo; The model finishing a response is&#10;not the caller having heard it, so a transfer or a hangup&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/docs/design/02-ai-voice.md#L166" target="_blank" rel="noreferrer"&gt;waits for the audio to drain&lt;/a&gt;. That gets its own post.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The model: Cascade Realtime Gateway&#10; &lt;div id="the-model-cascade-realtime-gateway" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-model-cascade-realtime-gateway" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;&lt;strong&gt;The pain.&lt;/strong&gt; Speech-to-speech models are the easy path, but they tie you&#10;to one vendor&amp;rsquo;s voices, languages, regions and prices. Building your own&#10;pipeline looks easy too, and isn&amp;rsquo;t. In Cascade&amp;rsquo;s README:&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L25-L37" target="_blank" rel="noreferrer"&gt;Wiring ASR → LLM → TTS together takes an afternoon. The rest of a voice agent is what takes months&lt;/a&gt;&amp;rdquo;:&#10;turn detection, barge-in, context after an interrupt, streaming, races, and&#10;a stable client protocol.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The project.&lt;/strong&gt; Cascade&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L3-L4" target="_blank" rel="noreferrer"&gt;speaks the OpenAI Realtime (GA) WebSocket protocol to clients and runs a cascaded ASR → LLM → TTS pipeline behind it&lt;/a&gt;.&#10;The backend is a &lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L44-L45" target="_blank" rel="noreferrer"&gt;named profile, changed at runtime through an Admin API&lt;/a&gt;;&#10;new calls pick up the change and calls in progress don&amp;rsquo;t notice. Providers&#10;today are &lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L156-L161" target="_blank" rel="noreferrer"&gt;Deepgram, OpenAI and Qwen&lt;/a&gt;. Tools are declared by&#10;the client and executed by the client, and a tool call&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/internal/session/pipeline.go#L89-L90" target="_blank" rel="noreferrer"&gt;is never handed to TTS&lt;/a&gt;, so the caller never hears JSON.&lt;/p&gt;&#10;&lt;p&gt;One number shows why &amp;ldquo;context after an interrupt&amp;rdquo; is on that list. In one&#10;measured barge-in,&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/docs/decisions.md#L80" target="_blank" rel="noreferrer"&gt;10260 ms of audio had been generated and the caller had heard 2740 ms&lt;/a&gt;.&#10;Without trimming, 117 characters the caller never heard would have entered&#10;the next turn&amp;rsquo;s context.&lt;/p&gt;&#10;&lt;p&gt;On latency, the gateway itself&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/docs/qwen-latency.md#L154-L157" target="_blank" rel="noreferrer"&gt;adds 0.6 ms&lt;/a&gt; between the provider&amp;rsquo;s first audio frame and&#10;the client&amp;rsquo;s. End of user speech to first playable audio was&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/docs/qwen-latency.md#L28-L36" target="_blank" rel="noreferrer"&gt;p50 1223 ms&lt;/a&gt; on 2026-09-01, measured from a machine in China&#10;against a dedicated Qwen deployment over 20 turns; 391 ms of that is the&#10;500 ms VAD silence window doing its job.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The boundary.&lt;/strong&gt; Cascade &lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L107-L119" target="_blank" rel="noreferrer"&gt;does not speak SIP or G.711&lt;/a&gt;; audio is&#10;24 kHz PCM in and out. The telephone side is AICC&amp;rsquo;s job, which is why AICC&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/internal/provider/profile.go#L248-L254" target="_blank" rel="noreferrer"&gt;resamples 8 kHz to 24 kHz and back&lt;/a&gt; when it talks to the&#10;gateway. Cascade is also single-node:&#10;&lt;a href="https://github.com/rasonyang/cascade-realtime-gateway/blob/74127be2f982b11552d6115b07585815e484d313/README.md#L65" target="_blank" rel="noreferrer"&gt;one static API key, no clustering, no persistence&lt;/a&gt;.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;The agent&amp;rsquo;s ear: Web SIP Phone&#10; &lt;div id="the-agents-ear-web-sip-phone" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-agents-ear-web-sip-phone" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;&lt;strong&gt;The pain.&lt;/strong&gt; A desktop softphone is one more thing to install and support.&#10;A browser softphone tab has its own failures: refresh the page and the call&#10;drops; close the laptop lid and the agent stays &amp;ldquo;available&amp;rdquo; while being&#10;unreachable.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The project.&lt;/strong&gt; Web SIP Phone is a Chrome extension, and&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/README.md#L22-L26" target="_blank" rel="noreferrer"&gt;it is &lt;strong&gt;not&lt;/strong&gt; a softphone&lt;/a&gt;&amp;rdquo;: all call control comes from the SIP&#10;server, and the page&amp;rsquo;s own softphone bar owns every call-facing button. Its&#10;one job is to keep one agent&amp;rsquo;s WebRTC SIP leg registered and healthy:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;The SIP runtime lives in an offscreen document, so&#10;&lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/README.md#L284-L291" target="_blank" rel="noreferrer"&gt;refreshing the page mid-call does not interrupt audio or change the Call-ID&lt;/a&gt;.&#10;That was &lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/docs/FREESWITCH.md#L168" target="_blank" rel="noreferrer"&gt;checked live against FreeSWITCH on 2026-09-02&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;A &lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/src/offscreen/offscreen.ts#L35-L48" target="_blank" rel="noreferrer"&gt;10-second heartbeat&lt;/a&gt; notices a gap of more than 90 seconds,&#10;which means the machine slept, and forces a reconnect.&lt;/li&gt;&#10;&lt;li&gt;With AICC, &lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/docs/web-sip-phone-protocol.md#L5-L6" target="_blank" rel="noreferrer"&gt;nobody configures it&lt;/a&gt;. Signing in mints SIP&#10;credentials and the web app hands them to the extension, as&#10;&lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/README.md#L184-L186" target="_blank" rel="noreferrer"&gt;a digest hash, never a plaintext password&lt;/a&gt;.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Where each pain is handled&#10; &lt;div id="where-each-pain-is-handled" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-each-pain-is-handled" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Pain&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Handled by&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;PBX exposed to scanners and toll fraud&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FreeSBC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;One-way audio and dropped calls behind NAT&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FreeSBC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Browsers (WebRTC) talking to a plain-RTP switch&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FreeSBC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;DTMF menus&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Callers repeating themselves after a transfer&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;One call, three fragmented records&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Dialplan and queue config sprawl&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;AICC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Locked into one speech-to-speech vendor&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cascade Realtime Gateway&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Barge-in, turn detection and context after an interrupt&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cascade and AICC&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Installing and supporting desktop softphones&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Web SIP Phone&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Agents dropping off on refresh, sleep or Wi-Fi changes&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Web SIP Phone&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Why four repositories&#10; &lt;div id="why-four-repositories" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-four-repositories" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;Each project has a written rule about what it refuses to do. FreeSBC&#10;leaves anything the switch can do to the switch. AICC keeps the cascaded&#10;pipeline out. Web SIP Phone&amp;rsquo;s spec says&#10;&amp;ldquo;&lt;a href="https://github.com/rasonyang/web-sip-phone/blob/fdb1c2c2f856609762fa6894fb1f3bb407f19cd0/design.md#L19" target="_blank" rel="noreferrer"&gt;do not add a dial pad, contacts, call history, a CTI API&lt;/a&gt;&amp;rdquo;.&#10;Cascade does not do telephony.&lt;/p&gt;&#10;&lt;p&gt;Those refusals make the pieces usable on their own. FreeSBC can front any&#10;FreeSWITCH or Asterisk. Cascade can serve any Realtime client. AICC runs&#10;fine with a hosted model and no Cascade. The joints between them are&#10;ordinary protocols: SIP and RTP, the OpenAI Realtime protocol, and a small&#10;&lt;code&gt;postMessage&lt;/code&gt; protocol between a web page and an extension.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;What I am not claiming yet&#10; &lt;div id="what-i-am-not-claiming-yet" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-i-am-not-claiming-yet" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;p&gt;AICC&#10;&lt;a href="https://github.com/rasonyang/ai-native-callcenter/blob/86ec6e1d05be5dfdcd06d92a1b675a659a3fd76a/docs/load-tests.md#L7-L8" target="_blank" rel="noreferrer"&gt;publishes no performance figures&lt;/a&gt;, because its load tests&#10;have not run. FreeSBC&amp;rsquo;s capacity numbers are&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/performance.md#L3-L5" target="_blank" rel="noreferrer"&gt;explicitly preliminary&lt;/a&gt;, and&#10;&lt;a href="https://github.com/rasonyang/freesbc/blob/8bbc4e6ddb9ce752d14c64fa7362145a303ead99/docs/performance.md#L19" target="_blank" rel="noreferrer"&gt;there has been no run against FreeSWITCH&lt;/a&gt;. Everything here&#10;is single-node. When those numbers exist, they will get their own posts,&#10;with the setup next to them.&lt;/p&gt;&#10;&#10;&lt;h2 class="relative group"&gt;Coming up&#10; &lt;div id="coming-up" class="anchor"&gt;&lt;/div&gt;&#10; &#10; &lt;span&#10; class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"&gt;&#10; &lt;a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#coming-up" aria-label="Anchor"&gt;#&lt;/a&gt;&#10; &lt;/span&gt;&#10; &#10;&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Turn end is not playback end: why a bot cuts off its own goodbye&lt;/li&gt;&#10;&lt;li&gt;Barge-in is a truncation problem&lt;/li&gt;&#10;&lt;li&gt;One whitespace cut Qwen TTS first audio from 1.05 s to 0.25 s&lt;/li&gt;&#10;&lt;li&gt;Keeping a browser phone registered through sleep, Wi-Fi changes and F5&lt;/li&gt;&#10;&lt;li&gt;Why FreeSBC is a proxy and not a B2BUA&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Code: &lt;a href="https://github.com/rasonyang/ai-native-callcenter" target="_blank" rel="noreferrer"&gt;AICC&lt;/a&gt; · &lt;a href="https://github.com/rasonyang/freesbc" target="_blank" rel="noreferrer"&gt;FreeSBC&lt;/a&gt; · &lt;a href="https://github.com/rasonyang/cascade-realtime-gateway" target="_blank" rel="noreferrer"&gt;Cascade Realtime Gateway&lt;/a&gt; · &lt;a href="https://github.com/rasonyang/web-sip-phone" target="_blank" rel="noreferrer"&gt;Web SIP Phone&lt;/a&gt;&lt;/p&gt;&#10;</description><media:content xmlns:media="http://search.yahoo.com/mrss/" url="https://rasonyang.dev/posts/ai-native-call-center-in-four-projects/featured.jpg"/></item></channel></rss>