Self-learning voice agents

Problem

Voice agents fail in ways a transcript does not show. They talk over each other, leave long pauses or take turns clumsily. Testing for that by hand means placing call after call and listening to every one.

I wanted a phone call to become a repeatable test. That needs three things: agents that hold a real phone conversation, a grader that scores the recording, and a loop that acts on the score and calls again. I built all three, and kept the whole audio path on one laptop.

How it works

Two Telegram accounts, Katey and Chippy, each run their own copy of the same local voice agent. Katey dials, Chippy answers, and the audio travels over the real Telegram network.

1. Two processes, one call.Each side runs in its own Python process. Two call clients in one process crash with SIGABRT inside the shared WebRTC library, because they share global state. A parent script starts Chippy, waits for it to report that it is ready, then starts Katey to dial.
2. Hearing.The other side's voice arrives as MP3 frames written into a pipe. ffmpeg decodes them to 16 kHz mono audio. FireRedVAD checks the last 100 ms for speech. A turn ends only after two silent windows in a row, once it has heard speech, and only if the utterance is at least 1.5 s long and loud enough to be real.
3. Understanding.Qwen3-ASR 0.6B turns the utterance into text. If the text overlaps the agent's own last reply by more than 60%, it is its own voice bouncing back through the call, and the agent drops it.
4. Replying.Qwen3.5 (0.8B to 4B, on Apple's MLX) streams its reply token by token. As soon as a phrase reaches a comma or full stop it goes to Kokoro-82M, and that audio is sent back over the call in 10 ms frames while the model is still writing the rest.
5. Interrupting.While an agent is speaking it keeps measuring the incoming loudness. If the other side clearly starts talking, it stops between phrases and clears what it had queued, so the other person takes the floor.
6. Recording.Each side saves a stereo recording, its own voice on the left and what it heard on the right, plus per-turn timings and a merged transcript. That recording is what the judge listens to.

One detail cost an hour. MLX ties the GPU stream to the thread that loaded the model, so each agent runs a single worker thread that loads all three models and serves every inference call.

Making it quick

The first end-to-end call had about 5 seconds of dead air between turns. These changes brought it down, in order of impact, as reported in the build story:

“Time to first audio” is measured inside the agent: from the moment it starts writing a reply to the moment the first synthesised audio is ready. The pause the other person hears is longer, because it also includes the network both ways, decoding, the wait for silence and speech recognition.

The judge

Once the calls got long, I could not listen to every one. judge_call.py mixes the stereo recording down to mono, uploads it to Gemini (Gemini 3 Flash Preview by default) and asks for JSON: a pass or fail, scores from 0 to 10 for pacing, overlap, coherence and naturalness, the typical gap between turns in seconds, and a list of issues.

The prompt asks the judge to describe what it hears, not to prescribe fixes. Deciding what to change is the loop's job. A call passes only if pacing is at least 8, overlap at least 8, coherence at least 7, naturalness at least 6 and the gap 2.5 s or less. Each evaluation cost under half a US cent.

{
  "passed": false,
  "pacing_score": 4,
  "overlap_score": 10,
  "coherence_score": 9,
  "naturalness_score": 7,
  "observed_response_latency_s": 3.7,
  "issues": [
    "Long silences between turns ranging from 2 to 6 seconds",
    "Chippy's voice is more monotonous and robotic…"
  ]
}

The loop

loop_optimize.py ties it together. Each iteration places a two-minute call, has it judged, and, if it did not pass, makes one small change before calling again. An iteration takes three to five minutes, including a 75-second cooldown that lets Telegram's call state settle.

1. Score the call.The composite is the average of the four 0–10 scores, minus one point for every second of gap beyond the first (capped at six). It is only used to compare iterations.
2. Pick one rule.Rules are checked in a fixed order: talking over each other first, then cut-off speech, then junk transcripts, then slowness, then rambling. The first that matches fires. A slow call, for example, trims the post-speech guard by 12%, the opening deafness by 10% and the reply length by 8%.
3. Stay in bounds.Every one of the eight settings is clamped to a safe range. If a call scores 1.5 points below the best so far, the loop goes back to the best settings. A call with more than 30 s of silence is treated as a network glitch and retried unchanged.
4. Stop.The loop ends when the judge passes a call or the iteration cap is reached. Low naturalness or coherence is reported but not acted on: the levers there are the voice model and the prompt, which I left to change by hand.

Results

• ~600 msmedian time to first audio per turn, measured in the agent
• 5.75best judge composite out of 10, from nine optimisation loops
• 12turns in a live call with a person on a real phone, interruption working
• ~$0.05Gemini judge spend for the whole session

I ran nine optimisation loops in one evening. The best judged call scored 5.75 out of 10: pacing 6, overlap 10, coherence 8, naturalness 7, with a 3.0 s gap between turns. It came on the second iteration of one loop, with the 0.8B model, and those settings became the repository's defaults. No call passed the judge.

Then I spawned only Katey and answered the call myself on a phone. The first call ran ten clean turns with no overlap and about 120 ms of speech recognition per turn.

Gemini's verdict on that first human call was harsh: “severe overlap”. Katey's own voice was leaking back through Telegram and being transcribed as if I had said it. The echo check and the interruption logic above both came from that. On the next call, Katey argued for Android against my iPhone for 12 turns. I cut her off mid-sentence at the start and she stopped and listened. Time to first audio stayed between 550 and 930 ms. She also stated things that are not true, and doubled down when I pushed back.

The loop also ran into a wall that no setting could fix. After about 30 rapid calls from the same pair of accounts, Telegram kept connecting the calls but relayed only the first few seconds of audio.

While the throttle cleared I built a bypass, run_duet_local.py: both agents in one process, with one agent's audio piped straight into the other's ears. It showed the local pipeline worked without the network and exposed a VAD bug that Telegram's jitter had been hiding.

One agent with the 4B model uses about 6.4 GB of memory; both together about 13 GB. The 0.8B model brings an agent down to about 2.5 GB.

Limits

• The “learning” is iterative evaluation and tuning of eight pipeline settings by hand-written rules. No model weights are trained.
• The judge is a model, not human listeners. Its scores were not checked against people, and each call was judged once.
• The numbers come from a single session on one laptop with one pair of accounts. There are no repeat runs or error bars, and no call passed the judge's bar.
• About 600 ms is the agent's own time to first audio. The silence the other side hears is several times longer, and that gap is where the best call failed.
• Telegram throttled the account pair after about 30 rapid calls, which cut the run short.
• The small models drift from their persona and state false facts with confidence. The 4B model refuses some topics outright.
• It is an experiment, not a product.

Credits and links

I built the harness, the judge and the loop. It stands on open work: mlx-lm and mlx-audio for running models on Apple Silicon, Kokoro-82M for speech, Qwen3-ASR and Qwen3.5 from the Qwen team, FireRedVAD from FireRedTeam, and pytgcalls and Telethon for the Telegram bridge. Gemini is used only to grade finished calls and never touches the agents at runtime. I have no affiliation with any of these projects or companies.

• voice-evolve on GitHub: the agents, the duet runner, the judge and the loop, with the full list of tunable settings. MIT licence.
• The build story: the night it was built, with transcripts, latencies and judge scores.