Optimize & Deploy
CPU-Based Voice Assistant
1.25s time-to-first-audio using sub-sentence streaming and multi-threaded TTS, no GPU required.
The problem
Voice assistants feel sluggish because most TTS pipelines wait for a full sentence before speaking, stacking latency on CPU-only hardware.
- The assistant goes silent for a beat that feels like forever before it starts talking.
- You'd add a GPU just to fix latency, but the deployment target doesn't have one.
- Users repeat themselves because the pause before the response reads as "did it hear me?"
What NEO built
NEO built a multi-threaded pipeline with punctuation-triggered chunking and an ONNX-based TTS model running concurrently with LLM streaming and audio playback, with no GPU involved.
The result
1.25s time-to-first-audio
Reaches 1.25s time-to-first-audio on CPU alone, inside the threshold for natural conversational responsiveness.
From the blog · 8 min
Building a Low-Latency CPU-Based Voice Assistant with Streaming TTS
NEO built a CPU-only voice assistant with 1.25s time-to-first-audio using sub-sentence streaming, KittenML TTS, and a multi-threaded producer-consumer pipeline—no GPU required.
Try this in your workspace
Paste this into NEO chat to kick off the same workflow on your own data.
Build a CPU-only voice assistant that streams TTS audio as soon as it has a punctuation-complete chunk, instead of waiting for the full LLM response, so it feels instant without a GPU.
Paste it in · review the plan · get the diff
Get NEO
