Evaluating Voice Cloning Models on CPU: A Practical Benchmark of Pocket TTS, Kokoro, Audio8, and XTTS-v2

Performed using NEO AI Engineer

If you ship TTS or voice cloning, you eventually need a straight answer: which model sounds natural, stays intelligible, actually clones the reference speaker, and still runs at a usable speed on CPU. This post walks through a full objective evaluation of four models on a CPU-only box, how the dataset and metrics were chosen, what broke along the way, what the numbers mean, and how you can clone the project and extend it with Neo.

Why this evaluation exists

Vendor demos and single-speaker samples are easy to game. A fair comparison needs:

• Many speakers, not one celebrity voice
• Gender balance and accent diversity
• Held-out text that never appears in the reference clip
• Metrics that separate cloning fidelity, naturalness, intelligibility, and latency
• An honest split between true zero-shot cloners and fixed-voice TTS

This case study was produced end to end by Neo, an autonomous AI engineering agent you can run from VS Code or Cursor. From a high-level goal, Neo planned the harness, integrated the models, built the data pipeline, ran the sweep, fixed a real data-quality bug, regenerated reports, and pushed the repo.

Models under test

That split matters. Only Audio8 and XTTS do genuine zero-shot cloning here. Pocket TTS and Kokoro are still useful as CPU quality and speed baselines, but their speaker-similarity scores are not cloning-fidelity evidence.

Metrics (what we measured and why)

All scoring is automatic. No MOS listening panel in this run.

1. Speaker similarity
   WavLM-Large x-vector embeddings, cosine similarity, both streams at 16 kHz. Higher is closer to the reference speaker. In practice this metric saturated near 1.0 for all four models, so it is a sanity check more than a ranking signal.

2. Naturalness (UTMOSv2)
   Predicted MOS on a 1-5 scale. Higher is more natural. This is where fixed-voice Kokoro pulled ahead of the true cloners.

3. Intelligibility (WER / CER)
   faster-whisper small (CTranslate2, CPU int8) → normalized text → jiwer. Lower is better. Hypotheses were spot-checked against real generated WAVs so WER was not a silent fallback.

4. Latency
   Wall-clock seconds and RTF (wall / audio duration). Peak memory for XTTS is not trustworthy in the parent process because XTTS runs in an isolated subprocess (venv_xtts).

Dataset and bias control

Early setup: LibriSpeech

The first passes used LibriSpeech test-clean (small speaker counts, few sentences). That is fine for wiring scorers and wrappers. It is not enough if someone will argue the benchmark is biased or too narrow.

Final design: VCTK sweep

Final numbers come from VCTK (British English multi-speaker corpus; HF mirror badayvedat/VCTK):

Different corpus from LibriSpeech also reduces “we only ever tested American read speech” risk.

VCTK evaluation design summary: 22 speakers, 11 accents, 12 unique texts per speaker, 1056 result rows, zero leakage

Figure: Final VCTK sweep design used for all published numbers.

The data bug Neo found and fixed

VCTK reuses elicitation sentences across recordings. The first manifest builder took the first N non-reference utterances without deduplicating by transcript text. Result:

• Only 6 unique texts per speaker out of 12 rows
• Distinct audio files, but repeated text pairs
• WER looked artificially strong on short repeated lines

Neo diagnosed that during score validation (not during the first green run), then fixed it:

1. Deduplicate candidates with a seen_test_text set in scripts/build_vctk_manifest.py
2. Expand the per-speaker candidate pool from 30 → 60 so 12 unique texts still fit after reference selection
3. Enforce separation on both utt_id and text
4. Rebuild the manifest, re-run all 1056 evaluations, regenerate aggregates and the report

Post-fix check: 12 unique test texts per speaker, 0/264 leakage rows, all 1056 rows status=OK.

That is the main non-trivial failure mode in this project. Integration pain (XTTS transformers pin, separate venv, CPU device for UTMOSv2, torchvision CPU wheel) was real but expected plumbing. The sentence-dedupe bug is the one that would have quietly misled an expert reader.

Harness architecture (short)

manifest_vctk.jsonl
        │
        ▼
   run_eval.py
   ├── wrappers/   pocket_tts, kokoro, audio8, xtts (subprocess)
   ├── energy gate (reject silent / empty audio)
   └── scorers/    speaker_sim, naturalness (UTMOS), wer, timing
        │
        ▼
output/results.csv → aggregated_results.csv
                   → zero_shot_comparison.*
                   → report.md

XTTS lives in venv_xtts and is invoked through xtts_subprocess_entry.py so Coqui’s transformers pin does not break the main env. Audio8 uses ONNX registration against a short reference clip plus transcript. Kokoro and Pocket TTS take documented fixed voices when true cloning is unavailable.

Hardware for the published run: CPU only, 8 cores, ~62 GB RAM, no GPU.

Results

All four models (264 unique sentences each)

Four-model comparison on UTMOS naturalness, WER intelligibility, and RTF latency

Figure: Aggregate UTMOS, WER, and RTF across 264 unique sentences per model. Kokoro leads naturalness and speed; Audio8 leads intelligibility.

CPU wall-clock latency per utterance for Pocket TTS, Kokoro, Audio8, and XTTS

Figure: Mean wall-clock seconds per utterance on the CPU-only evaluation box.

All 1056 rows passed the energy gate with non-null metrics.

Zero-shot only: Audio8 vs XTTS (528 rows)

Zero-shot comparison overview: Audio8 vs XTTS on UTMOS, WER, RTF, and speaker similarity

Figure: Zero-shot-only head-to-head (528 rows). Naturalness and speaker-sim are near ties; WER and RTF favor Audio8.

Per-speaker UTMOS naturalness for Audio8 vs XTTS across 22 VCTK speakers

Figure: Per-speaker mean UTMOS. Neither cloner wins every speaker.

Per-speaker WER intelligibility for Audio8 vs XTTS across 22 VCTK speakers

Figure: Per-speaker mean WER (%). Audio8 is at or near zero on many speakers; XTTS shows more residual error.

Per-speaker UTMOS: Audio8 ahead on 13/22 speakers, XTTS on 9/22. Naturalness is not a clean sweep either way. Intelligibility and speed favor Audio8 on this box.

What changed after the sentence fix

With duplicated elicitation text, WER looked too good (for example Kokoro ~0.0017). After unique texts, Kokoro WER moved to ~0.0051 and XTTS to ~0.0091. The ranking story stayed directionally similar, but the absolute WER numbers became honest for a multi-sentence claim.

LibriSpeech vs VCTK also mattered for naturalness among cloners: on the smaller LibriSpeech set Audio8 led XTTS by a larger UTMOS margin; on VCTK they are effectively tied. That shift is a reason not to overfit conclusions to one small speaker pool.

How to read the outcome

Speaker similarity
Do not pick a winner from WavLM cosine here. Everything sits near 0.99. Use it to confirm “sounds like a real same-speaker condition,” not to rank models.

Speaker similarity scores clustered near 1.0 for all four models

Figure: WavLM speaker similarity is saturated. Treat it as a sanity check, not a leaderboard.

Naturalness
Kokoro is the clear leader (UTMOS ~3.69) if you can accept a fixed voice. Among true cloners, Audio8 and XTTS are a coin flip on average MOS.

Intelligibility
Audio8 posts the best WER overall (~0.0037), including among true cloners. XTTS is still strong in absolute terms (~0.009) but clearly behind on this ASR-based measure.

Latency on CPU
Kokoro is comfortably real-time (RTF ~0.12). Pocket TTS is also real-time (~0.24). Audio8 is about 1.4× real-time. XTTS is about 5.3× real-time (~21 s wall per utterance on this machine). If your product is interactive CPU TTS, XTTS is a hard sell without a GPU path.

Practical recommendations

Caveats worth keeping in the open: objective-only (no human MOS), Whisper small for WER, Pocket/Kokoro not true cloners in this run, XTTS peak memory not measured in-process, no paired bootstrap or significance test on the deltas.

Full tables and methodology live in output/report.md in the repo.

What Neo actually did

Neo is a fully autonomous AI engineering agent for fine-tuning, evaluation, classical ML, RAG pipelines, and related shipping work. In VS Code or Cursor it takes a high-level goal, plans, writes code, runs commands, and iterates until the job is done.

For this case study, the human side was essentially the goal and follow-up pressure (“make this defensible for people who know TTS”). Neo:

1. Designed a multi-model CPU evaluation harness (run_eval.py, wrappers, scorers)
2. Integrated four different model stacks, including an isolated XTTS environment
3. Started on LibriSpeech for bring-up, then moved to a VCTK multi-speaker, multi-accent design
4. Built manifest generation with gender balance, accent coverage, and leakage checks
5. Ran the full 1056-row sweep and wrote comparative reports
6. When asked to prove scores were real, re-validated scorers (including direct Whisper checks on WAVs)
7. Found the VCTK repeated-sentence bug, fixed the builder, re-ran the sweep, and rewrote the report’s methodology section
8. Packaged README, .gitignore, Apache-2.0 license, and pushed github.com/gauravvij/voice-clone-eval

You do not need to re-implement that loop by hand to build on it. Clone the repo and drive the next experiment with Neo.

Replicate or extend this with Neo

1. Clone the project

git clone https://github.com/gauravvij/voice-clone-eval.git
cd voice-clone-eval

Follow README.md for the two venvs (./venv, ./venv_xtts), model downloads, and:

./venv/bin/python run_eval.py \
  --manifest data/manifest_vctk.jsonl \
  --results output/results.csv

Large artifacts (model weights, raw VCTK audio) are gitignored on purpose. Rebuild or download them locally as documented in the README.

2. Open the folder in VS Code or Cursor with Neo

Install the Neo extension, open this repo, and give Neo a concrete next goal. Examples that map cleanly onto the existing harness:

Add a fifth model

Clone https://github.com/gauravvij/voice-clone-eval and add OpenVoice (or another model I name) as a fifth wrapper. Keep the same scorers and VCTK manifest. Re-run evaluation for the new model only if possible, then update aggregated_results.csv, zero-shot tables if it is a true cloner, and report.md.

GPU path and fair latency

Extend this repo so XTTS and Audio8 can run on CUDA when a GPU is present, keep CPU fallback, and log device + RTF side by side. Do not change the VCTK speaker/sentence design.

Stronger intelligibility scoring

Swap faster-whisper small for whisper medium (or large-v3) on CPU or GPU, re-score existing WAVs in output/audio without regenerating speech if files exist, and compare WER deltas in a short appendix in report.md.

Statistical tests

Add paired bootstrap confidence intervals over the 22 speakers for Audio8 vs XTTS on UTMOS and WER. Write results into output/stats_bootstrap.json and a short section in report.md. Do not claim significance without showing intervals.

Human spot-check set

Sample 40 clips stratified by model and gender, write a simple HTML listening sheet with hidden model labels, and store the sample list in data/listening_sample.jsonl.

New language or corpus

Replace VCTK with a multilingual corpus I specify. Keep leakage controls and unique-text dedupe. Rebuild manifest, run all models that support the language, and regenerate the report.

CI smoke test

Add a GitHub Actions workflow that installs CPU deps, runs a 2-speaker × 2-sentence smoke manifest, and fails if any row is not status=OK.

3. How to prompt Neo well

• Point at the repo path or clone URL first
• State the decision you care about (latency, WER, true cloning only, etc.)
• Say what must stay fixed (VCTK 22×12, metric definitions, Apache license)
• Ask for regenerated artifacts (results.csv, report.md) not only code edits

Neo’s job is the full loop: plan, implement, run, notice broken assumptions, fix, and leave evidence in the repo.

Bottom line

• For CPU fixed-voice quality and speed, Kokoro wins this sweep.
• For true zero-shot cloning on CPU, Audio8 is the practical pick: similar naturalness to XTTS, better WER, much lower latency.
• Speaker-sim alone will not rank these models.
• The evaluation only became trustworthy after unique-text dedupe on VCTK and a full re-run.

If you want the raw tables, methodology, and code path Neo left behind, start here: https://github.com/gauravvij/voice-clone-eval.

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor