Making OmniVoice 6.2× Faster Than Its 32-Step Default Without Retraining

Performed using NEO AI Engineer

OmniVoice is a text-to-speech model that generates several synchronized streams of audio tokens, known as codebooks. Out of the box, it sounds good, but inference is too slow for many interactive applications. We used Neo to investigate a practical question on OmniVoice 0.2.1 and an NVIDIA RTX A6000: how far could inference be accelerated without retraining, and would the result still survive a reserved holdout?

The brief

Speed up OmniVoice at inference time. Do not retrain. Do not distill. Keep the same checkpoint. Change decoding and systems only. Measure batch-1 end-to-end latency honestly against both the default 32-step path and the official 16-step path. Freeze quality gates. Evaluate holdout only at predefined phase gates, never while tuning candidates. Publish what fails. Prefer a smaller true win over a headline that dies on holdout.

The work covered the full loop: build the harness, profile the model, add a compile stack, test step counts, investigate algorithmic alternatives, check FlashInfer, package listening samples, and leave behind a reproducible final YAML configuration.

The short answer: the final configuration uses torch.compile + 10 unmask steps + guidance 2.5. That is about 6.2× faster than the default 32-step eager path, and about 3.2× faster than the official 16-step eager path, at batch size 1, with holdout still inside the gates.

Outcome at a glance

• Task: make OmniVoice faster without changing weights
• Final configuration: torch.compile (reduce-overhead) + num_step=10 + guidance_scale=2.5
• Post-warm-up speedup: ~6.2× vs default 32-step eager · ~3.2× vs official 16-step eager
• Quality rule: holdout must be faster and stay inside word error rate (WER), speaker similarity (SIM), and predicted speech quality (UTMOS) gates
• Tried and rejected: confidence-parallel unmasking, CFG late-off, Flash Attention 2 hooks, FlashInfer on batch-1
• Proof: JSON ladders, final YAML configuration, charts from artifacts, and listening WAVs embedded below

This writeup is the narrative companion to the research repository, which includes raw JSON results, listening samples, the final YAML configuration, and charts generated directly from those artifacts.

Host: NVIDIA RTX A6000 48GB · Model: OmniVoice 0.2.1 · No weight updates · No distillation

Research journey: cumulative post-warm-up speedup vs baseline A

Figure 1: How far each phase moved post-warm-up speedup vs baseline A. The hollow red point is compile + 8 steps: faster on paper, holdout WER fail, never shipped.

Table of contents

1. The brief
2. Rules set before optimizing
3. What OmniVoice actually does at generate time
4. Phase 1: Build the harness
5. Phase 2A: Profile before changing anything
6. Phase 2B: Make compile the stack
7. Phase 3: Choose the final configuration
8. Phase 4: Test additive bets
9. Phase 5: Check FlashInfer, then retain the configuration
10. Listen: speed and cloning under the final configuration
11. Honest ceiling
12. How the work was run
13. Artifact map
14. Takeaways

1. The brief

Speech models are judged twice: once by how they sound, and once by how long you wait. Training-time tricks can move both, but they change the model. We did not want a new student checkpoint. The goal was to keep the same OmniVoice weights and find a meaningful speedup for the tested A6000 workload.

That is a hard systems problem. OmniVoice unmasks eight codebooks at once, with knobs like step count, guidance, t_shift, temperatures, and layer penalty. Clever decoding tricks can put those codebooks out of sync and ruin naturalness. The confidence-parallel experiment later demonstrated that failure mode directly.

2. Rules set before optimizing

Before testing candidates, we locked the comparison rules. Any idea that broke them would not ship.

Dual baselines (report both, not just the easy one)

A speedup only versus A is easy to overclaim. Beating B is the harder, more honest bar. Both are reported throughout.

Primary speed metric S (never redefined mid-run)

S = Σᵢ wᵢ · median_RTFᵢ
RTF = t_generate / t_audio

All latency results below are measured after the required one-time compilation and graph-capture warm-up. “Post-warm-up” refers to fresh benchmark runs after that setup step.

• Batch size 1
• End-to-end model.generate with CUDA synchronize
• Fixed English operating-point grid and weights in configs/operating_points.json
• Seeds paired {11, 22, 33} on ladders

Quality gates (set from measured noise, not gut feel)

From Phase-1 repeated baseline noise:

Duration ratio was demoted to a diagnostic check after it falsely failed good configs (the manifest expected_duration was often about 2× the real audio length).

Holdout only at predefined phase gates

Reserved holdout clips (hold_ss_01, hold_sm_01, hold_dm_01) were excluded from development tuning. They were evaluated only at predefined phase gates, where a candidate had to be both faster and inside the quality thresholds.

Publish negatives

Failed ideas are part of the result, not something to hide. Dead ends go in ledger/failed_approaches.md. Reopening them without a new mechanism was out of scope on purpose.

3. What OmniVoice actually does at generate time

Before changing anything, the generate path had to be understood. This simplified picture drove the later choices.

The generate path is roughly:

generate
  → preprocess (voice clone prompt / encode)
  → iterative unmask loop  (dominates wall-clock)
       • each step: one forward on batch 2B (conditioned + unconditioned CFG fused)
       • multi-codebook token updates under schedule knobs
  → decode + post-process

Two facts drove the whole ladder:

1. Unmask steps dominate (~95% of end-to-end time on the profiled short and medium inputs). Cutting steps is the first-order lever if quality holds.
2. Classifier-free guidance (CFG) already runs as one fused 2B forward, where B is the request batch size, rather than two serial passes. “Turn CFG off late” does not automatically halve time.

Instruct mode is picky: free-form prose fails; comma-separated tokens like female, american accent, young adult work. Clone mode needs accurate ref_text for the reference WAV.

4. Phase 1: Build the harness

We began with the measurement harness: pin the environment, define the workload, lock baselines A and B, and test the inexpensive knobs before touching the model loop.

Deliverables: environment pins, workload manifest, benchmark runner, metrics, failure ledger, and the first baseline table.

Initial baseline picture (later refined; order of magnitude stable)

CFG late-off: a clean negative

One experiment patched the iterative loop so later steps could run conditioned-only, using half the usual CFG batch. Quality often survived, but wall-clock did not beat simply cutting steps. The fused CFG forward plus Python step overhead ate the theoretical win. Reference code stays at src/harness/cfg_economy.py.

Lesson

On this model, shortening the unmask schedule beats clever CFG width tricks for end-to-end latency.

Artifacts: artifacts/cold_results.json · reports/phase1_status.md · reports/profile_baseline.md

5. Phase 2A: Profile before changing anything

Before introducing more optimizations, profiling measured where the wall-clock time actually went. It ranked the real opportunities:

Artifact: reports/deep_profile.md (full traces were generated under artifacts/deep_profile/; large dumps are gitignored, summary remains).

6. Phase 2B: Make compile the stack

Next, torch.compile went under the hot path. Step counts were tested again on top of that compiled baseline, and only holdout-safe points survived.

What changed

1. Wrap OmniVoice.forward with torch.compile(mode="reduce-overhead")
2. Warm each input shape once so CUDAGraph capture is paid before measurement
3. Test the step-count ladder again on the compiled path
4. Run a fresh post-warm-up benchmark, then the reserved holdout evaluation

Result

Eight steps narrowly exceeded the aggregate holdout WER gate (ΔWER ≈ +0.052 against a +0.05 limit), and one clip reached 0.30 WER. That miss is why Phase 3 tested intermediate step counts.

Artifacts: artifacts/phase2_cold_results.json · artifacts/phase2_holdout_results.json · reports/report.md

7. Phase 3: Choose the final configuration

Compile + 8 steps was faster on the development ladder, but failed holdout. This phase searched for an intermediate point that was both faster and defensible.

Hypothesis

Because each unmask step costs roughly the same under compile, fewer steps should translate into predictable speed gains:

The search also tested knobs that might recover quality at lower step counts: guidance_scale=2.5, position_temperature=0.0, layer penalty 3/7.

Development ladder (compressed)

• Kept for post-warm-up remeasurement: 10/12/14-step defaults and guidance 2.5 variants
• Best on development: compile + 10 steps + guidance 2.5 (S ≈ 0.025)
• Rejected: position_temperature=0.0, because long-form WER and UTMOS collapsed (e.g. ΔUTMOS −0.5 on some cells)

Fresh post-warm-up benchmark

Post-warm-up benchmark: batch-1 speed ladder

Figure 2: Primary metric S (weighted median RTF) on a fresh post-warm-up benchmark. Lower bar = faster. The final 10-step configuration reaches S = 0.0255, or 6.17× faster than baseline A.

Median RTF by operating point

Figure 3: Same four configs, split by operating point. The final configuration wins everywhere on the grid; short-short still has the highest RTF (fixed overhead), but absolute latency is tiny.

Final holdout evaluation

Holdout quality deltas vs gates

Figure 4: Holdout quality tax vs reserved baseline A. Green band = inside gate; red band = fail. Both the compiled 16-step and final 10-step configurations pass; the 10-step configuration is faster, so it is accepted. The tax is real but small (ΔWER ≈ +0.019, ΔSIM ≈ −0.013, ΔUTMOS ≈ −0.035).

The validated configuration for this A6000 benchmark was written to configs/production_infer.yaml. The previous compile + 16-step configuration remains available as previous_safe_phase2.

Artifacts: artifacts/phase3_ladder.json · artifacts/phase3_cold_results.json · artifacts/phase3_holdout_results.json · reports/phase3_report.md

8. Phase 4: Test additive bets

With a strong validated configuration in hand, Phase 4 looked for additive wins. None beat the batch-1 result without breaking quality, so the Phase-3 configuration stayed.

Confidence-parallel unmask (Fast-dLLM-style)

This approach unmasks every position above a confidence threshold at each step, fills anything left on the final step, and exits early if no masked positions remain.

Result: every confidence-threshold × step-limit combination failed quality. Speaker similarity and UTMOS collapsed, and WER often did too. Early exit almost never triggered; the only speed came from reducing the step limit, which is the same lever as Phase 3 with a worse schedule.

Why it hurts here: the eight codebooks are sensitive to which positions commit together. Confidence-parallel updates changed that coordination, so naturalness collapsed even when the transcript remained partly intelligible.

t_shift micro-grid

Only t_shift=0.05 showed a possible ~1% post-warm-up improvement over the final configuration. It was not faster on holdout, so it was rejected.

Flash Attention 2

The prebuilt wheel installed, but OmniVoice still reported that it does not support Flash Attention 2. Forcing the nested attention implementation then failed because the expected packed-sequence inputs did not match OmniVoice's 4D masks.

Throughput (secondary chapter)

Simple batches of 2, 4, and 8 under compile delivered fewer utterances per second than batch 1 because of padding and graph-capture overhead. Higher throughput needs a serving design; it is not another batch-1 S optimization.

Decision: retain the existing Phase-3 configuration.

Artifacts: artifacts/phase4_ladder.json · artifacts/phase4_holdout_results.json · artifacts/phase4_throughput.json · reports/phase4_report.md

9. Phase 5: Check FlashInfer, then retain the configuration

OmniVoice's upstream research path includes a FlashInfer implementation for packed classifier-free guidance. We added that official integration to the harness (src/harness/omnivoice_flashinfer.py + flashinfer_accel.py) and ran a full smoke test → microbenchmark → development ladder → post-warm-up benchmark → holdout gate protocol without stacking compile by default.

FlashInfer vs final configuration

Figure 5: Same decoding settings (10 steps + guidance 2.5), three backends. FlashInfer achieved 63–72% of the final configuration's speed, making it roughly 1.4–1.6× slower. The integration worked correctly but did not improve latency.

Throughput vs batch size: final configuration vs FlashInfer

Figure 6: Throughput in utterances per second (higher is better). The final compiled configuration dominates batch 1, the target setting for this study. FlashInfer improves at batch 8, a useful serving signal but not a reason to change the batch-1 configuration.

Interpretation: The final configuration's torch.compile(reduce-overhead) path already fuses the backbone efficiently for single short and medium utterances. FlashInfer's packed prefill and Python-side iterative loop did not beat that complete batch-1 path, even if individual kernels looked better. A speedup over eager execution is not necessarily a speedup over an already-compiled baseline.

Artifacts: artifacts/phase5_smoke.json · artifacts/phase5_micro.json · artifacts/phase5_ladder.json · artifacts/phase5_holdout_results.json · reports/phase5_report.md

10. Listen: speed and cloning under the final configuration

Numbers are necessary. Ears provide an important check on whether the measured quality tradeoff is acceptable. The clips below are the same WAVs committed in the research repo, hosted here so you can compare them without cloning.

Baseline A vs final configuration

The listening pack contains nine matched pairs at seed 11, with the same text, reference audio, and instruction settings in each comparison.

Listening pack: paired baseline A vs final-configuration latency

Figure 7: Nine paired clips (seed 11). Grey = baseline A wall-clock; purple = the final configuration. Numbers on the right are per-clip latency speedups. Short lines can exceed 10×; long lines still clear ~3.5–4×.

Short clips show larger multipliers because fixed overhead dominates their baseline latency. Longer clips still reach roughly 3.5–4× on this pack.

Short clip (dev_ss_01, seed 11)

Baseline A (32-step eager)

Final configuration (compile + 10 steps + guidance 2.5)

Medium clip (dev_sm_01, seed 11)

Baseline A (32-step eager)

Final configuration (compile + 10 steps + guidance 2.5)

For the remaining pairs and listening notes, see artifacts/listening_trial_p3/LISTEN_HERE.md.

Voice clone demo (stock speakers)

The clone test gives P3 a reference WAV and its transcript, then asks it to speak new target text in the same voice.

Reference (dev_ss_01, female short)

final-configuration clone (same speaker id, new target text, seed 11)

For the full clone pack, see artifacts/voice_clone_demo/README.md.

Side-by-side listening on the speed pack confirmed the final configuration is usable in practice. You can hear a quality cost on hard long lines, but it stays inside the agreed quality gates.

11. Honest ceiling

This is the measured ceiling on this host and workload, plus the ideas that are unlikely to deliver another large batch-1 gain without a genuinely new mechanism.

On this English operating-point mix, A6000, OmniVoice 0.2.1, batch 1:

S ≈ 0.025–0.026   under compile + 10 steps + guidance 2.5 + SDPA

That is roughly:

• 6× vs default 32-step eager
• 3× vs official 16-step eager
• Now bounded by roughly 10 unmask steps, the compiled graph, and fixed per-request overhead

What will not unlock another factor on batch-1 (given what we tried)

• More official knob grids without a new mechanism
• Classifier-free guidance width tricks
• position_temperature=0.0 for deterministic positions
• Confidence-parallel force-fill schedules
• Flash Attention 2 flags the model rejects
• FlashInfer alone vs an already-compiled graph

What still might help (outside this study’s scope)

1. Serving: continuous batching or ragged multi-utterance batches (FlashInfer's scaling above batch 1 is a useful signal)
2. Training-time students / consistency / fewer-step distillation
3. Custom kernels designed around OmniVoice’s actual mask layout
4. Broader validation: multilingual and difficult long-form inputs, plus Whisper large-v3 scoring, before making broader claims

12. How the work was run

Neo executed the research loop from the high-level brief. Human involvement focused on setting the goal and acceptance criteria, confirming the audio by ear, and deciding how the result should be packaged.

What humans did

• Set the goal: training-free OmniVoice speedup with holdout-gated quality
• Accepted the speed/quality tradeoff once a final candidate was ready
• Confirmed listening on the speed and clone packs
• Steered packaging so the repo and this writeup stayed usable

What the run delivered

1. Built the harness, pins, operating-point grid, and quality gates
2. Profiled the generate path and ranked real bottlenecks before inventing knobs
3. Put torch.compile under the hot path and re-laddered step counts
4. Closed the 8-step vs 16-step gap and froze configs/production_infer.yaml
5. Tried additive algorithmic bets and FlashInfer, then retained the existing configuration when they lost
6. Published negatives in the failure ledger instead of burying them
7. Packed listening WAVs, clone demos, JSON artifacts, and phase reports

How the loop ran

plan → harness → measure noise → profile → rank windows
    → implement one window → ladder with gates
    → post-warm-up remeasure → holdout at a predefined phase gate
    → either accept the candidate or retain the existing configuration
    → log failures → next phase only if headroom remains

That pattern is why the result is narrowly scoped and trustworthy:

• Keeps the speed metric stable (S is frozen early)
• Stops holdout overfitting (phase-gated evaluation, never development tuning)
• Avoids stacking optimistic development numbers as if they were final benchmark results
• Turns failures into reusable knowledge instead of undocumented dead ends

The same harness can re-run every phase via scripts/run_phase*.py.

Replicate or extend the work

Clone the repo, open it in VS Code or Cursor, and use Neo to continue from the existing harness. Useful next goals include:

• "Re-measure the final 10-step configuration on a different GPU and report post-warm-up S vs A and B with the same gates"
• "Extend the holdout set with longer hard lines and decide whether to accept the candidate or keep production_infer.yaml"
• "Build a tiny serving benchmark for batch sizes 1/2/4/8 under the final configuration, and say whether FlashInfer wins anywhere"

13. Artifact map

Validated configuration and evaluation rules

Key result files

Phase reports

Listening packs

Section 10 embeds six representative clips from these packs. The full WAV sets stay in the research repo.

Chart sources

Every chart is generated from the JSON artifacts above; docs/figures/manifest.json records the source mapping.

14. Takeaways

1. Training-free optimization can still deliver a multi-fold gain when inference is dominated by repeated steps and compilation can fuse the hot path.
2. Dual baselines keep the claim honest: beating the 32-step default is useful; beating the official 16-step path is the stronger comparison.
3. Holdout protects you from pretty development numbers: eight steps would have shipped without it.
4. A decoding trick does not transfer automatically: confidence-parallel unmasking damaged multi-codebook TTS quality.
5. A faster kernel does not guarantee a faster request: FlashInfer worked, but the compiled final path still won at batch 1.
6. Knowing when to stop is part of the result: Phases 4 and 5 found no defensible replacement for the final configuration.
7. Pair quantitative metrics with listening samples: the committed audio is part of the evidence, not decoration.

Validated configuration for this benchmark

If you only need the validated configuration, use this:

# Validated configuration from configs/production_infer.yaml
num_step: 10
guidance_scale: 2.5
t_shift: 0.1
torch_compile: true
torch_compile_mode: reduce-overhead
# position_temperature: 5.0  # do not set to 0.0

Warm up the compiled graph, measure RTF with CUDA sync, gate on WER / speaker similarity / UTMOS, then listen.

Closing

We asked for honesty over a headline multiplier. Neo ran a disciplined loop built around frozen metrics, phase-gated holdout evaluation, and public negative results. The result is a narrow, trustworthy configuration for the tested A6000 workload and a clear map of what not to try again.

If you only take one thing from this repo, take the process. The validated YAML configuration is the reusable result. The failure ledger is how you avoid replaying the same dead ends.

Companion README: README.md · Upstream model: k2-fsa/OmniVoice · Research loop executed with Neo

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor