Grok 4.6 vs Kimi K3 vs Opus 5 on Kokoro-82M TTS: Strong Research, No Production-Ready Speedup

Study scores: Opus 83, Kimi 60, Grok 51. Grok's best verified gain was about 6–8% in one setup, but audio checks failed, so nothing shipped.

In plain English

Kokoro-82M turns text into speech. We asked three frontier models (Kimi K3, Opus 5, and Grok 4.6) to make it faster on CPU through the same Neo Bring Your Own Key (BYOK) workflow. (Earlier: Kimi vs Opus only.)

None of the three proved a change that should ship unchanged.

The numbers that matter

83/100 is not "fastest speech." It ranks how trustworthy the optimization study was. Speed still had to survive a second measurement, and the audio still had to match.

What we asked them to do

1. Profile Kokoro on CPU
2. Try optimizations
3. Report what to ship

Then we scored the reports, re-ran the speed claims, and checked whether the speech still matched the original.

The journey, simply

1. Same task, same model, three submissions
2. All three found the same rough bottleneck (heavy convolution / vocoder work)
3. Score the studies: Opus 83, Kimi 60, Grok 51
4. Re-measure Grok's INT8 idea under a stricter protocol
5. Speed shows up in one setup (~6–8%); shrinks when both models sit in memory (~2–3%)
6. Audio gates fail, so reject for production

Opus 5: best study (83); thread speedup did not hold as a shippable win
Kimi K3: mid study (60); thread idea also rejected after replay
Grok 4.6: weakest study (51); real ~6–8% idea, audio failed

Quick glossary

Hear why audio blocked shipping

These clips are from the blind retest pack (unlabeled on purpose). If the optimization were safe, pairs should sound essentially the same. They did not pass the objective gates.

Sample 01

Sample 02

More samples and a score sheet are in the technical section below.

Bottom line for a quick read

• Opus 5 wrote the strongest study (83/100).
• Grok 4.6 found a modest real speedup (~6–8% isolated), not the huge factor in the original writeup.
• Audio failed on that path, so it does not ship.
• Lesson: profiling skill, a clever idea, hard proof, and production judgment are different things. This task needed all four.

Explore the source on GitHub · Try Neo AI Engineer →

Full technical writeup

TL;DR

Opus 5 remains the strongest research artifact at 83/100, ahead of Kimi K3 at 60 and Grok 4.6 at 51.

Grok's score does not mean it had no useful idea. Two full retests found Linear-only INT8 6.18-7.91% faster in process isolation, but only 2.45-3.38% faster with both pipelines resident. Every objective audio comparison failed.

The fair verdict is: a modest deployment-sensitive optimization, weakly justified in the original submission, and not safe to ship as tested.

Three-way study quality scores for Kimi K3, Opus 5, and Grok 4.6

Table of contents

1. Outcome in one table
2. Why Grok scored low
3. The profile was useful
4. Audit the scoreboard
5. First replay: the headline disappears
6. Full retest: deployment shape matters
7. The audio still blocks production
8. What the result means
9. What should be tested next?
10. Artifacts map
11. Technical bottom line

Outcome in one table

This ranks three repositories for one task, not the underlying language models generally.

Why Grok scored low

Grok was good at exploration. It earned 29 of 40 points for profiling and optimization coverage. It profiled MKLDNN operators, measured memory, tested BF16, compilation, threads, caching, dynamic quantization, ONNX, and OpenVINO, and preserved useful failures.

Most lost points came from the trust layer:

• 7/25 for experimental and statistical rigor.
• 5/15 for correctness and audio validation.
• 2/5 for maintainability.

The submitted report mixed workloads, lacked paired confidence gates, ranked dummy OpenVINO inference beside working TTS, used an unsupported perceptual MSE threshold, and recommended output it described as noticeably degraded. That is why the artifact scored 51. It is not a claim that Grok is generally bad at optimization.

Rubric breakdown for the three Kokoro optimization artifacts

Inspect the output: original Grok report · machine-readable scorecard

The profile was useful

All three submissions point to convolution-heavy waveform synthesis.

Hotspot comparison across the three Kokoro CPU profiles

Grok measured MKLDNN convolution at 57.3% of self CPU time, followed by normalization and elementwise work. That independently agrees with the ONNX studies: Kokoro's generator/vocoder is the real optimization surface.

The surprising part is that Linear layers were a small profile component, yet Linear-only dynamic quantization became the production recommendation.

Inspect the output: submitted profiling report · PyTorch profiler summary · submitted spectrogram figure

Audit the scoreboard

Grok's 7.08-second FP32 baseline is real, but belongs to a 15-row short/medium/long run. Its displayed 52-pass, four-length baseline averages 15.80 seconds.

The submitted 1.82× INT8 ratio also reproduces arithmetically, but its FP32 rows omit text labels and audio durations while INT8 rows describe different-duration outputs. Correct arithmetic can answer the wrong question.

OpenVINO's 1.12-second result is not TTS at all: it consumes random dummy tensors, records zero audio duration, and produces no validated speech. It is a backend probe, not a deployable winner.

Inspect the output: submitted timing CSV · submitted comparison tables · claim-by-claim audit

First replay: the headline disappears

The first paired replay used 21 comparisons and found +2.78% median, CI [-0.11%, +4.20%]. That missed the fixed 5% gate.

The result was directionally useful but noisy, especially for long speech. We therefore ran a much stronger protocol twice.

Full retest: deployment shape matters

Each run included:

• 30 balanced FP32-vs-FP32 A/A pairs.
• 60 resident FP32-vs-INT8 pairs.
• 12 process-isolated block pairs.
• Fixed prompts, voice, speed, affinity, seeds, thresholds, and ABBA ordering.
• Two warm-ups and five timed syntheses per isolated process block.

Full retest speed for Grok Linear-only INT8

Cold start did not provide a separate win: median fresh-process totals were 13.31 s for FP32 and 13.34 s for INT8, because quantization preparation offsets the faster first synthesis.

This is the important correction: Grok's optimization idea was not empty. It repeatedly produced a modest warm-inference gain when only one model pipeline was resident. The benefit shrank below the keep threshold when FP32 and INT8 pipelines coexisted.

That is a deployment-sensitive result, not the submitted 1.82× universal win.

Inspect the output: two-run summary · run 1 records · run 2 records · cold-start records

The audio still blocks production

The fixed gate required duration drift ≤2%, log-mel RMSE ≤3 dB, waveform correlation ≥0.30, RMS drift ≤25%, and finite non-silent output.

All 18 of 18 full-retest comparisons failed. Median log-mel error was:

Full retest audio quality for Grok Linear-only INT8

If you already heard samples above, here is the full blind pack. Listen without revealing which files are FP32 or INT8:

Sample 01

Sample 02

Sample 03

Sample 04

Sample 05

Sample 06

Record intelligibility, naturalness, artifacts, and overall quality in the listening score sheet. Only after scoring, open the answer key to identify the FP32 and INT8 samples. Human ratings remain pending.

What the result means

The 51/100 artifact score stays unchanged. Independent evaluator work cannot retroactively improve the methodology of the submitted repository.

Separately, the verified performance assessment is now:

Quality-limited rejection. Linear-only INT8 has a repeatable 6-8% process-isolated latency benefit, only 2-3% resident benefit, and unacceptable objective audio differences.

That is more favorable, and more precise, than saying Grok was simply bad.

What should be tested next?

These are follow-up experiments, not work already completed in this comparison:

1. Find the source of the audio damage. Separate weight-normalization removal from INT8 quantization to see which change breaks fidelity.
2. Quantize selectively. Test small groups of Linear layers and retain only combinations that preserve speech.
3. Strengthen the quality gate. Add intelligibility checks and scored human listening before calling an optimization deployable.
4. Finish the OpenVINO path. Run real text through the complete TTS pipeline instead of timing dummy tensors.
5. Optimize the real hotspot. Explore convolution kernels and Snake/sine fusion, where all three profiles show more potential than Linear layers.

Artifacts map

Technical bottom line

Opus 5 still produced the most trustworthy research artifact (83/100). Grok 4.6 did find a modest real optimization (~6–8% isolated), but its original evidence overstated the gain and its output failed quality validation. Kimi’s and Opus’s thread-based speed claims also did not survive as shippable wins after independent replay.

The lesson is not "Grok is bad." It is that profiling skill, optimization ideas, experimental proof, and production judgment are separate capabilities, and this task rewarded all four.

Try NEO in your IDE

Use NEO with your preferred model keys while keeping the workflow, tools, and evidence in one place:

• VS Code: Install NEO for VS Code
• Cursor: Install NEO for Cursor