Three AI Optimizers Took On Kokoro-82M. The Fastest Claim Lost.

Kokoro-82M turns text into speech. After Kimi K3 and Opus 5 optimized it on CPU through the same Neo Bring Your Own Key (BYOK) workflow, we added Grok 4.6 to the same task.

Kimi K3, Opus 5, and Grok 4.6 found the same CPU bottleneck. None proved that its chosen optimization should ship unchanged.

TL;DR

Opus 5 remains the strongest research artifact at 83/100, ahead of Kimi K3 at 60 and Grok 4.6 at 51.

Grok's score does not mean it had no useful idea. Two full retests found Linear-only INT8 6.18-7.91% faster in process isolation, but only 2.45-3.38% faster with both pipelines resident. Every objective audio comparison failed.

The fair verdict is: a modest deployment-sensitive optimization, weakly justified in the original submission, and not safe to ship as tested.

Three-way study quality scores for Kimi K3, Opus 5, and Grok 4.6

Outcome in one table

This ranks three repositories for one task, not the underlying language models generally.

Why Grok scored low

Grok was good at exploration. It earned 29 of 40 points for profiling and optimization coverage. It profiled MKLDNN operators, measured memory, tested BF16, compilation, threads, caching, dynamic quantization, ONNX, and OpenVINO, and preserved useful failures.

Most lost points came from the trust layer:

• 7/25 for experimental and statistical rigor.
• 5/15 for correctness and audio validation.
• 2/5 for maintainability.

The submitted report mixed workloads, lacked paired confidence gates, ranked dummy OpenVINO inference beside working TTS, used an unsupported perceptual MSE threshold, and recommended output it described as noticeably degraded. That is why the artifact scored 51. It is not a claim that Grok is generally bad at optimization.

Rubric breakdown for the three Kokoro optimization artifacts

Inspect the output: original Grok report · machine-readable scorecard

The profile was useful

All three submissions point to convolution-heavy waveform synthesis.

Hotspot comparison across the three Kokoro CPU profiles

Grok measured MKLDNN convolution at 57.3% of self CPU time, followed by normalization and elementwise work. That independently agrees with the ONNX studies: Kokoro's generator/vocoder is the real optimization surface.

The surprising part is that Linear layers were a small profile component, yet Linear-only dynamic quantization became the production recommendation.

Inspect the output: submitted profiling report · PyTorch profiler summary · submitted spectrogram figure

Audit the scoreboard

Grok's 7.08-second FP32 baseline is real, but belongs to a 15-row short/medium/long run. Its displayed 52-pass, four-length baseline averages 15.80 seconds.

The submitted 1.82× INT8 ratio also reproduces arithmetically, but its FP32 rows omit text labels and audio durations while INT8 rows describe different-duration outputs. Correct arithmetic can answer the wrong question.

OpenVINO's 1.12-second result is not TTS at all: it consumes random dummy tensors, records zero audio duration, and produces no validated speech. It is a backend probe, not a deployable winner.

Inspect the output: submitted timing CSV · submitted comparison tables · claim-by-claim audit

First replay: the headline disappears

The first paired replay used 21 comparisons and found +2.78% median, CI [-0.11%, +4.20%]. That missed the fixed 5% gate.

The result was directionally useful but noisy, especially for long speech. We therefore ran a much stronger protocol twice.

Full retest: deployment shape matters

Each run included:

• 30 balanced FP32-vs-FP32 A/A pairs.
• 60 resident FP32-vs-INT8 pairs.
• 12 process-isolated block pairs.
• Fixed prompts, voice, speed, affinity, seeds, thresholds, and ABBA ordering.
• Two warm-ups and five timed syntheses per isolated process block.

Full retest speed for Grok Linear-only INT8

Cold start did not provide a separate win: median fresh-process totals were 13.31 s for FP32 and 13.34 s for INT8, because quantization preparation offsets the faster first synthesis.

This is the important correction: Grok's optimization idea was not empty. It repeatedly produced a modest warm-inference gain when only one model pipeline was resident. The benefit shrank below the keep threshold when FP32 and INT8 pipelines coexisted.

That is a deployment-sensitive result, not the submitted 1.82× universal win.

Inspect the output: two-run summary · run 1 records · run 2 records · cold-start records

The audio still blocks production

The fixed gate required duration drift ≤2%, log-mel RMSE ≤3 dB, waveform correlation ≥0.30, RMS drift ≤25%, and finite non-silent output.

All 18 of 18 full-retest comparisons failed. Median log-mel error was:

Full retest audio quality for Grok Linear-only INT8

The files are healthy and close in duration, but health is not fidelity. Listen without revealing which files are FP32 or INT8:

Sample 01

Sample 02

Sample 03

Sample 04

Sample 05

Sample 06

Record intelligibility, naturalness, artifacts, and overall quality in the listening score sheet. Only after scoring, open the answer key to identify the FP32 and INT8 samples. Human ratings remain pending.

What the result means

The 51/100 artifact score stays unchanged. Independent evaluator work cannot retroactively improve the methodology of the submitted repository.

Separately, the verified performance assessment is now:

Quality-limited rejection. Linear-only INT8 has a repeatable 6-8% process-isolated latency benefit, only 2-3% resident benefit, and unacceptable objective audio differences.

That is more favorable, and more precise, than saying Grok was simply bad.

What should be tested next?

These are follow-up experiments, not work already completed in this comparison:

1. Find the source of the audio damage. Separate weight-normalization removal from INT8 quantization to see which change breaks fidelity.
2. Quantize selectively. Test small groups of Linear layers and retain only combinations that preserve speech.
3. Strengthen the quality gate. Add intelligibility checks and scored human listening before calling an optimization deployable.
4. Finish the OpenVINO path. Run real text through the complete TTS pipeline instead of timing dummy tensors.
5. Optimize the real hotspot. Explore convolution kernels and Snake/sine fusion, where all three profiles show more potential than Linear layers.

Artifacts map

Bottom line

Opus 5 still produced the most trustworthy research artifact. Grok 4.6 did find a modest real optimization, but its original evidence overstated the gain and its output failed quality validation.

The lesson is not "Grok is bad." It is that profiling skill, optimization ideas, experimental proof, and production judgment are separate capabilities, and this task rewarded all four.

Try NEO in your IDE

Use NEO with your preferred model keys while keeping the workflow, tools, and evidence in one place:

• VS Code: Install NEO for VS Code
• Cursor: Install NEO for Cursor