Three ASR models tied on the benchmark. One was almost three times worse on real speech.

Performed using NEO AI Engineer

On clean read speech, Wav2Vec2, Distil-Whisper, and Whisper all hit 4.4% WER. On accented conversational English, Wav2Vec2 jumps to 56.1% while Whisper stays near 20%. Same models, same leaderboard-looking baseline, very different products.

This is a measured comparison of five ASR models on 296 real clips across 19 conditions. Transcripts are committed, the tables are computed from those transcripts, and CI recomputes every published number from raw output. Skim the clean-vs-accented table next if you only need the punchline. Keep reading for the audio samples, full results, and how to verify them yourself.

What's in this post

• The finding: a clean-speech three-way tie that breaks on accents, babble, and music
• Real audio: the same evaluation clips, with raw model hypotheses next to them
• Full results: per-scenario WER with 95% CIs, paired degradation costs, failure profiles, and CPU vs CUDA speed
• Coverage: LibriSpeech, MUSAN, EdAcc, and real codecs
• Verification: check_regression.py recomputes the tables from committed transcripts, with no model weights required

Architecture

The problem with a clean-speech leaderboard

Public ASR leaderboards rank models on LibriSpeech test-clean: voice actors reading public domain books into a good microphone in a quiet room. It is a legitimate benchmark and the numbers are real. It is also almost nothing like the audio most production systems actually receive.

Here is what that gap looks like in practice. Three of the models in this benchmark score exactly the same word error rate on clean read speech: 4.4% each. Wav2Vec2, Distil-Whisper and Whisper, indistinguishable, sitting well inside one another's confidence intervals. On the condition the leaderboards measure, there is nothing to choose between them.

Now play spontaneous conversational English from speakers whose first language is not English:

A three-way tie on the leaderboard condition, and on accented speech one of them makes almost three times as many errors as the others. More than half the words wrong, against about a fifth. Nothing in the clean number told you which model was which.

That is the whole argument for this benchmark. The interesting question is not which model wins on clean audio, because they mostly draw. It is how far each one falls when the audio degrades, and whether it falls in a way you can detect from the output.

Building audio that actually breaks things

The evaluation set is 296 clips, about 36 minutes of 16 kHz mono audio, committed to the repository at data/clips/ with a SHA-256 for every file.

Every clip is real human speech from a public corpus, and every degradation is a real signal or codec operation applied to that speech. There is no text-to-speech anywhere in the project, no simulated noise, and no filter pretending to be a telephone line. This matters because a synthetic "phone quality" low-pass filter and an actual GSM encoder do very different things to a consonant.

The 19 scenarios fall into four groups.

Untouched recordings cover clean read speech, naturally harder recordings, accented conversational speech, utterances under 2.5 seconds where a decoder has almost no context to work with, and long-form streams over 90 seconds.

Acoustic degradations are real recorded noise mixed into the speech: office ambience at 10 dB SNR, street noise at 5 dB, six-talker babble at 10 dB, music at 5 dB, and a second real speaker overlapped at 0 dB. Plus two electrical problems that show up constantly in the field, a 25 dB attenuated signal from a badly gained microphone, and hard clipping from an overdriven one.

Codec round trips push the audio through real encoders with ffmpeg and decode it back: G.711 mu-law at 8 kHz for the landline path, GSM 06.10 full-rate for 2G mobile, G.726 ADPCM at 16 kbps for DECT cordless phones, Opus at 6 kbps for a constrained VoIP link, and MP3 at 24 kbps.

Time scaling at 1.5x and 0.75x, with pitch preserved.

Two design decisions do most of the work here.

The first is that the set is paired. Every LibriSpeech-derived scenario reuses the same 16 base utterances. Same speakers, same sentences, same recordings. Only the degradation changes. So when a codec costs a model six percentage points, that cost belongs to the codec and not to the luck of which clips got drawn. It also means the confidence interval on each degradation can be computed pairwise, which cancels out utterance difficulty and makes an effect visible with far fewer clips than an unpaired comparison would need.

The second is that noise is mixed against speech-active frames, not whole-file RMS. If a clip opens with half a second of silence, targeting an SNR against the average level of the entire file quietly gives you a cleaner mix than the label claims. The mixer measures the speech level only over frames within 40 dB of the loudest frame.

The five models

None of these are gated, so reproducing the run needs no credentials. All five decode greedily with num_beams=1, which keeps the comparison about the models rather than about who got a bigger search budget. Wav2Vec2 runs without an external language model. That is the honest baseline for this checkpoint. Adding a 4-gram LM would cut its error rate noticeably, but it would also be a different system, and quietly giving one model a decoding advantage is how benchmarks start lying.

Canary needs a separate virtualenv, because NVIDIA NeMo pins transformers 4.53 and numpy 1.26 while everything else here runs 4.57 and 2.2. Rather than force a version fight, the project keeps two environments and records which one produced which number.

All five are measured. Granite is a 3B-parameter speech LLM that needed a GPU: on the original CPU box it exhausted memory after one clip, so that partial run was deleted rather than published as a number. It has since been scored on CUDA (296 clips in results/raw/Granite/), and a full five-model CUDA speed table lives under results-gpu/. DEFERRED in models/__init__.py is empty.

Hear what the models heard

These are real evaluation clips from data/clips/, the same files scored in the tables above. The transcripts under them come from committed raw output in results/raw/, not from hand-written examples.

Same sentence, four conditions

Reference: Sister Nell, do you hear these marvels? (LibriSpeech clean_read/000 and its paired degradations)

Clean read

Six-talker babble at 10 dB

Music at 5 dB

GSM full-rate phone codec

Whisper's raw hypothesis on all four of those clips was the same readable sentence. The audio still changes a lot. That is why the framework keeps the clips, not just the scores.

Where the models actually diverge

Accented conversational English (accented_english/000 from EdAcc):

That gap is the story from the opening table: a clean-speech tie does not survive real conversational speech.

Babble makes the same point on a short LibriSpeech line (babble_10db/002):

Every hypothesis above is copied from the committed JSONL for that model and scenario. Open any file under results/raw/ to check another clip the same way.

The measured results

Pooled word error rate is computed across every scenario, so it is a robustness figure and is not comparable to a LibriSpeech leaderboard number. Real-time factor is device-dependent (see the RTF column header and per-model hardware in the report). Both are explained above.

Word error rate by scenario

Bracketed figures are the 95% bootstrap confidence interval over clips. They are wide because each cell holds at most 16 clips, and a gap smaller than the interval is not a finding.

Paired degradation cost vs clean_read

Same utterances and speakers; only the degradation changes. Values are percentage-point deltas. Bold means the paired 95% CI excludes zero.

Full tables, paired degradation costs, the failure profile per scenario, and the exact hardware and package versions each run used are in results/REPORT.md. The per-clip transcripts behind every cell are in results/raw/. Machine and package pins for every run are in results/run_metadata.json. The machine-readable aggregate is results/summary.json. Like-for-like CUDA speed numbers live under results-gpu/.

What a WER column will never tell you

A word error rate is one number covering three very different failure modes, and in production they are not equally bad. A model that drops a word is annoying. A model that invents a fluent sentence that was never spoken is dangerous, especially if something downstream is going to act on it.

So the report also tracks what the errors look like. It counts clips where the output is empty, clips where the hypothesis runs past 1.5 times the length of the reference (a decoder that has started looping), and clips under half the reference length (a decoder that gave up). It also lists the most frequent word confusions per scenario, which is usually the fastest way to understand what a model is actually mishearing.

The overlapping-speaker scenario is where this pays off. Every model is above 60% word error rate there, and Wav2Vec2 goes over 100%. That is not a bug in the arithmetic. WER counts insertions, so a model that transcribes both the target speaker and the interfering one can exceed 100% while producing perfectly plausible English. Five of those clips tripped the looping detector.

On the overlapping-speaker clips, the failure profile looks like this (from results/REPORT.md):

The error rate alone would have shown a bad score; the failure profile shows why, and that difference decides whether you can build a confidence threshold on top of the model or not.

The same effect shows up in babble, where two clips ran long. If you only watched the WER column you would read that as "noisy audio is harder", which is true but not actionable. What is actionable is that the model starts transcribing the wrong voice.

Speed, and why the number needs an asterisk

Real-time factor is inference time divided by audio duration. Below 1.0 means the model transcribes faster than the audio plays.

The catch is that RTF is a property of the model and the machine it ran on, which makes a bare RTF figure close to meaningless. Four of the five models in the headline table above were timed on CPU, four cores of an Intel Xeon Platinum 8272CL at 2.60 GHz with 16 GB of RAM. Granite was timed on CUDA (NVIDIA RTX 6000 Ada), because the same CPU box exhausted memory after one clip. Those RTFs are therefore not ranked against each other in one column: the generated table labels the device on every row, and a separate five-model CUDA sweep under results-gpu/ is the like-for-like speed table.

So compare CPU RTFs against other CPU RTFs, and CUDA RTFs against other CUDA RTFs. Do not compare either against figures measured somewhere else. To make that possible, every run writes a snapshot of its own hardware, Python version, ffmpeg build, git commit and the pinned version of every package that can move a result (torch, transformers, jiwer, numpy, nemo_toolkit and the rest) into results/run_metadata.json. The report prints it directly above the speed table.

On the CPU box the gap is large enough to matter. Wav2Vec2 runs at 0.13x RTF, so nearly eight minutes of audio per minute of compute on those cores. Distil-Whisper sits at almost exactly 1.0x on the same cores, roughly eight times slower, in exchange for the robustness advantage above. That is a real architectural tradeoff rather than a ranking, and which side of it you want depends entirely on whether your audio looks like the clean row or the accented row.

CUDA speed table (like-for-like)

From results-gpu/, regenerated into the report. Compare these RTFs to each other, not to the CPU column above.

One number in that table looks wrong

Distil-Whisper is the smaller, distilled model, and it comes out slower than the Whisper checkpoint it was distilled from: 1.0x against 0.59x. That deserves an explanation rather than a shrug.

Almost all of the gap is a single scenario. On clips under 30 seconds the two models are close, 0.83x against 0.73x. On the long-form streams they are not: 1.30x against 0.35x. Distil-Whisper's accuracy drops on those clips too, 6.3% word error rate against Whisper's 1.9%.

The cause is a harness decision, not a property of the model. Every encoder-decoder adapter here uses Whisper's own sequential long-form algorithm so that all of them get the same treatment. Distil-Whisper's model card recommends chunked long-form instead, with a 15 second chunk, and running it the way it was designed to run would very likely close most of that gap. It is reported this way because giving one model a different decoding strategy would quietly hand it an advantage the others do not get.

Worth sitting with, though, because long-form is 36% of the audio in this set, and that one scenario is enough to invert the pooled ranking of two models across the whole benchmark. Pooled metrics hide where the cost is. That is an argument for reading the per-scenario table rather than the headline, and it applies just as much to the headline numbers in this post.

Check the numbers yourself

You do not need a GPU or model weights to verify the published tables. Clone the repo and recompute every figure from the committed transcripts:

git clone https://github.com/dakshjain-1616/Asr-Evaluation.git
cd Asr-Evaluation
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python scripts/check_regression.py

That recomputes WER from results/raw/, checks audio hashes against the manifest, and fails if a published number moved. Full report, hardware pins, and failure profiles live in results/REPORT.md and results/run_metadata.json.

To rerun models or score your own audio, see Replicate or extend this with NEO below.

The part that keeps it honest

Published numbers only count if a machine can recompute them. That check runs in CI:

1. Unit tests over the metric code, the text normaliser, the scenario registry, the resume logic, and the regression gate itself.
2. A reproducibility gate that validates the results file against a JSON schema, recomputes every number from the raw transcripts, re-hashes every committed audio clip, then regenerates the report, the summary, the scenario matrix and the README's own results table and diffs them against what is committed. Hand-editing any published figure turns the build red.
3. A smoke evaluation that actually loads a model, transcribes three clips, and compares the result against the committed transcripts for those same clips. A dependency upgrade that silently changes decoding shows up here rather than six months later.

When an adapter or a dependency pin changes, the same gate compares a fresh run against the committed one and fails if any model and scenario got worse by more than two percentage points, or if speed regressed by more than a quarter:

python run_evaluation.py --output-path /tmp/new --no-resume
python scripts/analyze.py --results-path /tmp/new
python scripts/check_regression.py --results-path /tmp/new \
    --baseline results/summary.json

One detail worth stealing regardless of what you think of the rest: the results tables in the README, in the report, and in this post are generated, and CI regenerates them and diffs. Even the specific figures quoted in the paragraphs above are pinned by a test that checks each one against the measured data and fails if it drifts. There is no path by which the prose and the numbers come apart, because the prose does not contain hand-maintained numbers.

A note on how the numbers are computed

Two choices here change the results enough to be worth stating.

Word error rate is computed over the corpus, not averaged across utterances. Errors and reference words are summed across the whole set and divided once. Averaging per-utterance rates sounds equivalent and is not: one wrong word in a three-word clip is a 33% rate, and averaging lets short clips dominate a figure that nobody intends to be about short clips.

And every rate carries a 95% bootstrap confidence interval, resampling whole utterances, 10,000 times, from a fixed seed. At most sixteen clips per scenario is a small sample and the intervals are correspondingly wide. That width is the point rather than an embarrassment. It tells you which gaps in the table are real findings and which are noise. The accented-speech gap between Wav2Vec2 and the two Whisper models is far outside every interval involved. Most of the codec differences are not, and the report does not pretend otherwise: degradation costs are printed in bold only when the paired interval excludes zero.

All hypotheses and references pass through the same text normaliser before scoring, the standard Whisper English normaliser used by the Open ASR Leaderboard. Without it, Wav2Vec2's uppercase unpunctuated output would be penalised against Whisper's cased and punctuated text, and you would be measuring formatting instead of recognition.

Run it on your own audio

None of this is tied to LibriSpeech, and honestly that is the recommendation. Point the runner at a folder of 16 kHz mono clips plus a manifest in the same shape and it works the same way:

python run_evaluation.py --data-path ./my_data --output-path ./my_results
python scripts/analyze.py --data-path ./my_data --results-path ./my_results

Thirty of your own clips will tell you more about which model to ship than any benchmark someone else published, including this one. The value of what is written above is mostly in the method: pair your degradations against a clean baseline, put intervals on everything, keep the raw transcripts, and let a machine check that your published numbers still match them.

The clean-speech tie was real. So was the two-to-one gap hiding behind it.

Replicate or extend this with NEO

This case study was produced end to end with Neo. Clone the repo, verify the published numbers without loading any model weights, then open the folder in VS Code or Cursor and ask Neo for the next experiment.

1. Verify the published numbers (no GPU, no weights)

git clone https://github.com/dakshjain-1616/Asr-Evaluation.git
cd Asr-Evaluation
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python scripts/check_regression.py

2. Rerun models or point at your own audio

pip install -r requirements-models.txt
python run_evaluation.py
python scripts/analyze.py

# Or score your own 16 kHz mono clips:
python run_evaluation.py --data-path ./my_data --output-path ./my_results
python scripts/analyze.py --data-path ./my_data --results-path ./my_results

3. Drive the next change with Neo

Build with NEO →

Or open the cloned repo with Neo and ask for a concrete next step: add a sixth model adapter, extend the codec set, tighten the regression thresholds, or rebuild the paired set from a corpus you care about. Keep the rule that published tables are generated from committed transcripts and that CI fails if they drift.

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor