Three local language models answered the same prompts under matched inference conditions. Grade the answers with a mechanical checklist, and one model wins clearly. Grade the same answers with a blind LLM-as-a-judge comparing them side by side, and a different model wins, just as clearly. Both graders can be internally consistent, because they are measuring different things.
This whole bake-off — prompts, matched local FP8 runs, checklist scoring, blind pairwise judging, charts, and writeup — was built and run end to end by Neo, an autonomous AI engineering agent.
Benchmark at a glance
Models:
Qwen/Qwen3.6-27B-FP8,Qwen/Qwen3.8-27B-FP8,RedHatAI/Muse-Glimmer-30B-FP8-blockSource prompts: 15 across coding, instruction following, reasoning, and system design Comparable answer instances: 38 (from 11 of those 15 prompts) Deterministic checks: 23 instances with mechanically verifiable constraints Blind pairwise judge calls: 228 (38 × 3 pairings × 2 orders) Resolved pairwise outcomes: 114 Judge order-flip rate: 12.3% Blind-judge cost: $0.5699 Host: NVIDIA A40 45 GB · vLLM · matched FP8
Here's what you'll find below:
- Why one LLM benchmark produced two contradictory winners instead of one
- The two ways we graded the same outputs: deterministic evaluation and blind pairwise evaluation
- Named examples of the coding, instruction-following, reasoning, and system-design tasks
- The actual numbers, category by category, with the charts to match
- What it cost, in real dollars, to run an LLM-as-a-judge on every answer
- Which model to reach for, depending on what "better" means to you
Two kinds of grading
Picture two ways a teacher can grade an exam.
The first is a multiple-choice test. Every answer is either right or wrong against a fixed key. No judgment calls, no partial credit for style. Grading is boring and completely unambiguous.
The second is an essay, graded by a reader who does not know whose essay they are holding. The reader compares two essays side by side and picks the stronger one. Each pair gets graded twice, once with the essays in each order. A habit like always preferring the first essay read would then show up as a contradiction, not a real preference.
This benchmark ran both kinds of grading on the same three models: Qwen3.6-27B, Qwen3.8-27B, and Muse Glimmer 30B.
The multiple-choice test is a set of mechanical checks: does the code parse, does it respect a word limit, does it avoid a banned word. That is deterministic constraint satisfaction. The blind essay reader is a large language model (LLM) used as a judge. It sees two anonymous answers and picks the one it prefers under its evaluation rubric. Preference is not the same thing as objective correctness.
Keep that picture in mind. The rest of this post is what happens when the checklist grader and the blind evaluator pick different winners.
Same evaluation environment, same prompts, different models
Comparing language models fairly is harder than it sounds. A hosted service might run one model at full precision and another at a heavily compressed one. It then reports the gap as a difference in ability, when it's really a difference in setup.
This comparison holds the relevant inference conditions constant. All three models ran on one NVIDIA A40 (45 GB), served with vLLM, at matched FP8 precision. Same sampling configuration (temperature=0.6, top_p=0.95, top_k=20), same system prompt (You are a helpful AI assistant.), same 15 source prompts across four categories: coding, instruction following, reasoning, and system design.
| Model | Hugging Face checkpoint | Quantization |
|---|---|---|
| Qwen3.6-27B | Qwen/Qwen3.6-27B-FP8 | Official first-party fine-grained FP8 (block size 128), from Qwen/Qwen3.6-27B |
| Qwen3.8-27B | Qwen/Qwen3.8-27B-FP8 | Official first-party fine-grained FP8 (block size 128), from Qwen/Qwen3.8-27B |
| Muse Glimmer 30B | RedHatAI/Muse-Glimmer-30B-FP8-block | LLM Compressor FP8_BLOCK (weights 128×128 blocks; activations dynamic, group 128), from meta-models/Muse-Glimmer-30B |
That is not a claim that every implementation detail was identical. The two Qwen checkpoints share an official first-party FP8 recipe. Muse Glimmer 30B uses a different FP8 recipe (llm-compressor block). Qwen-versus-Qwen is clean on compression. Any claim that crosses model families carries that asterisk. On this A40, vLLM's FP8 path falls back to W8A16 via FP8 Marlin (weights stay FP8; compute runs bf16). All three models take that same path, so relative comparisons stay fair.
The underlying experiment generated more than 38 answers. Each of the 15 prompts was asked up to five times. Some runs failed, and a handful came back empty. Counting a missing or empty answer as a loss would have unfairly penalized whichever model that happened to.
So the comparison uses only the (prompt, repetition) pairs where all three models produced a real, non-empty answer. That filtering produced 38 comparable answer instances, spanning 11 of the 15 source prompts. Those 38 are the shared, apples-to-apples set. They are not 38 distinct questions, and they are not the full generation count.
Checkpoints and serving config, plus the scripts that rebuild this set, are in the GitHub repository.
What the models were asked to do
Abstract category labels are easy to skim past. The actual prompts are concrete engineering and reasoning work. All fifteen live in prompts/benchmark_prompts.json in the repo. Here is what each category meant in practice.
Coding (5 prompts)
These ask for working Python under real constraints: type hints, retries, tests, and production-shaped APIs.
| ID | Task | What “good” looks like |
|---|---|---|
coding_01 | Multi-step customer-support agent | A SupportAgent class that classifies tickets, checks subscription, searches a knowledge base, drafts a reply, and escalates when confidence is low — with retries, logging, and type hints |
coding_02 | Code review of a pandas pipeline | Find bugs, performance issues, and style problems in transaction-processing code; severity-rank each issue and show a corrected version |
coding_03 | Refactor a payment monolith | Break a duplicated credit-card / PayPal / crypto handler into testable pieces (Strategy / Factory / Repository where they fit) plus a small pytest-style suite |
coding_04 | Token-bucket rate limiter | Thread-safe multi-rule limiter with X-RateLimit headers, a route decorator, and a swappable backend abstraction |
coding_05 | Async database-client test suite | pytest-asyncio coverage for connect, execute, transactions, timeouts, concurrency, and cleanup — meaningful assertions, not “didn’t crash” |
A mechanical check on several of these is simply: does the Python parse? The blind judge scores design quality, error handling, and completeness under a rubric.
Instruction following (2 prompts)
These are trapdoors for models that write fluently but ignore format rules.
| ID | Task | Hard constraints (examples) |
|---|---|---|
instruction_01 | LSTM vs Transformer write-up | Start with exactly ## Response, then three paragraphs of 3 / 5 / 4 sentences; ban “additionally / furthermore / moreover”; every sentence must include one numeric fact |
instruction_02 | CAP theorem plus a movie question | No bullet lists; ban “however” and “significant”; every paragraph under 60 words and ending in a question; refuse the non-technical movie ask with a fixed phrase |
This is where the checklist grader is strongest: starts-with, banned words, paragraph and sentence counts, and required refusal text can be verified without an opinion.
Reasoning (3 prompts)
These reward careful multi-step work, not vibes.
| ID | Task | What the model has to do |
|---|---|---|
reasoning_01 | Train meeting-time problem | Solve a relative-speed meeting time, then a harder variant with a delayed second train and a 15-minute stop; verify with a second method |
reasoning_02 | Friends / colors / pets logic puzzle | Assign five people to colors and pets from ten clues using constraint propagation, not brute force |
reasoning_03 | City delivery routing | Plan trips for eight packages under capacity, cold-storage windows, and working hours; report distance, time, and feasibility |
Some reasoning prompts have no mechanical checklist. Preference then rests almost entirely on the blind judge’s rubric for correctness, clarity, and constraint handling.
System design (5 prompts)
These mix implementation, architecture judgment, and debugging narrative.
| ID | Task | Shape of the answer |
|---|---|---|
sysdes_01 | Dark analytics dashboard from a text brief | One self-contained HTML/CSS/JS file: sidebar, stat cards, chart placeholder, responsive table — no external libraries |
sysdes_02 | Real-time event pipeline | An asyncio EventPipeline for ~10K events/s: validate, route, dedup, enrich, batch-write, dead-letter |
sysdes_03 | Architecture from a file tree | Name the pattern, dependency directions, where to add document signing, likely stack, testing strategy, and gaps |
sysdes_04 | Intermittent segfault diagnosis | Root-cause a pybind11 / GIL / memory-growth crash from a stack trace, then propose monitoring and regression tests |
sysdes_05 | S3 compliance CLI | A boto3-stubbed Python CLI that audits public access, encryption, versioning, lifecycle, and object lock into a Markdown report |
Taken together, the suite is not a trivia quiz. It is a short, opinionated sample of agent-style coding work, format obedience, multi-step reasoning, and systems thinking — the kinds of tasks local 27B–30B models are often asked to do.
The checklist: who follows the rules
On the mechanical checklist, Muse Glimmer 30B wins.
The checklist applies to 23 of the 38 comparable instances: the ones with at least one fixed, checkable requirement. A strict pass means every applicable mechanical condition was satisfied. Examples include: the code parses, a word-count limit is met, a prohibited term is absent, a required format is respected. Fail any one of them and the answer fails. These checks measure mechanically verifiable compliance, not general intelligence or overall answer quality.
| Model | Strict pass rate | Answers checked |
|---|---|---|
| Muse Glimmer 30B | 78.3% | 23 |
| Qwen3.8-27B | 69.6% | 23 |
| Qwen3.6-27B | 65.2% | 23 |
Deterministic strict pass rate by model, with 95% bootstrap confidence intervals
Split by category, the picture gets more specific. On coding and system design prompts with checks, both Qwen models hit 100%, while Glimmer lands closer to half on system design. On instruction-following prompts, ones that say "stay under this word count" or "use this exact format," Glimmer leads and Qwen3.6-27B drops to 20%. Reasoning prompts, and one system-design prompt, have no deterministic checks by design; they are scored only by the judge.
| Category | Muse Glimmer 30B | Qwen3.8-27B | Qwen3.6-27B |
|---|---|---|---|
| Coding | 100% | 77.8% | 100% |
| System design | 50% | 100% | 100% |
| Instruction following | 70% | 50% | 20% |
Deterministic strict pass rate by model and category
Twenty-three checked instances split three ways is a small sample. Direct pass-fail comparisons between any two models, on the same instance, come back tied more often than not, and none of those head-to-heads reach statistical significance on this set. Read the checklist percentages as a real directional signal on this benchmark, not as a population-level ranking.
The blind reader: who the evaluator prefers
On blind pairwise LLM evaluation, Qwen3.8-27B wins, and it isn't close.
The judge was openai/gpt-5.6-luna-pro, blind to model identity. Every pair of models was compared on every shared instance, twice each, once in each presentation order:
38 comparable instances × 3 model pairings × 2 orders = 228 judge calls.
The two orders for each pair were then reconciled into one resolved pairwise outcome, so 228 / 2 = 114 resolved verdicts. A model's overall strength comes from a Bradley-Terry model. It fits latent relative-strength parameters that best explain the full set of pairwise outcomes at once, instead of just counting wins. Those parameters sum to 1. They are not accuracy percentages, and they are not a claim of objective correctness.
| Matchup | Model A wins | Model B wins | Ties | Preferred |
|---|---|---|---|---|
| Qwen3.6-27B vs Qwen3.8-27B | 1 | 29 | 8 | Qwen3.8-27B |
| Qwen3.6-27B vs Muse Glimmer 30B | 24 | 10 | 4 | Qwen3.6-27B |
| Qwen3.8-27B vs Muse Glimmer 30B | 35 | 1 | 2 | Qwen3.8-27B |
Take that last row as a worked example. Qwen3.8-27B and Muse Glimmer 30B answered the same instances, and the blind evaluator preferred Qwen3.8-27B's answer in 35 of 38 resolved comparisons. That is not a coin flip landing one way. It is a consistent preference under this judge and rubric for answers the evaluator judged stronger, not an independent proof that those answers were objectively more correct.
The fitted Bradley-Terry strengths: Qwen3.8-27B at 0.939, Qwen3.6-27B at 0.043, Muse Glimmer 30B at 0.019. Qwen3.8-27B dominates this pairwise ranking. Remember the sample: 38 comparable instances. An extreme relative-strength score here is still a result on this set, not universal model superiority.
Blind evaluator preference by model and category
The evaluator also flagged its own order sensitivity. Each pair was judged twice with answer order reversed. In 12.3% of those paired evaluations, the outcome changed when only the order changed. That is position sensitivity, not a direct measure of "judge error," because a flip does not tell us which of the two judgments was right. It is why every comparison ran in both orders instead of one.
Running the judge wasn't free. Judging all 228 calls cost $0.5699, under a dollar for a full round of blind pairwise evaluation across three models.
Why the two graders disagree
The mismatch lines up with one measured difference: how much each model writes.
Qwen3.8-27B's answers averaged 13,579 tokens. Qwen3.6-27B averaged 6,297. Muse Glimmer 30B averaged 3,066, less than a quarter of Qwen3.8-27B's length. All three generated at a similar speed, about 16 to 17 tokens per second.
A checklist doesn't care how long an answer is, unless length causes a constraint violation. It cares whether the answer fits inside a word limit or avoids a specific word. A short, disciplined answer can score well. A long one risks tripping a rule.
A blind LLM judge compares two full answers side by side. It may prefer the one that covers more ground and reads as more thorough. That is exactly what a longer answer can do.
Neither grader is wrong about what it measures. They just weight verbosity differently.
The benchmark observes a strong association between output length and evaluator preference, but it does not isolate length as a causal variable. We did not run a length-controlled evaluation. The next experiment would be to control for answer length and rerun the blind pairwise evaluation, to help separate judged quality from a preference for verbosity.
One more reading note before the takeaway: treat the category breakdowns as directional on this prompt set. The sample is small by design — a focused local bake-off, not a leaderboard that settles the field — and the blind ranking comes from a single judge model. What the run does show clearly is the split itself: checklist obedience and side-by-side preference are not the same contest.
The takeaway
There is no single best model here, because "better" was never one thing to measure.
On this benchmark's mechanically checkable constraints, Muse Glimmer 30B performed best. On this benchmark's blind LLM-as-a-judge pairwise evaluation, Qwen3.8-27B was strongly preferred. Those are different evaluation objectives on the same answers — support-agent code, format traps, logic puzzles, dashboards, pipelines — not a claim that one model wins every workload forever.
If you need a hard constraint followed exactly — an exact word count, a banned term, code that has to parse — Glimmer is the checklist leader here. If you need the answer this blind evaluator judged as stronger side by side, Qwen3.8-27B is the pairwise leader here. Read the category tables and the task examples above, then pick the grader that matches the job you actually care about. The lasting lesson is about LLM evaluation methodology: the winner depends on how you grade.
Try NEO in Your IDE
Install the NEO extension to bring AI-powered development directly into your workflow:
- VS Code: NEO in VS Code
- Cursor: Install NEO for Cursor