Gemma-4-26B vs Qwen 3.5 27B: Head-to-Head Benchmark on Real-World Tasks
The Problem
Choosing between two similarly-sized open-weight models is usually a gut call based on leaderboard numbers that do not reflect your actual workload. Marketing benchmarks measure what model makers optimize for, not what you need.
NEO built this benchmark suite to give developers a reproducible, task-specific comparison between Gemma-4-26B MoE and Qwen 3.5 27B across the task categories that matter for real applications.
Benchmark Design and Task Coverage
The benchmark covers five categories: coding, multi-step reasoning, instruction following, math, and multilingual generation. Each category uses a standardized prompt set drawn from real-world use cases rather than academic datasets code prompts ask for runnable implementations, math prompts require intermediate reasoning steps shown, and multilingual prompts are evaluated by native speakers using reference translations.
Every prompt runs five times per model to account for sampling variance. Results are scored automatically: code is executed and checked against expected outputs, math answers are verified symbolically, and instruction-following responses are scored by a separate LLM judge on a 1-5 rubric. The harness writes raw outputs and scores to results/ partitioned by model and category, so individual runs are always reproducible.
categories/
coding/ # 40 prompts, execution-verified
reasoning/ # 35 prompts, chain-of-thought required
instruction/ # 30 prompts, LLM-judge scored
math/ # 35 prompts, symbolic verification
multilingual/ # 30 prompts, 6 languages
Scoring, Statistics, and Cost Accounting
Raw scores feed into a statistical significance layer built on the Mann-Whitney U test. Each category comparison reports a p-value alongside the score gap if p > 0.05, the gap is flagged as not statistically meaningful, preventing over-interpretation of small differences across 5 runs.
The benchmark also tracks cost-per-correct-answer: total API tokens consumed divided by the number of passing responses. This surfaces an important tradeoff that aggregate accuracy hides. A model that scores 82% on coding at $0.003 per correct answer may be a better fit than one scoring 85% at $0.012 per correct answer, depending on call volume.
| Category | Gemma-4-26B | Qwen 3.5 27B | p-value | Winner |
|---|---|---|---|---|
| Coding | 81.4% | 84.2% | 0.03 | Qwen 3.5 27B |
| Reasoning | 76.8% | 74.1% | 0.09 | No sig. diff |
| Instruction | 88.0% | 85.5% | 0.01 | Gemma-4-26B |
| Math | 79.3% | 82.6% | 0.02 | Qwen 3.5 27B |
| Multilingual | 83.1% | 71.4% | <0.001 | Gemma-4-26B |
Radar Chart Output and Per-Category Reports
After all runs complete, the suite generates a radar chart (report/radar.png) plotting normalized scores for both models across all five axes. A separate per-category breakdown in report/breakdown.html shows score distributions as box plots with individual run points overlaid, making outlier runs immediately visible.
The report also includes a recommendation section: given a hardware and cost profile entered at run time, the suite recommends which model to deploy for each category based on score, statistical confidence, and cost-per-correct-answer. If the models are not significantly different on a category, the recommendation defaults to whichever is cheaper to run.
How to Build This with NEO
Open NEO in VS Code or Cursor and describe what you want to build. A good starting prompt for this project:
"Build a head-to-head LLM benchmark suite in Python that compares two models across coding, reasoning, instruction following, math, and multilingual tasks. Run each prompt 5 times per model, score results automatically using execution verification and LLM judges, apply Mann-Whitney U statistical significance testing, track cost-per-correct-answer, and generate a radar chart and HTML breakdown report."
NEO generates the project structure and core implementation. From there you iterate ask it to add new task categories, extend the LLM judge with a custom rubric, or build a CI integration that reruns the suite on each model release. Each request builds on what's already there.
To run the finished project:
git clone https://github.com/dakshjain-1616/Gemma-4-26B-MoE-vs-Qwen-3.5-27B
cd Gemma-4-26B-MoE-vs-Qwen-3.5-27B
pip install -r requirements.txt
python benchmark.py --models gemma-4-26b qwen-3.5-27b --runs 5
The suite runs all five categories, writes results to results/, and produces the radar chart and HTML report in report/.
NEO built a reproducible head-to-head benchmark that measures Gemma-4-26B MoE and Qwen 3.5 27B on real-world task categories with statistical significance testing and cost-per-correct-answer accounting. See what else NEO ships at heyneo.com.
Try NEO in Your IDE
Install the NEO extension to bring AI-powered development directly into your workflow:
- VS Code: NEO in VS Code
- Cursor: Install NEO for Cursor →