Kimi K3 vs GLM 5.2: Benchmarking Frontier Models on Production ML Engineering

Two AI-generated AutoML frameworks, one rubric, and what test-suite-based evaluation systematically fails to catch.

TL;DR

We gave Kimi K3 and GLM 5.2 the same brief (build a production-ready ML automation framework) via NEO BYOK, then reviewed both like competing open-source projects.

Both shipped working software. Install, train on Titanic, FastAPI, Docker, green suites: 45/45 for Kimi K3, 83/83 for GLM 5.2.

Static review favored GLM. More code, more tests, better docs and abstractions.

Execution inverted the ranking. GLM reports 1.000 on all six Titanic metrics via a leaked alive column; its leakage detector is dead (ngroups() typo swallowed by except Exception); regression selection maximizes RMSE; seed is inert.

Scores: Kimi K3 68/100, GLM 5.2 54/100. Neither is production-ready. This is n=1 per model (two artifacts, not a universal ranking). Full technical review: REPORT.md.

Why this benchmark

Frontier models increasingly do long-horizon ML engineering. Existing evals (HumanEval, SWE-bench, agent suites) mostly ask whether a test that already exists passes. ML pipelines fail differently: leakage, inverted metrics, and inert seeds produce plausible wrong numbers with no exception.

We asked both models to build a full tabular AutoML stack (validate → EDA → features → train → tune → evaluate → explain → serve → deploy), then scored them on a 100-point rubric weighted toward production impact. Review order: structure → source → execution → adversarial probes. Every critical finding below was re-derived by running code.

Objective results

Published Titanic accuracy clusters around 0.80-0.85. Six perfect metrics are a symptom, not a win.

Where each was stronger

GLM 5.2 (design): typed ModelWrapper + registry/factory, YAML config inheritance, per-stage pipeline toggles, feature transforms in a real sklearn Pipeline, stronger CI/Docker DX, richer docs, and a structured pipeline_report.json.

Kimi K3 (verification): a leakage test that would catch its own regression, end-to-end seed determinism checks, preprocessor persisted with the model (no train/serve skew), Optuna pruning, MLflow version shims, non-root container user, and multi-seed variance reporting.

What only execution surfaced

Perfect score = leaked target proxy

Titanic's alive column is survived re-encoded as yes/no. Kimi detected and dropped it. GLM one-hot encoded it as a feature and committed a run report listing alive as active. Result: 1.000 on all six metrics.

Dead leakage detector (one character)

GLM did write a leakage check. It calls grouped.ngroups() (a property, not a method). The TypeError is swallowed by except Exception: continue. The detector has never flagged a column; the leaked run still reports "passed": true.

Takeaway: exception handlers are a correctness property. Scope them so they cannot absorb bugs in your own code. Silent-handler ratio tracked where the silent failures were (Kimi 1/11, GLM 14/49).

Clean-before-split is not always leakage

Dropping IDs or target proxies before the split is a schema decision. Fitting imputation means or IQR bounds before the split is leakage. Kimi's clean stage does the first; GLM's DataCleaner.fit does the second (and can even clip or impute the target before split). Feature engineering after the split looked correct, which made the bug harder to spot in review.

Inverted regression selection

GLM maximizes its primary metric unconditionally. For RMSE (lower is better) that selects and persists the worst model. Missed by tests that only ever compare one candidate.

Inert seed

GLM exposes a seed field and a set_seed() helper whose docstring claims broad coverage. Varying seed 42 vs 999 yields bit-identical RMSE: constructors and Optuna hardcode 42. The missing test is different seeds → different metrics.

Kimi's own failures

Multiclass scoring crashes (roc_auc_score without multi_class). Health returns HTTP 200 with model_not_loaded. .dockerignore lives under docker/, so Docker never sees it. SMOTE is applied before CV folds (the docstring claims otherwise). Higher score does not mean production-ready.

Security

Neither HTTP service is hardened. GLM's worst issue: an unsanitized model_id is joined into a filesystem path and passed to joblib.load (pickle). That is path traversal plus unsafe deserialization if an attacker-writable file is reachable. Prefer an allowlist of known models; never treat request identifiers as filesystem paths.

Scorecard

Categories you can score by reading alone nearly tied. Categories that need execution produced most of the gap (about 9 of the 14 points). Proxies like LOC, test count, docs, and CI favored the artifact with the silent defects.

Trade: Kimi optimized for the system being correct, and under-designed. GLM optimized for the design being correct, and under-verified. Test count is not test value: ask whether a named test would fail if that behavior broke.

Lessons

1. Execute before you review. Ten minutes of running beat hours of reading.
2. Interrogate outputs against domain reality. 1.000 on Titanic is a tell; author-written suites will not catch it.
3. Grep for over-scoped except Exception.
4. Verify config knobs move outputs (same seed → identical; different seeds → different).
5. Test shapes off the happy path (multiclass, continuous targets, at least two models in a comparator).
6. Prefer structural guarantees (one Pipeline fit after the split) over statement ordering.

Which model, for this task

For ML and data pipelines: Kimi K3. Its defects are mechanical and localized (roughly a week of fixes). For scaffolding and architecture patterns: borrow from GLM (factory, config inheritance, stage toggles), then run a separate correctness pass aimed at leakage, metrics, and seeds. With BYOK this is a per-project choice, not a default.

Limitations

This is one generation per model; prompts and sampling settings were not captured for byte-comparison; about half the rubric is judgment; single reviewer; one task class (tabular AutoML). Execution findings are reproducible facts; totals are informed opinion. File-and-line citations and extra repro scripts are in REPORT.md.

Reproduce the headline result

git clone https://github.com/dakshjain-1616/Kimi-k3-Vs-Glm-5.2-.git
cd Kimi-k3-Vs-Glm-5.2-

# set up each package (venv + pip install -e .) per its README, then:

cd kimi-k3 && python -m automl.cli.main train --config configs/titanic.yaml
# → Dropping ID/leakage columns: ['alive'] → roc_auc ~0.92

cd ../glm-5.2 && python -m mlforge train --config configs/examples/titanic.yaml
# → 1.0000 across all six metrics

Conclusion

Every conventional quality proxy pointed the wrong way. GLM anticipated leakage and still shipped a dead detector: the gap was verification, not knowledge. Both models validated one problem shape and broke off it.

In ML systems, the failure that costs you is not the one that raises. It is the plausible number that flows into every decision downstream. As more engineering is generated rather than typed, the scarce skill is knowing which invisible properties to interrogate.

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor