Kimi K3 vs GLM 5.2 vs Fable 5: Benchmarking AI-Generated ML Engineering

Three AI-generated AutoML frameworks, one rubric, and what a green test suite still cannot tell you

TL;DR

We compared three generated tabular-ML frameworks: Kimi K3, GLM 5.2, and Fable 5. Each claims to cover validation, EDA, preprocessing, training, tuning, evaluation, explainability, tracking, persistence, and serving.

All three native test suites passed: 45/45, 83/83, and 37/37. That is where the easy comparison ends.

Execution found three different trust failures. Kimi's health endpoint returns HTTP 200 with no model loaded. GLM's leakage detector calls pandas' ngroups property as a method and silently swallows the exception. Fable accepts an exact copy of the target as a feature without raising a leakage warning.

Artifact scores: Kimi K3 68/100, Fable 5 61/100, GLM 5.2 54/100. These are scores for three individual generated repositories, not universal rankings of the underlying models. Full evidence is in REPORT.md.

Evidence-backed walkthrough: green suites, adversarial probes, and the production scorecard.

Why this benchmark

Long-horizon engineering is different from generating a function. Existing benchmarks usually ask whether a pre-existing test passes. ML systems fail in a more dangerous way: a leaked feature, an inverted metric, or an inert seed can produce a plausible number without throwing an exception.

We asked three generated repositories to solve the same broad production ML problem, then reviewed them as competing open-source artifacts. The review order was structural inventory → source tracing → native execution → adversarial probes. The goal was not to reward the largest codebase. It was to ask whether each framework can recognize when its own output is untrustworthy.

Methodology

The evidence harness is verify_comparison.py. It records commands, elapsed time, exit status, logs, source/test/doc line counts, and dataset hashes under evidence/.

The native lane runs each repository in its own shipped virtual environment. The adversarial lane probes the behaviors that self-authored happy-path tests often miss: target proxies, readiness without a model, report/API mismatches, unsafe bundle loading, and off-path data shapes.

One fairness caveat matters: Kimi and GLM contain byte-identical Titanic CSVs; Fable's committed CSV has a different SHA-256. Native model metrics are therefore reported as artifact evidence, not as a controlled prediction race.

Objective results

All three suites passed. Runtime is an observation, not a quality score.

The important outputs are not the biggest numbers. They are the behaviors that tell an operator whether a number deserves to be used.

Where each was stronger

Kimi K3: verification discipline. It has a working leakage-drop path, seed propagation, preprocessing persisted with the model, multi-seed benchmarking, and tests that exercise train-only preprocessing and whole-pipeline determinism. Its defects are consequential but comparatively mechanical: multiclass scoring, SMOTE/CV ordering, Docker context hygiene, and readiness semantics.

GLM 5.2: design surface. It has the largest implementation, a typed model registry, configuration structure, broad documentation, and 83 tests. Those strengths are real. They also make the leakage failure more instructive: the framework anticipated the problem architecturally but did not verify the critical path.

Fable 5: coherent pipeline. Its preprocessing is assembled inside the cross-validation pipeline and its model constructors consistently receive a seed. The validator is clear for structural quality checks. It does not implement a target-proxy detector, and its bundle loader treats a caller-selected joblib path as trusted input.

What execution surfaced

GLM: a one-character bug disarms the leakage defense

Titanic's alive column is a perfect yes/no proxy for survived. GLM's validator groups by candidate features and calls grouped.ngroups(). In pandas, ngroups is a property. The call raises TypeError, and except Exception: continue turns the failure into silence.

The committed GLM run can consequently report perfect metrics while the proxy reaches the model. The lesson is not merely "spell the property correctly." Exception scope is a correctness boundary; a broad handler can erase the only signal that a safety check is broken.

Fable: no leakage check means no warning

We added known_proxy = survived to Fable's Titanic input. Its validator returned Validation: PASSED and reported no leakage issue. That is consistent with the source: Fable performs missingness, duplicate, constant-column, and target sanity checks, but has no target-proxy routine.

Kimi: readiness says healthy when prediction cannot work

Kimi's /health probe returned HTTP 200 with status: model_not_loaded for a nonexistent model path. A deployment system that interprets HTTP 200 as readiness can route traffic to a service that will fail every prediction.

All three: passing tests are necessary, not sufficient

The three suites are useful evidence of installability and expected workflow coverage. They do not test every property that matters: proxy rejection, readiness truthfulness, safe artifact boundaries, or behavior outside the binary-classification happy path. This is why the adversarial probes are kept in the repository rather than summarized away.

Full scene sequence: surface read → adversarial probe → decisive defects → scorecard → verdict.

Security and operational boundaries

Fable's bundle loader calls joblib.load on a path selected by the serving configuration. Joblib is pickle-based; model artifacts must be treated as trusted code, and path selection should be allowlisted and contained. The comparison does not claim a remote exploit was executed. It records the unsafe trust boundary for remediation.

Docker image builds were not run because a Docker daemon was unavailable in the environment. Those checks remain explicitly static; no successful build is claimed.

Scorecard

The score is a compact summary of evidence and judgment. It is not a claim that one model family will always produce better software.

Lessons

1. Execute before forming a ranking. A short probe can overturn a long static review.
2. Interrogate outputs against domain reality. Perfect Titanic metrics are a warning sign.
3. Add a known target proxy to every pipeline safety test.
4. Vary configuration knobs; same seed should reproduce, different seeds should move stochastic models.
5. Exercise multiclass, regression, two-candidate selection, and no-model serving paths.
6. Prefer structural guarantees, such as preprocessing inside the fitted pipeline, to comments about ordering.
7. Treat test count as coverage evidence, not correctness evidence.

Which artifact for this task?

For this specific set of generated repositories, Kimi has the highest verified artifact score because its leakage handling and reproducibility machinery are more trustworthy. Fable is a coherent second but needs a real proxy detector and a stricter artifact boundary. GLM has the strongest design surface and the largest test suite, but its silent leakage failure makes its headline ML output unsafe to accept without a separate verification pass.

That is a project-level recommendation, not a universal model ranking.

Limitations

• One generated artifact per model; a second generation could differ materially.
• The committed Titanic files are not all byte-identical.
• Prompt text, temperature, sampling, and agent scaffolding were not captured for byte-level comparison.
• Roughly half the rubric is informed reviewer judgment.
• One reviewer and one primary task class: tabular AutoML.
• The GLM leakage probe records an adapter/API mismatch after printing the proxy crosstab; the source defect and prior committed run remain independently evidenced.

Reproduce

git clone https://github.com/dakshjain-1616/Kimi-k3-Vs-Glm-5.2-.git
cd Kimi-k3-Vs-Glm-5.2-/kimi\ vs\ glm\ vs\ fable
python3 verify_comparison.py
python3 make_gif.py
python3 make_gif_short.py

The complete report, logs, hashes, and machine-readable results are in this folder. The existing two-way comparison remains at the repo root and is covered in our Kimi K3 vs GLM 5.2 post.

Conclusion

Every conventional quality proxy (source volume, test count, documentation, and architectural ambition) can look good while a safety property is broken. The scarce engineering skill is not producing a plausible number. It is knowing which invisible properties to interrogate before that number flows downstream.

In ML systems, the failure that costs you is often not the one that raises. It is the plausible number that nobody stopped to question.

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor