Q-Heatmap: Visualizing Quantization Error Across Model Layers

The Problem

Quantizing a model to 4-bit cuts memory in half, but uniform quantization silently destroys accuracy in a handful of sensitive layers while leaving most layers nearly lossless.

NEO built Q-Heatmap to identify exactly which layers cannot survive aggressive compression and which can be pushed further without measurable degradation.

Per-Layer Error Metrics

Q-Heatmap loads both the full-precision model and its quantized counterpart and computes three complementary error signals for each layer. This multi-metric approach matters because different layers fail in different ways — some show large weight distance but survive functionally, while others show small weight shift but produce dramatically different activations.

The three metrics computed per layer are:

• Weight distance (L2): Frobenius norm of the difference between the full-precision and dequantized weight matrices, normalized by weight magnitude.
• Activation MSE: Mean squared error between full-precision and quantized activations on a 512-sample calibration set drawn from the model's training distribution.
• KL divergence: KL divergence between the full-precision and quantized output distributions on calibration inputs, measured at the layer's output logits before the next layer's transformation.

from q_heatmap import QuantizationAuditor

auditor = QuantizationAuditor(
    fp16_model="meta-llama/Llama-3.1-8B",
    quantized_model="./models/llama-3.1-8b-Q4_K_M.gguf",
    format="gguf",
    calibration_data="./calib/c4_512samples.jsonl"
)

results = auditor.compute_layer_errors()
auditor.render_heatmap(results, output="heatmap.html")

Interactive Heatmap Rendering

The heatmap renders as a Plotly HTML file with layers on the Y-axis and bit-depth configurations on the X-axis (Q2, Q3, Q4, Q5, Q6, Q8). Each cell is colored by the composite error score — the weighted average of normalized weight distance, activation MSE, and KL divergence. Hovering a cell shows the raw values for all three metrics.

Typical output reveals a clear pattern: the first two and last two transformer layers are consistently high-error under aggressive quantization, while middle layers tolerate Q3-Q4 with near-zero activation drift. Attention projection layers (q_proj, k_proj) tend to be more sensitive than MLP layers (gate_proj, down_proj):

Mixed-Precision Recommendations

Q-Heatmap goes beyond visualization and outputs a mixed-precision configuration file compatible with llama.cpp's --tensor-type argument and AutoGPTQ's per-layer bit assignment. High-error layers are assigned Q6 or Q8; low-error layers drop to Q3 or Q2.

In practice, mixed-precision configurations produced by Q-Heatmap recover 80-90% of the perplexity lost in uniform Q4 quantization while adding only 5-8% to the total model size:

python q_heatmap.py \
  --fp16 meta-llama/Llama-3.1-8B \
  --quantized ./models/llama-3.1-8b-Q4_K_M.gguf \
  --format gguf \
  --calibration ./calib/c4_512samples.jsonl \
  --output heatmap.html \
  --export_config mixed_precision.json

The exported mixed_precision.json maps each tensor name to its recommended bit depth and can be passed directly to llama-quantize for re-quantization.

How to Build This with NEO

Open NEO in VS Code or Cursor and describe what you want to build. A good starting prompt for this project:

"Build a quantization error analysis tool that loads a full-precision model and its GGUF or GPTQ quantized counterpart, computes per-layer weight distance, activation MSE, and KL divergence on a calibration dataset, then renders an interactive Plotly heatmap with layers on one axis and bit depths on the other, and exports a mixed-precision configuration file."

Build with NEO →

NEO generates the project structure and core implementation. From there you iterate — ask it to add AWQ format support, integrate perplexity benchmarking against Wikitext-2, or build a CLI batch mode that audits an entire model zoo and compares quantization sensitivity across architectures. Each request builds on what's already there.

To run the finished project:

git clone https://github.com/dakshjain-1616/q-heatmap
cd q-heatmap
pip install -r requirements.txt
python q_heatmap.py --fp16 meta-llama/Llama-3.1-8B --quantized ./models/model-Q4_K_M.gguf --format gguf

Open heatmap.html to explore per-layer quantization sensitivity and download the mixed-precision config for re-quantization.

NEO built a per-layer quantization error visualizer that identifies which transformer layers break under compression and generates mixed-precision configurations to recover accuracy. See what else NEO ships at heyneo.com.

Try NEO in Your IDE

Install the NEO extension to bring AI-powered development directly into your workflow:

• VS Code: NEO in VS Code
• Cursor: Install NEO for Cursor