---
title: Four Bits. Full Consequences.
slug: quantization-without-magic
canonical_url: https://blog.openfactoryai.com/quantization-without-magic
published_at: 2026-07-14T20:47:00.289665+00:00
author: OpenFactoryAI
tags: LLM quantization, GPTQ, AWQ, SmoothQuant, KV cache
tldr: Quantization maps model tensors to fewer representable values, reducing storage and often memory traffic. It does not automatically deliver proportional speed or preserved quality. Weight, activation, and KV-cache quantization attack different bottlenecks; scales, packing, kernels, hardware, batch shape, and task sensitivity decide the realized outcome. The right deployment is the lowest precision that passes workflow quality and SLO gates on the actual serving stack.
key_takeaways:
  - Name the tensor and numerical format, not just “4-bit.”
  - Weight footprint falls roughly with bit width before metadata and runtime memory.
  - Speed appears only when supported kernels exploit the representation at the target shape.
  - Activation outliers and KV-state behavior require different methods from weight-only compression.
  - Validate verified outcomes and tails, not perplexity alone.
---

A 70-billion-parameter model at 16 bits per parameter has a 140 GB decimal lower-bound weight footprint. At four bits, the arithmetic says 35 GB.

That 4x reduction is real arithmetic. It is not a promise of 4x throughput, 4x lower latency, or unchanged output quality.

Quantization replaces a high-precision tensor representation with a smaller set of values plus the information needed to map between them. The benefit can be smaller artifacts, fewer bytes moved, larger batches, longer context, or cheaper hardware. The cost can be numerical error, calibration work, conversion overhead, unsupported kernels, and a larger validation surface.

The useful question is:

> Which tensor is the bottleneck, which lower-precision operation does the deployed hardware execute efficiently, and what workflow quality may not regress?

::figure{template=layers caption="Quantization can target different parts of the inference memory stack" items="Weights :: fixed model parameters | Activations :: intermediate computation | KV cache :: per-sequence attention state | Scales and metadata :: reconstruction parameters | Runtime workspace :: kernels and communication"}

## Quantization is a mapping, not a file suffix

For a simple affine integer quantizer, a real value `x` is mapped to an integer `q` using scale `s` and zero point `z`:

```text
q = clamp(round(x / s) + z, q_min, q_max)
x_hat = s * (q - z)
```

`x_hat` approximates `x`. The difference is quantization error. Symmetric schemes may omit a nonzero `z`. Group-wise schemes calculate scales for subsets of a tensor. Non-uniform codebooks allocate representable values differently. Mixed precision retains sensitive components at more bits.

“INT4” therefore leaves important questions unanswered:

- Which tensors are four-bit?
- Symmetric, asymmetric, integer, floating, or codebook representation?
- Per tensor, channel, group, or token scales?
- What group size?
- Which layers remain higher precision?
- How were calibration examples selected?
- Are low-bit values directly consumed by a kernel or unpacked first?
- What accumulator precision is used?

Two files called 4-bit can have different size, accuracy, and speed.

## Weight-only quantization attacks model traffic

Weights are the largely fixed parameters loaded for inference. A dense 70-billion-parameter lower bound is:

| Nominal format | Bits/parameter | Decimal weight bytes | Reduction from 16-bit |
|---|---:|---:|---:|
| FP16/BF16 | 16 | 140 GB | 1x |
| INT8 | 8 | 70 GB | 2x |
| INT4 | 4 | 35 GB | 4x |
| INT3 | 3 | 26.25 GB | 5.33x |

::figure{template=bar caption="Dense 70B model weight lower bound, decimal GB before metadata" items="16-bit :: 140 | 8-bit :: 70 | 4-bit :: 35 | 3-bit :: 26.25"}

Real artifacts add scales, zero points, group metadata, padding, packing alignment, and sometimes retained high-precision layers. Runtime adds activations, KV state, communication buffers, graph workspace, and allocator reserve. Do not use the table as a device-fit guarantee.

[GPTQ](https://openreview.net/forum?id=tcbBPnfwxS) is a post-training weight quantization method using approximate second-order information to quantize weights while compensating error. [AWQ](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html) uses activation observations to identify salient weight channels and protects them through scaling in a hardware-oriented weight-only method.

Both are more considered than rounding every value independently. Neither creates a universal “negligible quality loss” result. Calibration data, model family, bit width, group size, backend, and task determine what transfers.

## Activation quantization is an execution decision

Activations are intermediate values produced while the model runs. Quantizing both weights and activations can enable integer matrix multiplication and reduce traffic across more of the computation. Activations are dynamic and can contain outlier channels whose range makes a shared low-precision scale wasteful.

[SmoothQuant](https://proceedings.mlr.press/v202/xiao23c.html) observes that weights are easier to quantize than activations and applies a mathematically equivalent channel-wise rescaling that shifts quantization difficulty from activations into weights. Its evaluated W8A8 system quantizes both to eight bits.

The mechanism matters more than the label. Suppose most activation values occupy `[-2, 2]`, but one channel reaches `60`. A scale covering the full range gives coarse steps to the dense central values. Per-channel treatment, rescaling, clipping, or mixed precision can protect useful resolution, each with a different kernel and quality cost.

Activation quantization also depends on batch and sequence shape. A kernel optimized for large matrix operations may underperform on small decode batches. A format accelerated on one GPU generation may be emulated or converted on another.

::figure{template=flow caption="Post-training quantization is a calibration, conversion, kernel, and validation pipeline" items="Representative data :: real task slices | Observe tensors :: ranges and outliers | Choose scheme :: bits, groups, exceptions | Convert model :: pack plus metadata | Run target kernels :: real hardware | Validate :: quality, latency, memory"}

## KV-cache quantization buys active-sequence capacity

The KV cache stores attention keys and values for earlier tokens. Unlike weights, it grows per active sequence and context length. Compressing it can permit more concurrent sequences or longer context even when the model already fits.

A simplified KV-size equation is:

```text
KV bytes ≈
tokens * layers * KV_heads * head_dimension
* 2 for keys and values * bytes_per_element
```

For an illustrative model with 80 layers, eight KV heads, head dimension 128, 32,768 tokens, and 16-bit elements:

```text
32768 * 80 * 8 * 128 * 2 * 2 bytes
= 10,737,418,240 bytes
= 10.0 GiB per sequence
```

An ideal packed four-bit representation is 2.5 GiB, before scales, residual high-precision windows, alignment, fragmentation, and kernel workspace.

[KIVI](https://proceedings.mlr.press/v235/liu24bz.html) studies asymmetric 2-bit KV-cache quantization and treats key and value tensors differently based on observed channel and token behavior. That design detail is a warning against applying a weight quantizer blindly to serving state.

KV compression affects every later attention step. Validate long-context retrieval, position sensitivity, and generation trajectories, not only short prompts. Memory savings may increase maximum batch size, which then changes latency and throughput through the scheduler described in [Continuous Batching](/continuous-batching).

## Fewer bytes help only if the kernel uses them

Autoregressive decode at small batch sizes is often constrained by moving model weights relative to arithmetic performed. Reducing bytes can reduce this pressure. A crude lower bound illustrates the opportunity.

If a step effectively streams 140 GB of weights over a 3.0 TB/s path:

```text
weight-read lower bound = 140 GB / 3000 GB/s = 46.7 ms
```

At 35 GB:

```text
35 / 3000 = 11.7 ms
```

This is not a latency prediction. Real systems shard weights, reuse data across a batch, communicate between devices, perform arithmetic, access caches, unpack low-bit values, and achieve less than a nameplate bandwidth in workload-specific ways.

A low-bit model can fail to speed up when:

- the backend dequantizes into a larger format before every operation;
- the hardware lacks a fast kernel for that format and shape;
- scale lookup and unpacking dominate tiny batches;
- another bottleneck, such as KV traffic or interconnect, remains;
- reduced model memory is not converted into a larger useful batch;
- the quantized route falls back to unsupported operators;
- request latency is dominated by queue, tools, or verification.

Measure kernel coverage. A model that is “mostly INT4” but repeatedly crosses formats can spend the saving on conversions.

::figure{template=waterfall caption="Nominal 4x byte reduction is reduced by metadata and execution overhead" items="FP16 weight traffic :: 140 | Packed INT4 :: -105 | Scales and exceptions :: 3 | Unpack/dequant overhead :: 8 | Other uncompressed traffic :: 22 | Effective traffic scenario :: 68"}

The numbers in this visual are an illustrative decomposition, not a benchmark. The correct diagram for a deployment comes from profiler bytes and kernel traces.

## Calibration data defines what error the method sees

Post-training methods commonly use a calibration set to estimate ranges, sensitivities, or error compensation. A generic text sample may not exercise the values induced by your prompts.

Include representative slices:

- system and tool schemas;
- long retrieved documents;
- code and exact identifiers;
- supported languages and scripts;
- numerical reasoning;
- structured output constraints;
- safety and refusal cases;
- typical and extreme context lengths.

Hold out evaluation data. Choosing quantization parameters and declaring success on the same examples invites overfitting even without gradient training.

Version the calibration manifest with the artifact:

```text
base model hash
quantization method and implementation commit
tensor formats, group sizes, and exceptions
calibration dataset IDs and sampling procedure
conversion hardware and software
expected inference backend and kernels
```

Reconversion should be reproducible. Downloading an untraceable community artifact saves time while transferring trust to unknown calibration and packing choices.

## Perplexity is a screen, not an acceptance test

Perplexity can detect broad language-model degradation, but a small average change may conceal a large failure in a narrow high-value behavior. Quantization error can change a token near a branch point, after which an autoregressive trajectory diverges.

Evaluate at least two task types and the exact product workflow. For an agent platform:

| Layer | Example measure |
|---|---|
| token/model | held-out loss or perplexity |
| capability | code tests, retrieval accuracy, math exact match |
| protocol | schema-valid tool calls, stop behavior, citation format |
| workflow | verified task completion and side-effect correctness |
| serving | TTFT, TPOT, goodput, memory, power |
| economics | cost per verified successful outcome |

Use paired examples when comparing formats so differences are attributable to precision rather than changing prompts. Report confidence intervals for task success and inspect disagreements, especially severe errors.

For stochastic generation, control seeds where supported but do not mistake exact text mismatch for failure. The outcome verifier should define acceptable behavior.

## Quantization-aware training and QLoRA answer different questions

Post-training quantization changes a trained model without full retraining. Quantization-aware training simulates or includes low-precision effects during training so weights adapt. It can recover quality at additional data and compute cost.

[QLoRA](https://papers.nips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html) stores a frozen pretrained model in a four-bit representation while training low-rank adapters, reducing fine-tuning memory. It introduced NormalFloat and double quantization in that training system.

QLoRA does not mean every resulting deployment executes the base weights with the fastest four-bit inference kernel. Fine-tuning representation, merged artifact, adapter serving, and production kernel are separate choices. State them separately.

## A worked device-fit decision

An architect has two 80 GB devices and a dense 70B model.

At 16-bit, the 140 GB weight lower bound leaves only 20 GB total before runtime overhead if perfectly split. KV state, workspace, and communication buffers make the fit fragile or impossible.

At nominal 8-bit, the 70 GB weight lower bound leaves about 90 GB across the pair for runtime state, but kernel support and sharding still determine speed.

At nominal 4-bit, 35 GB weights may fit on one 80 GB device with room for KV and workspace, depending on metadata and engine. One-device placement can avoid some inter-device communication. Alternatively, two devices can use the freed memory for more concurrent state.

Those options create different business outcomes:

- single device: lower fixed cost and simpler topology;
- two devices: potentially more goodput or longer context;
- higher precision: potentially safer quality but lower capacity;
- mixed route: low precision for tolerant work, high precision fallback for failed verification.

The lowest-bit file is not automatically the lowest cost per outcome. A 2 percent drop in verified success can dominate a 40 percent serving-cost saving when retries or failure loss are high.

::figure{template=matrix caption="Choose precision by verified quality and realized serving value" items="High quality + high speed :: deploy candidate | High quality + no speed :: use only for fit/cost | Quality loss + high speed :: narrow route or fallback | Quality loss + no speed :: reject | Unknown kernel :: benchmark first | Unknown workflow quality :: evaluate first"}

## Precision needs production observability

Quantization is not finished when the artifact passes an offline suite. The traffic distribution, context lengths, prompts, adapters, runtime, and kernels continue changing.

Tag every inference span with:

```text
base_model_revision
quantization_recipe_hash
weight_activation_KV_formats
runtime_and_kernel_build
hardware_type
adapter_revision
route_and_fallback_reason
```

Without these dimensions, a latency or quality regression is attributed to “the model” even when only one packed kernel path changed.

Monitor operational measures by precision route:

- verified success and human escalation;
- schema, parser, and tool-call failure;
- fallback and retry rate;
- severe-error classes;
- output-length and stop-behavior shifts;
- TTFT, TPOT, deadline goodput, and queue time;
- weight, KV, workspace, and allocator peaks;
- low-bit kernel coverage and conversion operations;
- energy or accelerator time per verified outcome.

A shadow comparison against the higher-precision baseline is useful on a sampled, privacy-safe subset. Compare outcome decisions and protocol artifacts, not just exact wording. Route disagreements through deterministic validators where possible and selective human review where consequence justifies it.

Set rollback triggers before exposure. For example:

```text
rollback if severe-error upper confidence bound exceeds limit
or schema-valid tool calls fall below gate
or p95 TPOT exceeds baseline by budget
or memory headroom falls below safe admission reserve
```

Canary by workload slice, not random traffic alone. A random one-percent canary may contain too few multilingual, long-context, or rare tool requests to reveal the failure the quantizer introduces. Deliberately include protected slices and extremes.

When the base model changes, reconvert rather than assuming the old scales remain valid. When the runtime changes, rerun performance tests even if the artifact bytes are identical. Kernel selection, workspace, graph capture, and packing support can change independently of model quality.

Finally, retain the unquantized or accepted higher-precision route until rollback and fallback have been exercised under load. A compressed artifact that is cheap to store but impossible to replace within the incident window creates a reliability dependency that belongs in its economic cost.

## A builder playbook

### 1. Profile the current bottleneck

Measure weight memory, KV memory, activation/workspace peaks, bandwidth, compute utilization, communication, batch size, and queueing. Quantizing weights will not fix a tool-latency bottleneck.

### 2. Define the numerical artifact

Record tensor targets, format, group size, scale type, exceptions, accumulator, calibration, converter, and runtime kernel. Ban labels such as “4-bit model” from benchmark reports without these fields.

### 3. Establish quality gates before conversion

Select held-out capability and workflow tests, severe-error cases, protocol validity, and acceptable confidence bounds. Include at least one precision-sensitive domain slice.

### 4. Benchmark the deployed stack

Use the exact model artifact, runtime, kernels, hardware, prompts, outputs, concurrency, and SLOs. Compare warm and cold states. Report memory plus latency and goodput.

### 5. Sweep mixed precision

Test sensitive layers, activations, embeddings, output heads, and KV state separately. A modest higher-precision exception can recover quality with little memory cost.

### 6. Roll out by consequence

Start with low-loss, verifiable tasks. Route or fall back when confidence, protocol validation, or outcome verification fails. Monitor disagreement against a higher-precision shadow sample.

### 7. Recalculate outcome economics

Include conversion work, artifact storage, operational variants, accelerator count, energy, latency value, retry changes, and failure loss. Use the framework in [Cost per Successful Outcome](/cost-per-verified-outcome).

## The decision

Quantize the tensor that controls your capacity, using a method and kernel supported by the target hardware. Choose the lowest precision that passes product-level quality gates across realistic context, concurrency, and tasks.

The magic disappears when the deployment report names every relevant noun: weights, activations, KV cache, format, group, calibration, kernel, hardware, workload, and verified outcome. What remains is better than magic: a reproducible engineering trade.

## References

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://openreview.net/forum?id=tcbBPnfwxS), Frantar et al., ICLR 2023. A foundational second-order post-training weight quantization method.
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://proceedings.mlr.press/v202/xiao23c.html), Xiao et al., ICML 2023. Addresses activation outliers through equivalent rescaling for W8A8 execution.
- [AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html), Lin et al., MLSys 2024. Protects salient weight channels using activation observations.
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://proceedings.mlr.press/v235/liu24bz.html), Liu et al., ICML 2024. Treats key and value state according to their different observed structure.
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://papers.nips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html), Dettmers et al., NeurIPS 2023. Separates quantized base-model storage during adapter training from full-model updates.

All model footprint and bandwidth examples are derived illustrations. Inputs and limitations are preserved in the claim ledger.


## FAQ

### What is LLM quantization?

It represents selected tensors using fewer or differently allocated numerical values plus scales or other metadata. The goal is to reduce storage, memory traffic, or execution cost while keeping task quality within a declared bound.

### Does 4-bit quantization make inference four times faster?

Not necessarily. It can reduce raw weight bytes by roughly fourfold from 16-bit, but metadata, unpacking, unsupported kernels, communication, KV traffic, batching, and other bottlenecks determine realized speed.

### What is the difference between weight and KV-cache quantization?

Weights are fixed model parameters and their compression primarily reduces model footprint and weight traffic. KV state grows with each active sequence and context, so its compression primarily increases sequence or context capacity and changes attention-time traffic.

### What calibration data should be used?

Use representative but held-separate samples covering real prompts, tool schemas, languages, long context, code, numbers, structured outputs, and safety cases. Preserve a manifest and evaluate on held-out data.

### Is perplexity enough to validate a quantized model?

No. It is a useful model-level screen but can hide narrow failures. Validate capability tasks, schema and tool behavior, long-context retrieval, severe errors, verified workflow success, latency, memory, and cost.

### Is QLoRA an inference quantization method?

QLoRA is primarily a memory-efficient fine-tuning method that stores a frozen base model in four-bit form while training adapters. The final inference representation and kernel path still need separate deployment decisions and benchmarks.


## Test yourself

### 1. What is the decimal lower-bound weight size of a dense 70B model at 4 bits per parameter?

- [ ] 17.5 GB
- [x] 35 GB
- [ ] 70 GB
- [ ] 140 GB

**Explanation:** 70 billion multiplied by half a byte equals 35 billion bytes.

### 2. Which method specifically targets activation outlier difficulty through rescaling?

- [ ] PagedAttention
- [x] SmoothQuant
- [ ] Little's Law
- [ ] ReAct

**Explanation:** SmoothQuant shifts quantization difficulty from activations to weights using an equivalent transformation.

### 3. Why can a quantized file fail to run faster?

- [ ] Fewer bits always add network hops
- [x] The backend may lack an efficient native kernel and spend time unpacking or converting
- [ ] Quantization removes batching
- [ ] All activations become strings

**Explanation:** Realized speed depends on executed kernels and the actual bottleneck.

### 4. What does KV-cache quantization most directly expand?

- [ ] Training dataset size
- [x] Active sequence or context capacity
- [ ] Number of model layers
- [ ] API permissions

**Explanation:** KV state grows per active token and sequence, so compressing it frees serving memory.

### 5. What is the final acceptance criterion for a production quantization candidate?

- [ ] Smallest artifact alone
- [ ] Lowest perplexity alone
- [x] Workflow quality and serving SLOs on the deployed stack
- [ ] Most popular format

**Explanation:** The product needs verified outcomes plus realized latency, capacity, and cost.