Quantization Explained: What Q4 and Q8 Actually Cost You in a Local AI Model

Every local AI model is, at bottom, a very large pile of numbers. Those numbers are the weights the model learned during training, and the precision at which you store them decides how much memory the model occupies, how fast it runs, and how much of its original quality survives. Quantization is the practice of storing those weights in fewer bits than the training format used.

Training and reference runs typically use 16-bit formats such as F16 or BF16, and a 32-bit float is the usual reference point for "no loss at all." Quantization pushes each weight down to 8, 6, 5, 4, 3 or even 2 bits. Fewer bits per value means a smaller file, less memory pressure and faster inference, at some cost in output quality ^¹. The trade-off is not a cliff but a slope, and the shape of that slope is the whole story.

Why fewer bits help so much comes down to arithmetic. A model's memory footprint is roughly its parameter count multiplied by the bits per weight. Halving the bits roughly halves the file. The Aspose documentation's relative size table makes the pattern concrete: F16 sits at 1.0x, Q8_0 at about 0.5x, Q6_K at about 0.38x, Q5_K_M at about 0.33x and Q4_K_M at about 0.27x, with the classic Q4_0 at about 0.25x. Those are approximate and depend on architecture and quantizer, but the ordering is reliable.

The naming is less mysterious than it looks. The number is the nominal bits per weight, and the suffix describes the scheme. The K-quants (Q6_K, Q5_K_M, Q4_K_M and so on) mix precisions across different parts of the model rather than treating every weight identically; the M and S tags denote medium and small variants of the same nominal level. The 0 suffix, as in Q8_0 or Q4_0, marks an older, simpler scheme. Separately, the IQ family applies importance quantization, which spends bits where they matter more and can beat a plain Q4 at a similar size — the Aspose table puts IQ4_XS and IQ3_S at roughly 0.23x to 0.30x with smaller quality loss than the equivalent Q formats.

Concrete example one: a 7B model at Q4 versus Q8. The Hugging Face GGUF quantization guide lists a 7B model at FP16 as 13.0 GB, at Q8_0 as 7.0 GB and at Q4_K_M as 4.1 GB, with perplexity rising from a 5.9565 baseline to 5.9584 for Q8_0 (about 0.03% worse) and 6.0565 for Q4_K_M (about 1.68% worse) ^². That is the core of the Q4-versus-Q8 decision in one line: Q8_0 buys near-lossless quality for roughly 70% more memory than Q4_K_M, while Q4_K_M accepts a small, measurable perplexity hit to fit in a much smaller footprint. The same guide's token-rate column shows Q4_K_M at 40 tokens per second against 25 for Q8_0 on its test setup, though those figures are specific to that setup and should be read as directional rather than as a promise for your machine.

Concrete example two: the same decision on a different budget. SitePoint's comparison gives a 7B model at roughly 4.4 GB for Q4_K_M, 5.5 GB for Q6_K and 7.2 GB for Q8_0, with perplexity penalties of about 1.5–3%, 0.5–1.5% and 0.1–0.3% respectively ^³. The absolute numbers differ from the Hugging Face figures — different models, different test conditions — but the ranking holds, and the practical reading is the same. Note also that the two sources disagree on where Q4_K_M lands in absolute terms; neither is wrong, and the discrepancy is a useful reminder that quant quality is model-specific, not a universal constant.

Where the quality loss actually shows up matters more than the average. A perplexity increase of a percent or two is an average over a whole corpus; it does not tell you which tasks will degrade. SitePoint's framework assigns Q4_K_M to general chat and memory-constrained setups, Q6_K to code generation and structured output, and Q8_0 to math and multi-step reasoning. That mapping is a rule of thumb rather than a measured law, but it is consistent with how quantization errors compound: a small per-token inaccuracy matters little in conversational prose and much more in a chain of arithmetic where each step depends on the last.

The practical consequence is a decision rule, not a single right answer. If the model plus its context fits comfortably in your available memory at Q8_0, take Q8_0 — the quality cost is close to nothing and the speed penalty is modest. If it does not fit, step down to Q6_K or Q5_K_M before jumping to Q4, and treat Q4_K_M as the sensible default when memory is genuinely tight, which is roughly how the community treats it: the Aspose guide calls Q4_K_M the default for most community-uploaded GGUFs and a good balance for 7B-and-up models. Below Q4, quality loss becomes noticeable and then large, and those levels are best reserved for cases where fitting the model at all is the priority.

Two caveats are worth keeping in view. First, the size and perplexity tables are approximations; actual file sizes depend on architecture and the specific quantizer, and the sources themselves label their figures as approximate or test-specific. Second, none of these numbers substitute for checking the model repository you are actually downloading. The Hugging Face guide recommends opening the model page with the llama.cpp local-app view and preferring the exact quant labels and sizes shown there over generic heuristics, falling back to the tables only when the repository gives no clear recommendation. That is the honest version of the advice: understand the trade-off from first principles, then confirm the specifics against the files in front of you. If you are still working out whether a given model fits your hardware at all, our guide to local AI models that fit in 16GB of system RAM covers the memory side of the same question.

Sources

1. ¹ Understand quantization docs.aspose.com
2. ² skills/huggingface-local-models/references/quantization.md · multimodalart/skills - GGUF Quantization Guide huggingface.co
3. ³ Q4 vs Q6 vs Q8: How Much Quality Each Quantization Loses www.sitepoint.com