Fitting an LLM in your own server room: the memory math behind on-prem AI

Sooner or later, someone asks if we can run “the model” ourselves. Usually it’s a bank, a hospital or a government team. The reason is rarely ideology. Regulation says the data can’t leave the building, or the steady traffic makes per-token API pricing painful. Either way, the question becomes very concrete: which model fits on the hardware we actually have, and how good will it still be?

I’ve found that most of this decision can be made on paper, before anyone downloads a single checkpoint. It comes down to three things: how many bytes the model needs, how much damage you do by shrinking those bytes, and how fast you can move them.

Bytes first

The weights are the easy part. Every parameter costs a number of bits:

$$\text{weight memory} = N_{\text{params}} \times \frac{\text{bits per weight}}{8}$$

A 70B model in bf16 (16 bits) is $70 \times 10^9 \times 2 = 140$ GB. That’s two 80 GB cards before you’ve served a single request. The same model in 4-bit is a different story.

Weights only. Real deployments also need room for the KV cache and activations.

One detail people miss: 4-bit formats don’t cost exactly 4 bits. Weights are quantized in small groups, and each group stores its own scale, often a 16-bit number per 128 weights. That adds

$$\frac{16}{128} = 0.125 \text{ bits per weight},$$

so “4-bit” is really about 4.125 bits, and a 70B model needs ~36 GB, not 35. Small, but it’s exactly the kind of thing that turns “it fits” into “out of memory” at 2 a.m.

What quantization actually does to a number

The simplest scheme, symmetric uniform quantization, maps a real weight $x$ to a $b$-bit integer with one scale $s$:

$$s = \frac{\max |x|}{2^{b-1} - 1}, \qquad q = \operatorname{clamp}\!\left(\operatorname{round}\!\left(\frac{x}{s}\right),\, -2^{b-1},\, 2^{b-1}-1\right), \qquad \hat{x} = s\, q$$

Rounding error is at most half a step, and if you treat it as uniform noise (the classic high-resolution approximation; see Gray & Neuhoff, 1998), its mean squared error is

$$\mathbb{E}\!\left[(x - \hat{x})^2\right] \approx \frac{s^2}{12}.$$

Look at what controls $s$: the largest value in the group. That’s the whole story of LLM quantization in one line. If a group has one huge outlier, $s$ becomes large, the step gets coarse, and every normal weight in that group loses precision.

LLMs have outliers. A small number of feature dimensions carry values far bigger than the rest. LLM.int8() showed these emergent outliers are what break naive 8-bit inference in large models. That’s why the methods that work all deal with outliers somehow:

• Small groups. Use a separate scale for every 64 or 128 weights, so one outlier only hurts its neighbours.
• GPTQ. Quantize weights one at a time, and use second-order information to adjust the remaining weights so they absorb the error.
• AWQ. Notice that a small fraction of weights matter much more, because their input activations are large. Scale those channels to protect them before quantizing.
• SmoothQuant. Move the difficulty from activations to weights with a per-channel rescaling, so both can run in 8-bit:

$$\mathbf{Y} = \left(\mathbf{X}\,\operatorname{diag}(\mathbf{s})^{-1}\right)\left(\operatorname{diag}(\mathbf{s})\,\mathbf{W}\right), \qquad s_j = \frac{\max |\mathbf{X}_j|^{\alpha}}{\max |\mathbf{W}_j|^{1-\alpha}}$$

The product is mathematically identical. You’ve just moved the “spiky” part to where it’s easier to quantize.

Don’t forget the KV cache

Weights are fixed. The KV cache grows with every token of every active request:

$$\text{KV per token} = 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{bytes}$$

For a model with 80 layers and 8 KV heads of dimension 128 (the Llama 3 70B shape, per the Llama 3 paper), that’s 320 KiB per token in bf16. At 8,000 tokens of context, a single request holds about 2.4 GiB. Ten concurrent users is about 24 GiB, comparable to the quantized weights themselves.

So when I plan an on-prem box, I budget memory like this:

$$\text{GPU memory} \;\ge\; \underbrace{N \cdot \tfrac{b}{8}}_{\text{weights}} \;+\; \underbrace{C \cdot T \cdot \text{KV}_{\text{tok}}}_{\text{KV cache}} \;+\; \underbrace{\sim 10\text{–}15\%}_{\text{activations, runtime}}$$

Here $C$ is concurrent requests and $T$ is tokens per request. Two cheap wins on the KV side: an FP8 KV cache halves it with little quality loss for many models, and prefix caching shares the system prompt across users instead of storing it $C$ times.

Speed is a bandwidth problem

When one user is waiting for tokens, the GPU spends most of its time reading weights, not doing maths. Every generated token needs every weight once. So the ceiling for a single stream is roughly

$$\text{tokens/s} \;\lesssim\; \frac{\text{memory bandwidth}}{\text{bytes of weights}}.$$

A 70B model at ~36 GB on a card with 1 TB/s of bandwidth tops out near 28 tokens/s for one user. The same model in bf16 would be split across two cards and still be slower per stream. This is the underrated benefit of quantization: at low batch sizes, fewer bytes means proportionally faster generation, not just a smaller footprint.

At larger batch sizes, the same weight read is shared by many requests and you move towards being compute-bound. That’s where a serving engine with continuous batching, such as vLLM, earns its keep. For CPU or edge deployments, llama.cpp and its GGUF formats are the practical choice. The arithmetic is the same; only the bandwidth number changes.

Quality: measure on your tasks

Here’s my one strong opinion. Perplexity is not an acceptance test. A quantized model can look nearly identical on perplexity and still lose noticeably on the thing you care about. That’s especially true for languages and domains that were thin in its training data, and for long-context or tool-calling behaviour.

So before any on-prem rollout I run the same evaluation set twice, full precision and quantized:

1. Real prompts from the target domain and language, not generic benchmarks.
2. Structured-output and tool-call validity rates (does the JSON still parse?).
3. Long-context retrieval, where KV quantization shows up first.
4. Latency at the expected concurrency, measured as p95, not the average.

If the 4-bit 70B beats the bf16 8B on your tasks and fits your hardware, that’s usually the sweet spot. If it doesn’t, no amount of clever quantization will rescue the plan, and it’s better to learn that on paper.

The short version

• Weights: $N \times b/8$, plus ~0.125 bits per weight for group scales.
• Error: driven by the largest value in each group, so manage outliers.
• KV cache: often as big as the quantized weights. Use FP8 KV and prefix caching.
• Speed at low batch: bandwidth divided by bytes, which is why smaller is faster.
• Quality: decide on your own evals, never on perplexity alone.

On-prem AI isn’t a compromise if you do this maths first. It’s just engineering with the constraints written down.

References

• Gray, R. M. & Neuhoff, D. L. (1998). Quantization. IEEE Transactions on Information Theory, 44(6) — uniform-noise model, $s^2/12$.
• Dettmers, T. et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022 — emergent outlier features.
• Frantar, E. et al. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
• Lin, J. et al. (2024). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024.
• Xiao, G. et al. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.
• Llama Team, AI @ Meta (2024). The Llama 3 Herd of Models — 70B model shape.
• vLLM documentation. Quantized KV cache — FP8 KV cache.
• llama.cpp — GGUF quantization for CPU and edge.

Memory figures are computed from the formulas above. The 1 TB/s bandwidth figure is a round illustrative number, not a specific product.