Running a 13B Model on a 12GB GPU: How Offloading Works
A 13B-parameter model in common quantized formats does not fit comfortably in 12GB of VRAM, but GPU offloading lets part of it live in system RAM so the whole model still runs. This explainer covers what offloading actually does, why every offloaded layer costs speed, and what that trade-off means in practice.
admin
A 13B-parameter model sounds like it should fit on a modern graphics card, and in raw parameter count it nearly does. The trouble is memory, not math. To run a local language model at full speed, the whole model has to sit inside the GPU's VRAM, and a 13B model in a typical quantized format is larger than the 12GB a mid-range card provides. Load it the normal way and the result is an out-of-memory error, which is where most people conclude the model simply won't run on their hardware.
GPU offloading is the option between "everything on the GPU" and "nothing runs." Instead of placing every layer of the model in VRAM, you keep some layers on the GPU and leave the rest in ordinary system RAM, where the CPU processes them. The model still runs end to end; it is just split across two kinds of memory with very different speeds ¹. That split is the entire concept, and almost everything else about offloading follows from it.
The kitchen analogy in the AI/TLDR explainer is a fair one: VRAM is a small countertop within arm's reach, system RAM is a much larger pantry across the room. If a recipe needs more ingredients than the counter holds, you keep the frequently used ones close and walk to the pantry for the rest. Dinner still gets made, but every trip across the room adds time. In a language model, that trip happens on every token the model generates, because each layer must be processed in sequence for every output token.
Two concrete cases show how the arithmetic works. Take a 13B model stored at roughly 4-bit quantization, the format most people use to make models fit on consumer hardware. Four bits per parameter works out to about half a byte, so 13 billion parameters occupy on the order of 6.5GB before any overhead. That number alone suggests a 12GB card has room to spare, and it might, until you add the key-value cache that grows with context length, the runtime's own overhead, and whatever the desktop compositor is already using. Push the context longer and the headroom shrinks. In that situation, offloading a handful of layers to system RAM can be the difference between a working session and an out-of-memory crash, at the cost of some speed. The exact layer count that fits depends on the runtime, the context length and the quantization format, and no source here establishes a universal number.
The second case is the more dramatic one. XDA's write-up on running large models on older GPUs describes deploying a 35B-parameter mixture-of-experts model, Qwen3.6-35B-A3B, on a 12GB card through layer offloading ². A model that large clearly cannot sit entirely in 12GB of VRAM in any ordinary format, so the fact that it runs at all is the point: offloading turns a hard memory wall into a speed trade-off. The same article notes that sub-10B models can handle productivity tasks but are weaker for demanding coding work or tasks that need precise output, which is the practical reason someone would bother with a bigger model and accept slower generation instead of simply downloading a smaller one.
The cost is straightforward to reason about even without benchmark numbers. Layers on the GPU are processed at GPU speed; layers in system RAM are processed by the CPU and must move across the PCIe bus. The more layers you offload, the larger the share of each token's computation happens on the slower path, so tokens per second falls. The relationship is not perfectly linear, because attention and other operations interact with memory placement in ways that vary by runtime, but the direction is reliable: fewer offloaded layers means more speed, and more offloaded layers means a bigger model can run at all. What no source here quantifies is the exact tokens-per-second figure for a specific 13B model on a specific 12GB card, so treat any such number you see elsewhere as hardware- and settings-specific rather than a property of offloading itself.
The practical consequence is a decision rule rather than a verdict. If a 13B model at your chosen quantization fits in 12GB with room for context, run it fully on the GPU and enjoy full speed. If it doesn't, offload the smallest number of layers that eliminates the out-of-memory error, then increase context length only as far as your remaining headroom allows. The model will be slower, and how much slower depends on how much you moved to RAM, but it will run, and that is the trade the technique exists to offer. Readers weighing quantization levels against memory budgets will find the format trade-offs covered in our quantization explainer, and those working with smaller hardware can compare options in our roundup of local models for 16GB of RAM.
Sources
