Can I run a 125B model on a 12 GB GPU?

Yes, if it is a mixture-of-experts model. Qwen3.8-Flash-Next (125B, 78 GB IQ3_S) ran at 50-70 tokens/s on one RTX 3060 12 GB with 30 GB of RAM and an NVMe SSD using Strata-HotRod. Measured on one PC.

How can a model bigger than my RAM run at all?

A mixture-of-experts model uses only a few experts per token. The hot experts stay on the GPU, warm ones in RAM and cold ones on the SSD, and a learned look-ahead reads the next experts early.

Is 30 GB of RAM enough?

It was enough on the test PC: 50-70 tokens/s. Strata recommends 64 GB for this size and says less RAM means more reads from the SSD, which it calls much slower.

Run a 125B model on a 12 GB GPU with 30 GB of RAM

Short answer: yes, if it is a mixture-of-experts (MoE) model. Qwen3.8-Flash-Next has 125 billion parameters; its 78 GB IQ3_S files ran at 50-70 tokens/s on one RTX 3060 12 GB with 30 GB of RAM and an NVMe SSD. Nothing leaves the PC.

The machine: RTX 3060 12 GB, Intel i5-12400F, 30 GB RAM, a 500 GB PCIe 3.0 NVMe (WD Blue SN5000), Rocky Linux 9.6.

Why a 78 GB model fits in 12 GB of video memory and 30 GB of RAM

An MoE model uses only a few of its experts for each token, so most of it can wait somewhere slower. Strata-HotRod keeps the hot experts on the GPU, the warm ones in RAM and the cold ones on the SSD. A look-ahead that learns while it runs guesses which experts the next layers will want and starts reading them early: 85-88% of the needed experts one layer ahead, against 77-79% for the router's own guess. When an expert is not on the card, a close one that is can stand in for it, and tokens that a stand-in could flip are recomputed exactly.

Measured speed

Answer (131K context)	tokens/s
Code	52-55
Reasoning	66-71
Prose	50-53

Long prompts: 8,554 tokens read in 16.9 s; 45,398 tokens in 67-68 s; the next turn of a 45K conversation resumes in 0.4-0.5 s. 262K context fits.

How much RAM does it need?

This was measured on 30 GB. Strata's own setup recommends 64 GB for the IQ3_S size and says that with less RAM it reads the rest of the experts from the SSD, which its docs call much slower. On this PC the learned look-ahead and the RAM tier are what make 30 GB work. A faster or slower drive will change the numbers.

Run it

An NVIDIA card, Linux (Rocky 9 was tested), CUDA 13, and an NVMe with room for the 78 GB of model files.

sudo dnf install -y git gcc-toolset-13
git clone https://github.com/sangharshadhyeta/Strata-HotRod.git
cd Strata-HotRod
scl enable gcc-toolset-13 -- ./setup.sh --model IQ3_S --context 131072 --build --yes --no-start

Run it inside a memory limit (94% of RAM; two uncapped runs once froze the test PC):

systemd-run --scope -p MemoryMax=$(awk '/MemTotal/ {printf "%dK", $2 * 0.94}' /proc/meminfo) -p MemorySwapMax=0 \
  .venv/bin/python serve/server.py --engine strata --config strata-iq3_s.json --port 8095

When it prints ready: http://127.0.0.1:8095/v1, any OpenAI-compatible client works with that address.

Not tested

Other GPUs, AMD, Windows, macOS, other model sizes, several requests at once and two cards. Dense models do not work this way.

Project page · Run a 125B model on a 12 GB GPU · Run MoE models from an SSD · GitHub

Strata-HotRod is a fork of Strata by Niko1221 (MIT). All numbers were measured on one PC.