Qwen3.8-Flash-Next (125B) on an RTX 3060 12 GB with 30 GB RAM
Strata-HotRod is a fork of Strata that streams the model's experts from your SSD. 50-70 tokens/s on one gaming PC. Nothing leaves your PC.
🔥 hot experts on the card → 🌡️ warm in RAM → ❄️ cold on the SSD
Tested on one machine, with one model. Other cards, AMD, Windows, macOS and other model sizes may work in upstream Strata; they were not run here, so they are not described.
Can Qwen3.8-Flash-Next run on an RTX 3060 12 GB with low RAM?
Yes. On one RTX 3060 12 GB with 30 GB of RAM and an NVMe SSD, Strata-HotRod runs Qwen3.8-Flash-Next (Qwen 3.8 Flash Next, 125B) as the 78 GB IQ3_S at 50-70 tokens/s (131K context; 262K fits). The experts that do not fit in RAM are read from the SSD, and a look-ahead guesses ahead which ones.
How much RAM does it need?
Strata-HotRod was measured on 30 GB. Strata's setup recommends 64 GB for the IQ3_S size and says that with less RAM it reads the rest of the experts from the SSD, which its docs call much slower. Strata-HotRod works on that path.
Does it need a cloud service?
No. Everything runs on your PC and nothing leaves it. Setup downloads the model once from Hugging Face.
Can it run a model bigger than the RAM?
Yes. The 78 GB IQ3_S runs on a 30 GB PC. A separate DeepSeek-V4.1-Flash branch (341 GB of files) runs at about 5 tokens/s on the same PC, limited by the NVMe at about 2.8 GB/s.
How fast is it?
Measured on the machine below. Every answer in these runs was read and checked.
| Answer (131K context) | tokens/s |
|---|---|
| Code (a Python function, 500-650 tokens) | 52 - 55 |
| Reasoning (a two-train problem, ~1,000 tokens) | 66 - 71 |
| Story (a paragraph, 100-150 tokens) | 50 - 53 |
| Long prompts (262K context) | Read in | Answer written at |
|---|---|---|
| 8,554 tokens | 16.9 s | 52 - 54 tokens/s |
| 45,398 tokens | 67 - 68 s | 38 - 41 tokens/s |
| The same 45K conversation, next turn | 0.4 - 0.5 s | 34 - 45 tokens/s |
The first message of a chat is the slow part: it waits on the SSD for the model's experts. Follow-ups reuse what was already read.
The machine these numbers come from
| Graphics card | NVIDIA GeForce RTX 3060, 12 GB |
| Processor, RAM | Intel Core i5-12400F, 30 GB |
| Disk | 500 GB NVMe on a PCIe 3.0 link; the model files live on it |
| System | Rocky Linux 9.6 |
| Model | Qwen3.8-Flash-Next IQ3_S (about 3.5 bits per weight, 78 GB of files), by ISTA-DASLab |
How do I run a very large AI model on small local hardware?
If the model is a mixture-of-experts (MoE) model, you do not need to fit all of it in GPU memory or even in RAM. Only a few experts are used for each token, so the rest can wait on the SSD. Strata-HotRod keeps the hot experts on the GPU, the warm ones in RAM and the cold ones on the NVMe drive, and a learned look-ahead loads the next experts before they are needed.
- Qwen3.8-Flash-Next, 125B parameters, 78 GB of model files, on an RTX 3060 12 GB with 30 GB of RAM: 50-70 tokens/s.
- DeepSeek V4.1, 341 GB of files, on the same PC: 5.2-5.4 tokens/s (separate branch,
upstream-dsv41).
Longer guides: run a 125B model on a 12 GB GPU with 30 GB of RAM and run a mixture-of-experts model from an SSD.
Both measured on one machine, the one described below. This is for MoE models; dense models do not work this way. Other GPUs, operating systems and models are untested.
Run it the way it was measured
An NVIDIA card, Linux (Rocky 9 was tested; other distributions are untested), CUDA 13, and an NVMe with room for the 78 GB of model files plus the draft layer.
sudo dnf install -y git gcc-toolset-13
git clone https://github.com/sangharshadhyeta/Strata-HotRod.git
cd Strata-HotRod
scl enable gcc-toolset-13 -- ./setup.sh --model IQ3_S --context 131072 --build --yes --no-startThen run it inside a memory limit (94% of RAM; two uncapped runs once froze the test PC):
systemd-run --scope -p MemoryMax=$(awk '/MemTotal/ {printf "%dK", $2 * 0.94}' /proc/meminfo) -p MemorySwapMax=0 \
.venv/bin/python serve/server.py --engine strata --config strata-iq3_s.json --port 8095When it prints ready: http://127.0.0.1:8095/v1, any OpenAI-compatible client works with that address.
--build compiles this fork's engine; without it setup downloads upstream's, which lacks these changes. Full steps and
the DeepSeek install are in the GitHub README.
Not tested
- Any other graphics card, any AMD card, Windows, macOS or another Linux.
- Any other model or size.
- Several requests at once, or two graphics cards.
- A prompt that fills the whole 262K window (the longest served: 114K tokens).
Get it
Strata-HotRod on GitHub (Qwen3.8-Flash-Next) · DeepSeek branch · Write-up on dev.to · Write-up on Medium