Can a mixture-of-experts model run from an SSD?

Yes. Strata-HotRod keeps the hot experts on the GPU, warm ones in RAM and cold ones on the NVMe drive. Qwen3.8-Flash-Next (78 GB) ran at 50-70 tokens/s and DeepSeek-V4.1-Flash (341 GB of files) at 5.2-5.4 tokens/s on one RTX 3060 12 GB with 30 GB of RAM.

What limits speed when streaming from an SSD?

The drive. On the test PC the NVMe reads at about 2.8 GB/s, which is what limits the DeepSeek model. A learned look-ahead that starts reads early reduces the wait.

Does it work for dense models?

No. It relies on a mixture-of-experts model using only a few experts per token.

Run a mixture-of-experts model from an SSD on a consumer GPU

Short answer: when a mixture-of-experts (MoE) model is bigger than your RAM, you can keep the experts on an NVMe SSD and read only the ones each token needs. Strata-HotRod does this with the hot experts on the GPU, warm ones in RAM and cold ones on the drive.

The machine: RTX 3060 12 GB, Intel i5-12400F, 30 GB RAM, a 500 GB PCIe 3.0 NVMe (WD Blue SN5000), Rocky Linux 9.6.

What it looks like in numbers

Model	Files	tokens/s
Qwen3.8-Flash-Next (IQ3_S)	78 GB	50-70
DeepSeek-V4.1-Flash (separate branch)	341 GB	5.2-5.4

Same PC for both. The DeepSeek files are 138.6 GB of backbone plus a 202.8 GB Engram table (plus an 8 GB draft model). There the limit is the drive: this NVMe reads at about 2.8 GB/s.

What makes SSD streaming usable

A learned look-ahead guesses a few layers early which experts will be needed and starts the reads, so the card waits less.
Stand-in experts with a re-check: an expert that is not on the card can be replaced by a close one that is, and a token that the swap could flip is recomputed exactly.
Layer-by-layer prompt reading: a long prompt passes one layer at a time, so each layer's experts are read from the SSD once; the reads bypass the page cache.
A RAM floor: the engine keeps some RAM free and gives experts back when something else needs memory.

When this is not for you

If the whole model fits in your RAM and VRAM, you do not need streaming. Dense models, where every weight is used for every token, do not work this way. A slower drive means slower answers, because the drive is the limit.

Run it

sudo dnf install -y git gcc-toolset-13
git clone https://github.com/sangharshadhyeta/Strata-HotRod.git
cd Strata-HotRod
scl enable gcc-toolset-13 -- ./setup.sh --model IQ3_S --context 131072 --build --yes --no-start

Run it inside a memory limit (94% of RAM; two uncapped runs once froze the test PC):

systemd-run --scope -p MemoryMax=$(awk '/MemTotal/ {printf "%dK", $2 * 0.94}' /proc/meminfo) -p MemorySwapMax=0 \
  .venv/bin/python serve/server.py --engine strata --config strata-iq3_s.json --port 8095

Setup for the DeepSeek branch is in the README on the upstream-dsv41 branch.

Not tested

Other GPUs, AMD, Windows, macOS, other model sizes, several requests at once and two cards.

Project page · Run a 125B model on a 12 GB GPU · Run MoE models from an SSD · GitHub

Strata-HotRod is a fork of Strata by Niko1221 (MIT). All numbers were measured on one PC.