solid.cpp

Run it the way it was measured

Every feature on this page, enabled together. This is the exact command the numbers below were taken from — a 26B MoE model at 64k context on a single 12GB RTX 3060.

GGML_CUDA_MOE_CACHE_TRAIN_PREDICTOR=1 \
GGML_CUDA_MOE_CACHE_SUBSTITUTE=1 GGML_CUDA_MOE_CACHE_SUBSTITUTE_MIN_RANK=4 \
llama-server \
  -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf \
  -md mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
  --spec-type draft-mtp -bs --jinja \
  --moe-cache auto \
  -c 65536 \
  -ctk q8_0 -ctv q8_0 \
  --expert-atlas-file expert-atlas-evolve.json \
  --host 0.0.0.0 --port 8080

Measured on that command (predictor/substitute env vars not included in this specific run): ~64 tok/s generation warm (61.8–66.3 across runs), 4 concurrent slots. The first request after startup runs nearer 35 tok/s and climbs as the expert cache fills — the steady-state number is the honest one to plan around, and the cold one is the honest one to expect. The predictor and substitute numbers above are real but come from a separate hardware/model pairing (Ornith-1.5-35B) - not yet re-measured on this exact gemma-4 command.

Two optional extras, deliberately not in the line above. --cache-disk DIR persists the prompt cache so a restart costs 271ms instead of 5.8s — but it writes prompt content to disk (owner-only, model-fingerprinted), so it is opt-in on purpose. And GGML_CUDA_VMM_KV=1 reserves KV lazily, committing only what a conversation actually reaches.

Capable inference on hardware people actually own

Running a genuinely large model used to mean a rack of enterprise GPUs, or accepting whatever a hosted API gives you. The core unlock here — a GPU-resident cache for CPU-offloaded MoE experts — means a 26B-parameter MoE model that can't fit in a 12GB card doesn't just technically run: it runs fast, on a single consumer GPU. That's the whole point: put capability that used to require a data center within reach of hardware someone actually has on their desk.

But a fork that just claims speedups is exactly as trustworthy as any other unverified benchmark. So the second half of the project is making sure every claim can be checked: numbers are measured before/after on real hardware and reported honestly even when they're a loss (see the FR-Spec row below), and the cache itself is watchable live — not "trust me, it's caching," but a real-time view of which experts are hot right now. Prove it, then ship it, then let you go verify it yourself. Not a fork chasing every upstream feature — one willing to slow down and prove each claim before it ships.

Scope, honestly: the expert-cache fits the model's parameters into limited VRAM - it doesn't and can't stretch VRAM to hold arbitrary context length, which is a separate resource that scales with sequence length regardless of how the weights are cached. Pushed this to a real boundary on our own 12GB card: requesting 1M context used to hard-crash the whole process rather than fail gracefully - found and fixed that (--fit now searches by doubling from a safe floor and lands on a real, measured context size instead of guessing or crashing). A physical ceiling still exists - ask for 1M on a 12GB card and you will not get 1M - but it sits much higher than we first reported, and finding out why was the more useful result.

The first version of that fix asked the wrong question. It picked a placement, then shrank context until the model fit around it - so a 12GB card landed on 8192 tokens/slot, and we wrote that down as the hardware ceiling. It wasn't. The expert placement had been fixed first (by a command-line flag nobody revisited), and context was quietly paying for it. Inverting the order - decide placement from the context actually asked for, pushing more expert weights to CPU RAM until the request fits - gives 65,536 tokens across 4 concurrent slots on that same card and model, 8x the context and 4x the concurrency, at higher throughput than before. Nothing about the hardware changed; the program simply stopped trading away the one thing the user explicitly asked for. The lesson we'd pass on: when two automatic systems each optimise locally, check what the pair does to the number the user actually cares about.

What this actually buys, on one 12GB card

Every number below is from the same machine - an RTX 3060 with 12GB, a 26B MoE model, a SATA SSD - measured before and after, and reported whether or not it flattered us.

Context that follows the request, not the flag. Placement used to be chosen first and context shrunk to fit around it; now the requested context is fixed and expert placement gives way to it. Same hardware, 16× the context and 4× the concurrency.

Restarts stop costing a re-read. The prompt cache now persists to disk keyed by content hash, so a 3,600-token document that cost 5.8s to process is restored in 0.27s after a full restart. Files are owner-only and model-fingerprinted - they contain your prompts, and the docs say so plainly.

KV cache quantization that pays for itself twice. Symmetric q8_0 halves the conversation cache (2,180 → 1,158 MiB), and because placement is derived from measured memory rather than a fixed flag, the freed gigabyte automatically becomes two more layers of experts on the GPU - no code, no tuning. Verified lossless here: 6/6 byte-identical outputs at temperature 0, and full needle recall at 10/50/90% depth of a 24,700-token prompt.

And the ones that didn't work, because a fork that only publishes its wins is the thing this project exists not to be: pinning hot experts in RAM measured worse than doing nothing and ships disabled; page-cache readahead gained 5% on the first request and cost 8% thereafter, so it ships disabled too; and a claimed “+21% to +50%” speedup was retracted outright once measured properly - it was a single noisy sample, and the real effect is about 5%. The benchmark harness that caught it is in the repository.

A fork that measures its own claims

solid.cpp tracks llama.cpp upstream directly and layers on a curated set of performance and correctness fixes — triaged from real, reported issues, root-caused in code, and validated with actual measurements before anything is kept. Every change here is either a measured win or a documented, honest "no": a fix that was tried, found not to be the cause, and kept anyway if it was independently valid, or reverted if it wasn't.

The starting focus was MoE expert-cache placement and speculative decoding, but the scope covers the whole inference hot path — kernels, batching, KV-cache, sampling, quantization — and, increasingly, whether you can actually see what any of it is doing while it runs.

Big MoE models don't just fit here — they run fast

Every number below is a real before/after on the same hardware (RTX 3060, 12GB VRAM — about as constrained as it gets for a 26B-parameter MoE model), same prompt, same config except the one thing being measured. Chat-templated and content-verified, not a synthetic benchmark.

The throughput pipeline

Every row below is a real tok/s (or pp2048 tok/s) before/after on this hardware, ordered smallest gain to largest so the table reads as a rising case, not a grab-bag — each row is still its own independent A/B, not a single chained run, since these were measured at different points in the session's real history and some are pp2048 (prompt processing) rather than tg (generation).

Everything else: capacity, correctness, and honest losses

Not every real result is a tok/s number — some are context/VRAM capacity, hit-rate improvements too small to move tok/s yet, live-found bugs, or losses reported because a fork that only publishes wins is the thing this project exists not to be.

The Brain view: watch the cache, don't just trust it

A cache you can't see into is a black box you have to take on faith. The Brain view polls the running server and renders, live, which experts are actually resident in GPU memory right now — cold (never cached), warm (probation), or hot (protected) — with real-time hit flashes as tokens route through them.

Merged into the same view: the Atlas — each expert positioned by measured topic affinity (code, math, law, medicine, and five other categories, probed with greedy decode and a replication gate so one lucky prompt can't fake a specialist), colour still driven by live cache state. Position from measurement, colour from reality, one canvas. Cells flash yellow when serving as a live substitution stand-in — a resident expert doing someone else's work rather than the router's own missed pick paying CPU fallback.

The Atlas — a good map, and proof the system can't tell

Every expert gets a position from measured topic affinity: probed offline across nine categories with greedy decode and a replication gate, so one lucky prompt can't promote an expert to specialist. The map is genuinely predictive. The finding that matters is that being predictive doesn't reach the running system at all.

As a map, it holds up. Tested on link prediction — “do these two experts fire together?” — against embeddings factored directly out of the observed co-activation graph, on 31.6× more data than the atlas ever saw:

The probe atlas predicts co-firing better than an embedding built from co-firing itself, having never seen that traffic — a result from a test designed to be fair to the alternative. Two successive attempts to replace the nine hand-chosen categories with something label-free both lost.

That 2% figure is from the original design; a later rebuild (free-slot-only admission, a wider candidate pool, burst-mode firing on a detected topic shift) touches roughly 22% of decisions instead, and fixed two designs measured actively harmful (−0.31pp, −6.55pp) along the way. Current honest state: safe, not harmful, but still short of a consistent win — post-switch pooled hit-rate delta of −0.35pp overall, with individual topics mixed rather than uniform (one, law, is a real +0.40pp win, 3 of 4 rounds). Ships off by default either way.

Why the ceiling is that low is the deeper answer: atlas warming gains +0.31pp hit rate in simulation, while a topic oracle with perfect foreknowledge of the segment's experts gains +0.30pp. The atlas is already at the ceiling. Every topic touches ~6,000 of 7,680 experts and the top 1,000 cover only 52–61% of selections — there is no compact topic hot set to preload, so no prefetcher of this shape can win. One result explaining an entire family of refuted experiments is worth more than the feature would have been.

So the atlas ships as what it demonstrably is: the spatial layer of the Brain view — position from measurement, colour from live cache state — with its warming path rate-limited, admission-gated and off by default. Related reversals were reported the same way: combining atlas admission with eviction-weighting went net negative (−0.95pp, 1 of 4 rounds) against admission alone (+2.75pp, 4 of 4); and hit rate itself turned out to be purchasable — 79.25% is available for one environment variable and a slower server, which is exactly why throughput, not hit rate, is the objective here.

Sparse activation: not every neuron in a hot expert earns its VRAM

Caching decides which experts live in GPU memory. But an expert is not atomic — inside one, some neurons carry almost all the signal and others contribute essentially nothing. This fork measures that directly, per expert, and serves the ones that matter at reduced width.

The measurement came first, and it changed the design twice. Concentration is scored by value, not rank: “top 25% of neurons” picks a fixed cutoff regardless of what that expert's distribution actually looks like, so instead each neuron is compared against a threshold relative to its own expert's maximum, and the resulting count falls out as an observation rather than an input. Dead-neuron fraction is anchored on the median, not the mean — these distributions are skewed enough (max/mean ratios of 120–270× measured) that the mean is not a trustworthy reference.

An early read of the per-layer medians said layer 0 was uniquely concentrated (30.9% dead neurons vs ~0% everywhere else) and the mechanism was nearly scoped to that one layer. Checking at the individual-expert level instead found 17 more outlier experts scattered across layers 1, 2, 3, 4, 37 and 39, some on large sample counts. The layer median had been hiding a real per-expert effect that exists everywhere, just unevenly. The shipped mechanism is layer-agnostic because of that check, not despite it.

Converted experts are served from a second, narrower cache pool: gate/up only (down_exps's row axis is an output dimension, where dropping rows would delete real outputs rather than approximate sparsity), the chosen rows gathered once at fill time, and a small CUDA kernel scattering the computed values back into their correct positions in an otherwise-zeroed row. The CPU path never learns any of this exists. Measured 50% VRAM saved on every converted expert, and the reduced pool evicts under the same heat-driven LFRU policy as the primary one, so it keeps tracking whichever experts are currently hot instead of freezing at whoever converted first.

The safety margin that was quietly costing 30%

Chasing why the neuron work only bought 8%, the utilization numbers said something else was wrong: CPU at ~5.4 of 12 cores, GPU at 41%, and ~3 GB of a 12 GB card unused. Neither resource saturated means the bottleneck is serialization, not capacity.

Placement reserves 3 × --fit-target of VRAM before deciding anything, and that reservation silently raises a requested -ncmoe until the context fits. With the 1024 MiB default that is 3 GiB held back — and it invalidated an entire sweep before being noticed: requesting 22, 16 and 10 CPU layers all produced identical memory use and near-identical throughput, because all three were being raised to 27. The log line saying so had been there the whole time.

+11.7% from the margin alone (~+30% against the starting configuration), every candidate passing its correctness check, on Ornith-1.5-35B. The margin cannot simply be deleted — it covers allocation that happens after the fit probe can see it, and removing it reproduced a hard context collapse — so the honest answer is to measure it per machine rather than ship a constant.

Every fix, one card each

Inherited from upstream, unchanged

The core goal of llama.cpp — minimal-setup, state-of-the-art LLM/VLM inference on a wide range of hardware, locally and in the cloud — carries through untouched:

• Plain C/C++ implementation without any dependencies
• Apple silicon as a first-class citizen — ARM NEON, Accelerate and Metal
• AVX, AVX2, AVX512 and AMX for x86; RVV/ZVFH/ZFH/ZICBOP/ZIHINTPAUSE for RISC-V
• 1.5-bit through 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for NVIDIA GPUs, plus AMD (HIP) and Moore Threads (MUSA) support
• Vulkan and SYCL backends; CPU+GPU hybrid inference for models bigger than VRAM
• Newest speculative-decoding drafters — --spec-type draft-dflash (DFlash, block-diffusion parallel drafting, up to 6× lossless speedup) and --spec-type draft-dspark (DSpark, DeepSeek's confidence-scheduled semi-autoregressive Markov-head drafter, 16–18% higher accepted length than DFlash) — full convert-script and README coverage in docs/speculative.md. Not something this fork ported: real, working, upstream-inherited support, checked and confirmed present rather than assumed. DFlare (scales draft capacity further past DFlash) is a newer successor worth watching, not yet implemented anywhere.

Built on top of ggml. See upstream's own manifesto and API references for lib llama and llama-server REST.

Running in two commands

cmake -B build -DGGML_CUDA=ON   # or your backend of choice
cmake --build build --config Release -j

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch an OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

# A MoE model too big for VRAM - the expert-cache does the work
llama serve -hf <big-moe-model-gguf> -ngl 99 -ncmoe 15 --moe-cache auto

Docker: .devops/*.Dockerfile builds from this source tree, so it includes every fix on this page - the pre-built ghcr.io/ggml-org/llama.cpp images in the Docker docs are upstream-only and won't have them. Full guide either way in docs/build.md.

VLM session with llama cli

Brain view - live expert atlas and moe-cache placement

Runs where llama.cpp runs

Per-backend build flags in docs/build.md.

The mmap-prefetch-safety fix behind the GLM-5.3-Flash result above (skips eagerly populating the whole model file into RAM when it won't fit) reads /proc/meminfo directly, so it's Linux-only. It doesn't crash or fail to build elsewhere - Windows/macOS fall back to ggml's own device-memory query, which reports installed capacity rather than what's actually free, so the fix is less accurate (not absent) at catching the same OOM there. Verified on Linux; not yet verified on other platforms.

Everything, one click away

Same history, extra commits

This repo's git history is llama.cpp's history, plus the commits on top. Upstream changes are pulled in directly rather than re-implemented; nothing here is intended to diverge from upstream's own architecture or conventions more than a given fix requires. See upstream's README and manifesto for the base project this builds on.

Open to real fixes

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the repo and merge PRs into the main branch
• Any help with managing issues, PRs, and projects is very appreciated
• Read upstream's CONTRIBUTING.md for more

MIT, same as upstream

Real sources for real ported work, not vendored libraries but genuine prior art this fork builds directly on:

• leloch/llama.cpp (moe-cache-pr branch) — original design for the GPU-resident MoE expert-cache this whole fork is built around. The core feature, not a footnote.
• thecodacus/llama.cpp (fable5/prefetch-experts branch, via the Codacus YouTube channel) — expert-prefetch + lifetime host-pinning, overlapping expert-weight uploads with compute instead of blocking on them. Cherry-picked and re-measured on our own hardware: +36.0% prefill (1403 → 1908 tok/s, RTX 3060, Gemma-4-26B-A4B).
• JustVugg/colibri — LFRU eviction design (probation/protected segments, heat-based promotion) and the Expert Atlas topic-affinity measurement methodology (labeled probes, replication gate, specialization scoring) the Brain/Atlas view's positioning is built on. Its two-step shared-expert router prediction experiment also saved us building a predictor: it reports the best recall in this space (76.7%, beating PILOT) and no end-to-end gain, because cache capacity bound first — a result we confirmed applies to us before writing any code.
• FlashML-org/FreeToken — the hit/miss split for MoE prefill weight uploads (device-to-device gather for experts already cache-resident, one batched transfer for the rest, instead of always paying full PCIe cost). Design taken, not code: FreeToken's own version runs as a from-scratch engine with a single unified expert cache, so we re-derived it against our existing LFRU pool and the native sparse-copy path in ggml_backend_sched_compute_splits rather than porting its CUDA kernels directly — the residency check and device-to-device copy are ours, keyed off the same cache the decode path already fills.

Prior art that shaped decisions here without code being taken — including several that told us not to build something:

• antirez/ds4 (DwarfStar) — the three-tier design this fork independently arrived at (resident non-routed weights, a budgeted routed-expert cache, GGUF on SSD for misses), plus the explicit read/write buffer approach we tested against our mmap-based one. Reviewing it corrected two of our assumptions and sized the remaining gap honestly.
• vLLM — PagedAttention. We deliberately did not port it: block tables and gather kernels would mean new attention kernels for every backend llama.cpp supports. Instead we took its actual saving — don't commit memory for context nobody uses — via CUDA's virtual-memory API, which keeps addresses contiguous so existing kernels are untouched. Its automatic prefix caching turned out to already exist upstream here.
• Speculating Experts, Cross-Layer Gate / PILOT / Fate, FineMoE, SP-MoE — the expert-prefetch literature, which established that the field solves this by predicting and prefetching in time rather than holding experts resident, and that realistic gains are ~14% rather than the order of magnitude we had been chasing. That correction is why three of our residency experiments ship disabled.
• Checked the literature's two standard explanations for why holding hot experts resident measures worse than doing nothing (cross-NUMA-node remote access; missing huge-page backing causing page-fault storms — up to ~1M faults loading one 4GB expert at 4KiB pages) against this specific machine before accepting either as the cause: it has one NUMA node (no cross-socket penalty possible) and transparent_hugepage=always already covers a plain malloc buffer without code changes. Both explanations are ruled out here, not confirmed by assumption — the negative result stands for a different, still-undetermined reason.
• MoE-Lightning and MoE-Lens — throughput-oriented MoE serving under GPU memory constraints; LMCache-style three-tier KV hierarchies (hot on GPU, warm on host DRAM, cold on NVMe) are the closest published analogue to this fork's own NVMe-migration result and GreedyDual-Size cost-tier eviction, though those target KV-cache blocks, not expert weights — the natural next step if expert-weight NVMe residency is ever built as a real tier rather than left to the OS page cache.

Third-party libraries, inherited unchanged from upstream:

• yhirose/cpp-httplib — single-header HTTP server, used by llama-server — MIT
• stb-image — single-header image decoder, multimodal subsystem — public domain
• nlohmann/json — single-header JSON library — MIT
• miniaudio.h — single-header audio decoder, multimodal subsystem — public domain
• subprocess.h — single-header process launching — public domain