llama.cpp CUDA fusion: what the prefill results mean
llama.cpp merged a targeted CUDA prompt-processing optimization on October 25, 2026. It fuses dense feed-forward gate/up operations and activation. Downloadable-release and application-bundle availability remain unverified.
Workload restrictions
The implementation requires NVIDIA Turing MMA support, Q4_K, Q5_K, Q6_K, Q8_0 or IQ4_XS weights, F32 inputs and compatible tensor shapes and strides. Its 90 percent tiling-efficiency threshold is an eligibility rule, not a promised speedup. Mixture-of-experts support is unfinished in this path. Other selection checks apply.
Contributor-reported prefill results
Qwen3.8 27B Q4_K_M measurements used a 16384-token prompt, batch and microbatch sizes of 2048, F16 caches and five repetitions. ByteForward has not reproduced them.
| Hardware | Baseline, tokens/s | PR, tokens/s |
|---|---|---|
| DGX Spark | 840.1 | 899.9 |
| RTX 5090 | 3688.4 | 3790.9 |
How to evaluate it without confusing the measurements
The project’s benchmark documentation separates prompt processing, token generation and a combined test. It reports repeated run averages and standard deviation, with individual repetitions available in some output formats. Tokenization and sampling are excluded. Consequently, prompt throughput alone does not measure the complete wait for a response.
For a useful comparison, record the exact revision and keep the model file, quantization, GPU offload, batch sizes and cache types unchanged. Compare prompt processing separately from generated tokens. Include the prompt lengths and context depths that matter to your workload rather than treating one long prompt as representative of every interaction.
Keep individual repetitions when possible and examine the spread alongside the average. A small difference without its variability is difficult to interpret. Measure application response time separately if that is the outcome you care about. This is a suggested evaluation method, not a benchmark we performed.
For now, the useful decision is whether your actual workload merits a controlled comparison. Do not translate a prefill result directly into a promised improvement in typing speed, complete response latency or every stage of a local assistant.
For a different optimization, see our llama.cpp coverage of low bit GPU inference.
AI-generated conceptual illustration by ByteForward, with the official NVIDIA logo from its brand resources. Not a hardware photograph or benchmark visualization. NVIDIA is a mark of NVIDIA Corporation. No endorsement implied.
