llama.cpp improves selected GPU inference paths for low bit models
llama.cpp merged a GPU optimization on October 17, 2026. Its benefits depend on hardware and quantization. This updates the runtime for existing models.
Where the reported gains appear
The author reports Qwen3.5 9B Q3_K_S generation rising from 81.04 to 126.22 tokens per second on an RTX 3090. On an RTX 4090, it rose from 162.65 to 174.93. ByteForward has not reproduced these measurements.
Compare the matching workload rather than treating an isolated kernel measurement as conversation speed. Regressions remain, including RX 6800 Q2_K at n equal to eight. The discussion leaves further architecture tuning for later work.
What changed inside the runtime
The code changes touch quantized vector operations and how work is assigned within GPU blocks. For some low bit formats, handling a second row lets the implementation reuse input loads and increase memory parallelism. The affected path includes Q2_K and Q3_K alongside selected other formats.
The merged work also contains a targeted adjustment for AMD GCN scheduling. This is a reason to inspect the actual backend and format rather than treating the CUDA label as a complete compatibility list. The patch changes computation scheduling and arithmetic implementation. It is not evidence that a model has learned new capabilities or that its outputs have been evaluated for quality.
Check which build you actually use
The merge is available in the project’s source history. A desktop application using llama.cpp may package its own version, so the upstream merge alone does not establish that your installed application includes it. Check its release information and runtime version before attributing a speed change to this patch.
The official build guide distinguishes NVIDIA CUDA, Apple Metal and other backends. CUDA builds require NVIDIA’s toolkit and an explicitly enabled CUDA configuration. Metal is enabled by default on macOS. A CUDA performance result therefore cannot be carried over to a Mac simply because both computers run llama.cpp.
Measure the workload you care about
The project’s benchmark documentation separates prompt processing from text generation and supports combined tests. Prompt processing measures the input stage. Text generation measures the output sequence. A fast result in one stage need not describe the other.
For a useful comparison, keep the model file, GPU offload, context depth and batch settings consistent. Record the runtime revision and repeat the test. The benchmark reports average tokens per second and variation between repetitions, with structured output available for keeping the underlying results.
There is a further limitation. The benchmark excludes tokenization and sampling time, so its throughput is not a complete measurement of the delay inside a chat application. Check an actual task as well, especially if your workflow includes long prompts or several concurrent requests.
Our coverage of Janas on Windows examines a different local inference constraint, fitting models around limited memory. Both stories reinforce a practical point. A runtime improvement is worth assessing against the machine, model and task you already use, rather than a single speed headline.
Illustrative graphics card photo by Back2Gaming on Unsplash, used under the Unsplash License. The pictured card is not identified as a benchmark test unit.
