NVIDIA Dory brings adjustable compute to a small reasoning model
NVIDIA released Dory 1.0 on October 16 with four billion parameters and a way to spend more computation on each token. Its shared layers can run repeatedly instead of requiring a different checkpoint for each compute budget. The release offers an interesting option for developers experimenting with small reasoning models, although the main inference integrations are still under review. Model card.
What changes when a model loops
The released configuration defaults to two recurrent loops and sets the maximum position count to 262144. It combines sliding attention with full attention. These settings describe the model configuration rather than a guarantee that any particular machine can serve the maximum context at useful speed. Model configuration.
The practical question is how much work the model should do before producing its next token. Reusing parameters lets developers explore that question without swapping the checkpoint. It does not remove the need to measure latency, memory use and answer quality together. A smaller output is not automatically a faster answer when each generated token takes more computation. vLLM review.
NVIDIA recommends starting with two loops. Its reported results show that increasing the loop count does not improve every benchmark. Those measurements belong to NVIDIA, and the BF16 results should not be assumed to describe a quantized download. Model card.
Available weights are ahead of stable runtime support
The vLLM integration is an open pull request. Its implementation work began before the weight release and remains separate from a stable package release. Developers should check that the version they choose actually contains Dory support rather than assume that a generic installation command is sufficient. vLLM review.
This distinction matters for deployment planning. A public checkpoint gives developers access to the model, while a runtime integration determines whether their serving stack can load it correctly. Teams maintaining production services should treat a review branch as an evaluation dependency and keep the exact revision with their results. vLLM review.
The llama.cpp integration is still marked draft. Its author reports support for one through four loops and tests with BF16 and two quantized formats. The same report lists incomplete long context qualification and full continuous integration, with Metal untested. These are contributor observations rather than independent ByteForward tests. llama.cpp review.
The cache choice needs its own evaluation
The llama.cpp proposal offers separate caches for recurrent loops or a shared cache that retains the final loop. Its author explicitly warns that the shared option changes predictions and that the size of input chunks matters. That makes cache mode part of the experimental setup, not simply a memory switch that can be changed without checking the output. llama.cpp review.
A useful comparison would hold the prompt set, quantization, context length and runtime revision constant, then vary the loop count and cache mode. Record both answer quality and total response time. That separates a genuine improvement for the application from a change caused by another setting. The draft also excludes context shifting and loop counts above four. llama.cpp review.
Check the redistribution terms
Dory uses the OpenMDW 1.1 license. The agreement grants broad rights to the model materials, while requiring a copy of the agreement and applicable notices of origin when those materials are redistributed. The license also distinguishes model materials from generated outputs. Anyone packaging the weights into another product should read the agreement rather than treat a download button as the complete licensing information. License terms.
For developers following local inference changes, our coverage of the separate llama.cpp GPU update explains another part of the deployment picture. Dory changes the model and its compute choices, while that update concerns kernels used by supported low bit formats. Neither story establishes the performance of this new combination. Related coverage.
Illustrative DGX Spark photograph by Daniel Lu via Wikimedia Commons under Creative Commons Attribution ShareAlike 4.0. Existing reduced resolution rendition with no local edits.
