Hugging Face harness adapters target fewer agent tool calls
Hugging Face’s harness-hub team is expanding its work on small adapters that train a model for the tools and workflow of a particular coding agent. Its November 6 report describes reinforcement learning experiments across pi, OpenCode and Hermes, using LFM2.5-24B-A2B, LFM3-1.7B and experimental TwIL-8B.
The report attributes efficiency improvements to smoke tests and describes final runs being streamed. Readers should distinguish that ongoing work from the earlier downloadable checkpoints. Neither establishes a general improvement across coding workloads.
What developers can download
The published LFM2.5 collection contains adapters for pi, Claude Code and Hermes, with corresponding GGUF versions. These earlier releases do not constitute a complete release of the newer three-model experiment. An adapter also needs its compatible base model. It is not a standalone replacement for the entire model.
The pi model card reports 39% fewer tool calls on tasks that both the adapter and base model solved. Its evaluation used 100 SmolDataEnvs data-analysis tasks, with one attempt per task. Accuracy was 4.2 percentage points lower, a difference the authors describe as within the evaluation’s noise.
That denominator matters. Fewer calls among shared successes does not establish lower total costs across every attempted task. The card also cautions that results may not transfer to other coding work or another harness.
The results depend on the harness
Hermes offers another encouraging but qualified result. Its adapter reduced tool calls on jointly solved tasks from 4.97 to 3.97, approximately 20%. Pass@1 moved from 73% to 72%.
The authors selected the released checkpoint using the same 100 tasks used to report its score. They therefore warn that the accuracy estimate is slightly optimistic. Its GGUF conversion has not been re-evaluated, so readers should not automatically carry those measurements over to a different serving configuration. These limits are documented in the Hermes adapter model card.
Claude Code provides a useful counterexample. Its adapter card presents the checkpoint as a reference run rather than a demonstrated improvement. The reported checkpoints remain within noise of the base model. Some evaluation entries are still marked incomplete.
Taken together, these cards support a practical approach: choose the exact model and harness pairing, then compare it against the base model on representative tasks. A result from pi is not evidence that the Claude Code adapter delivers the same improvement.
Broader training still has a narrow task mix
The new report adds environments reconstructed from shared agent sessions. However, 253 of its 270 tasks are build tasks, alongside 11 repairs, six extensions and no question-answering tasks. One source supplies 243 tasks.
That concentration limits what the experiment can say about the wider range of software maintenance. A collection dominated by building Python projects cannot, by itself, establish comparable gains in debugging mature repositories.
The related Repo2RLEnv project offers broader routes for creating training environments from repositories, pull requests and other material. Its documentation explicitly warns that a generated task still needs validation. Reference solutions, verifiers and controls determine whether a reward is meaningful.
For developers creating their own training set, environment quality is therefore part of the work. Automatically producing tasks does not remove the need to inspect what their tests actually measure. Model choice needs the same care: our LFM3 guide covers the base model’s intended uses, runtime requirements and license limits.
What safeagents protects
The safety component is a proxy between the harness and model server. Safeagents checks proposed tool calls before the harness receives them and replaces blocked calls with a notice. Its documented rules cover undeclared tools, unfamiliar destination hosts and scanner findings above a configured severity threshold.
This protection depends on configuration and detection coverage. The quickstart’s log-only mode records potential blocks without stopping calls. Lower-severity findings can also proceed, and the optional judge reviews suspicious calls rather than every action.
Developers sharing training traces face another limitation: safeagents describes its redaction as best-effort and asks users to review the report before publishing. It should not be treated as a guarantee that confidential material has been removed.
The useful next step is a controlled trial: measure successful completions, total tool use and elapsed time on your own workload, while retaining sandboxing and reviewing blocked actions. The published evidence justifies experimentation, not an assumption that every agent becomes faster or safer.
Featured image: conceptual illustration by ByteForward using the official Hugging Face logo. It does not depict a physical product.
