AISI calls for AI safety controls before public release
The UK AI Security Institute set out a voluntary technical agenda on October 12 for managing frontier AI risks throughout development. Its central warning is that capable agents can create risks during training, evaluation and internal use, before a model reaches the public.
AISI wants developers and technical organisations to work together on practices that could eventually become international standards. The institute explicitly says it is not imposing requirements. The announcement is an agenda for developing safeguards, with important technical questions still open.
What the new agenda covers
The priorities include documenting risk, supervising systems throughout their lives, preserving oversight, reporting agent incidents and measuring progress in AI development. AISI also wants clearer expectations for independent evaluators, stronger measurements of security relevant capabilities and better tests of alignment.
What stronger evaluation controls look like
An earlier AISI engineering update from October 1 makes the agenda concrete. Following an incident disclosed in August, the institute paused its highest risk cyber evaluations. It said it had resumed most evaluation activity after an initial round of security changes.
Those changes included disabling internet access for agentic cyber evaluations and separately blocking outbound networking at the sandbox and cloud host levels. Automated checks confirm that required controls are active before a run begins. A monitor reviews agent activity before suspicious actions execute and can send them for human review.
AISI also described a staged testing process that moves from code analysis to supervised probing and controlled escape attempts inside additional containment. These are reported safeguards in AISI’s own environment. The institute says they reduce risk rather than eliminate it, and that stronger models will require continuing reassessment.
A monitor also needs testing
AISI’s July 23 account of its Control Red Team explains another challenge. A separate model can review an agent’s work, but the protection depends on when that review happens and what follows a warning. Reviewing reasoning after an action and checking an action before execution offer different opportunities to intervene.
The team reported testing internal monitors with Google DeepMind and Anthropic and finding weaknesses that helped guide improvements. Some failures involved the monitor’s instructions. Others involved its underlying model or rules that determined which actions reached review at all.
The researchers also identified a limit in their attack simulations. An automated attacker could repeatedly inspect monitor feedback in ways a deployed agent might not. Finding a weakness is therefore useful evidence for fixing a control, but does not by itself establish the likelihood of that failure in ordinary use.
Why oversight cannot remain fixed
A May 21 AISI report on oversight drew on 25 expert interviews, literature and analysis. It examined information from model internals, visible reasoning, external actions and communication between agents. Its concern was that the properties making these signals useful today may change as systems become more capable.
The report also recorded disagreements. Experts differed on whether hidden reasoning would replace readable reasoning, how much action monitoring could prevent harm and whether behaviours learned during oversight training would persist in deployment. It described existing methods as useful for current systems while arguing for ways to preserve them and develop alternatives.
Taken together, these documents put a practical question ahead of the next model release. Who can see what an agent is doing, who can stop it, and what evidence shows that those safeguards still work? The answer requires testing the surrounding system as well as the model.
For more context, read our earlier report on AISI’s evaluation network controls.
