Study finds limits in simple AI safety embedding scores
A new paper tests a simple safety scoring shortcut. Adding a reference improves ranking, while deployment limits remain substantial.
A new study of AI safety embeddings finds that a simple way to score model responses can confuse their subject matter with their safety. The method averages embeddings from known safe responses, then treats similarity to that average as evidence that another response is safe.
NYU researcher Sahil Kadadekar tested this raw centroid rule using four frozen encoders and three datasets. The first manuscript was submitted on October 1 and reports acceptance at the NeurIPS 2026 workshop on Foundations of Language Model Security.
Similar language can hide different behavior
Embeddings represent text as numerical coordinates. A harmless answer and an unsafe answer about the same topic can occupy nearby regions. Averaging safe examples identifies a location in that space, but does not necessarily establish which direction separates safe responses from unsafe ones.
The study compared that rule with a reference built from the difference between average safe and unsafe embeddings. On the two datasets using released human annotations, the raw rule scored between 0.457 and 0.545 in ROC AUC. The reference scored between 0.588 and 0.738 on the same embeddings. A value of 0.5 represents chance ranking.
Those ranges summarize particular dataset and encoder combinations. They do not measure the percentage of harmful answers blocked. The author also tested an auxiliary Aegis cohort labeled by an AI jury. Its results were stronger for the reference, but it is not an independent human replication and was not matched by prompt.
Better ranking still leaves an operational problem
The analysis grouped responses by prompt to reduce leakage and checked whether topic, response length or simple stylistic features explained the scores. This matters because a detector can appear effective when safe and unsafe examples happen to discuss different subjects.
At thresholds calibrated on validation data to target a 5% false safe rate, the reference admitted more benign responses on PKU SafeRLHF and Aegis. Gains on BeaverTails were uncertain. Even the referenced method accepted at most 9.5% of safe BeaverTails responses at that target. The target is a calibration setting, not a guarantee that a deployed filter meets it.
A reference drawn from unlabeled responses recovered some of the useful ranking, but worked less well when unsafe examples made up only 5% of that pool. A comparatively small labeled unsafe sample recovered most of the improvement in these tests.
A diagnostic result with clear limits
The findings concern English responses, four frozen encoders and one specific cosine scoring rule. The study does not establish failure of all methods trained on safe examples, specialized safety encoders or supervised guard models. Its author recommends the referenced score as a diagnostic baseline rather than a sufficient production guard.
The paper includes code and aggregate results in its arXiv ancillary files. ByteForward has not independently rerun the experiments. The study measures ranking and acceptance errors, not changes in model behavior, attack resistance or real world harm.
For teams evaluating lightweight monitors, the practical questions are how a reference was chosen, whether the evaluation controls prompt composition and how many harmless responses the operating threshold rejects. Our reporting on AISI evaluation network controls covers a different part of the same testing discipline.
Research by Sahil Kadadekar is available under Creative Commons Attribution 4.0. This article summarizes the paper in original language. Featured artwork is an original AI generated conceptual illustration.
