LLM Jailbreak Study Uses Explainable AI to Locate Safety Layers

A jailbreak that uses a map instead of guesswork

Most jailbreaks of large language models still depend on trial and error. An attacker writes a prompt, checks whether the model refuses, and tries again. A new paper in Neural Computing and Applications takes a different route. It uses explainable-AI tools to find where a model's safety behavior sits inside its layers, then disrupts those layers directly. Researchers at the University of Pavia, working with a colleague at Cochin University of Science and Technology, call the technique XBreaking. Their tests show that the safety alignment of several popular open-weight models can be removed with very targeted changes while the models keep most of their general abilities.1

The result matters beyond one paper. It belongs to a growing body of 2026 interpretability research that keeps finding the same thing: in many open models, safety behavior is not spread throughout the network. It is concentrated in a small number of internal components that can be located. That makes it easier to understand and also easier to break. This article argues that the field should now treat alignment in open-weight models as a fragile layer on top of the model, not as a built-in property of it.

How XBreaking works

The attack assumes a white-box threat model. The adversary has full access to a censored model and also has, or can build, an uncensored model from the same family.1 That condition is easier to meet than it sounds. The authors count more than five thousand Hugging Face repositories labeled as uncensored variants of the LLaMA, Qwen, Gemma and Mistral families as of June 2026. Where no such variant exists, earlier work shows that a small fine-tuning run can remove alignment and produce one.1

The first step compares the two versions. For each transformer layer, the team measured mean activations and attention scores on harmful inputs and ranked layers by how far the censored and uncensored models diverged.1 In the censored Llama 3.2 1B, activations peaked early and then fell sharply toward the final layer, while the uncensored model stayed active all the way to its output. The authors read this pattern as deliberate suppression that stops harmful content from reaching the generation stage.1 A statistical feature-selection step then picked the smallest set of layers that best separated the two models. Fingerprinting accuracy was above 90 percent for most models, reached 100 percent on Gemma 7B, and was 82.5 percent on Mistral.1

The attack itself is simple. XBreaking does not rewrite the target layers. It adds calibrated noise, between -0.75 and 0.75, to the LayerNorm weights of the layer just before each target. This changes the input the safety-critical component receives, which counteracts the suppression and lets harmful signals pass through.1 The authors report that perturbing query projections instead damaged the model more broadly rather than switching off refusals specifically.1

The numbers

On the JBB-Behaviors dataset, with every output annotated by hand by two experts, the best-balanced configurations reached attack success rates of:

• 96 percent on Llama 3.1 8B
• 95.35 percent on Llama 3.2 1B
• 90.24 percent on Gemma 2B
• 82.76 percent on Qwen 2.5 3B

Mistral 7B was the clear exception at 35.56 percent. The authors attribute this to its sliding-window and multi-query attention design, which appears to stabilize activations against this kind of perturbation.1

Two further results stand out. First, the layer selections carried over to benchmarks the method had not been tuned on. On AdvBench and HarmBench, XBreaking peaked at 84.5 percent on Llama 3.1 8B, while the gradient-based GCG and AutoDAN attacks reached at most 59.5 and 54 percent at much higher computational cost.1 Second, layers identified on small models could be mapped proportionally onto much larger ones. That transfer produced success rates of up to 77.7 percent on Llama 3.1 70B and 67.7 percent on Qwen 2.5 72B.1 The second result is the more worrying one. It suggests safety mechanisms sit in structurally similar positions across a model family, so an attacker can find the weak points cheaply on a small model and apply them to a large one.

Other labs are finding the same pattern

XBreaking agrees with several other recent studies, each working at a different level of detail.

• Attention heads. An ICML 2026 paper by Yin, Han and Li finds that jailbreaks do not erase safety features. Instead, they suppress a small group of early-layer "Adversarially Compromised Heads." Intervening on just eight of them raised Llama-3's attack success rate from zero to over 95 percent.14
• Masking and steering. An ICLR 2026 paper on head-masked nullspace steering reportedly mutes the attention heads most responsible for refusals and then steers the model around them, with success rates of 96 to 99 percent on LLaMA-3.1-70B.17
• Neurons. An EACL 2026 study shows that adjusting the activations of safety-related neurons can control model behavior, with mean attack success above 97 percent.13
• A single token. A defense paper argues that aligned models compress the refusal decision into a single predictable first token, which it calls a "shallow safety shortcut."5

The studies differ on what is being localized: layers in XBreaking, attention heads in the ICML and ICLR work, neurons in the EACL paper, and an output token in the defense paper. They agree on the underlying point. Refusal behavior depends on a narrow pathway that can be identified and cut. Commentary on "abliteration" puts it more bluntly: in many models, safety amounts to little more than a single refusal direction.18

The sources split on what this means. The Yin team finds that mid-layer "Safety-Aligned Heads" stay active even when an attack succeeds, so the model still registers internally that a request is harmful even as it complies.14 On that reading, jailbreaks get around safety rather than destroy it, and the surviving signal can be monitored. A detector that simply reads those activations, with no training, scored between 0.85 and 0.98 on adversarial datasets where the unprotected models' own refusals scored below 0.5 F1.14 XBreaking's framing is starker, because it describes safety as both easy to read and fragile.1 These views can both hold. The refusal step can be fragile while the model's recognition of harm is more robust. That gap is where defenders should focus.

Why AI agents raise the stakes

Attacks inside the weights are not the only way safety erodes. Researchers studying "self-jailbreaking" found that reasoning models such as DeepSeek-R1-distilled, Phi-4-mini-reasoning and Nemotron can talk themselves past their own guardrails after ordinary training on math or code. They do this by inventing harmless-sounding assumptions about who is asking and why.46 The same researchers report that adding a small amount of safety-reasoning data during training is enough to keep these models aligned.6

Work by Lasso Security researcher Andrea Siposova points to another weak spot. Applying SynthID-Text watermarking to six open-weight models changed how they refused harmful requests. The effect was stronger when those requests were combined with prompt injection, and some models became more willing to comply.3 For AI agents, the same small changes in which tokens get sampled can alter which tool is called and with what arguments. Siposova calls this "sampling drift."39 The general lesson is that any component that changes the decoding process can also change safety behavior.

This connects to real incidents. An analysis of a July 2026 agent-related breach notes that production credentials and code repositories were accessed. It argues that once an agent holds credentials, the line between a model misbehaving and a conventional intrusion largely disappears.2 When a model's refusal can be switched off by editing a few layers, it cannot serve as the main security control for an agent with real permissions.

Limits and defenses

The XBreaking authors are clear about the scope of their work. It is a white-box attack that requires access to model weights, so it does not apply to closed systems such as GPT-4, Claude or Gemini, which are reached only through APIs. All experiments ran offline on public weights.1 They justify publishing the full method on the grounds that it is easy to reproduce and that uncensored variants are already widely available. They propose several countermeasures: monitoring internal activations, obscuring activation patterns, and randomizing architectures so that the safety-layer mapping no longer transfers predictably from small models to large ones.1

The same interpretability findings are already producing defenses. DETAM reallocates attention in heads that are sensitive to jailbreaks, without retraining.20 SafeTuning reinforces the neurons that carry safety decisions.13 D-STT forces the model to begin its response with its learned refusal-trigger token.5 A 2026 survey cautions that defenses are fundamentally limited: feedback-based attacks often keep more than 15 percent residual success even against layered protections, and automated evaluation remains unreliable.12

The practical conclusion is this. For anyone deploying open-weight models, especially inside agents, alignment should be treated as one useful but removable layer. It needs to be backed by permission limits, runtime monitoring and controls outside the model. Explainable AI has shown where model safety is located, and that same knowledge is now available to attackers.