XBreaking Jailbreak: Explainable AI Exposes LLM Safety Layers

What happened

A research team from the University of Pavia, working with a collaborator at Cochin University of Science and Technology, has described a jailbreak technique called XBreaking. It targets the safety alignment of open-source large language models from the inside, rather than through clever prompts 2. The work appears in Neural Computing and Applications. Instead of the trial-and-error prompt engineering behind most jailbreaks, the researchers used explainable AI (XAI) methods to find where inside a transformer the refusal behavior actually lives 2.

The study poses three questions and answers yes to all of them [2]:

• Censored and uncensored versions of a model can be reliably told apart by their internal activity.
• Specific layers can be identified as the main carriers of safety behavior.
• Injecting carefully calibrated noise into those layers can suppress refusals while largely preserving the model's general abilities.

The core claim is that safety alignment is not spread evenly through a model. It is concentrated enough to be located and disrupted on purpose.

A second look at the same attack surface

XBreaking fits alongside other recent work treating a model's internal states as an exposed surface. A USENIX paper titled "Depth Gives a False Sense of Privacy" looks at a different threat: reconstructing a user's original input from a model's intermediate representations 1. Its authors propose four inversion attacks covering both white-box and black-box settings. They test them at layers 2, 8, 16 and 24 of a 32-layer Llama-2 1.

The results undercut the idea that deeper layers are safely abstract:

• One attack, labeled ER, achieves high similarity scores at shallow layers 1.
• Another, TBS, stays effective as deep as layer 24 1.
• Deeper inversions sometimes keep the meaning of the input but introduce noisy tokens. In one example, "2/2/2017" came back as a garbled string, which lowers exact-match rates without stopping the leak of meaning 1.

The two studies have different goals. XBreaking tries to change what a model will output. The USENIX work tries to extract what was put into it. Both rest on the same premise: once someone can observe or manipulate a model's internals, its layers stop being a black box and become a map.

Noise cuts both ways

The most interesting overlap between the two papers is the role of noise.

• In XBreaking, noise is the weapon. Targeted perturbation of the safety-relevant layers is what switches off refusal behavior 2.
• In the USENIX paper, noise is the defense. The authors evaluate quantization, dropout, noisy input embeddings, pruning, and differential privacy using the Laplace mechanism 1.

The defense findings are sobering. Under some measures, inversion was barely affected 1. Noise strong enough to block the attack, at scales of 0.05 and 0.1, also drove the model's MMLU benchmark performance down to random-guess level 1. Protection was bought by making the model nearly useless.

Read together, the papers suggest a narrow middle ground. Perturbing internal states can protect privacy, but only by damaging capability. Perturbing them precisely, in the right layers, can remove safety without damaging capability. The attacker can afford to be surgical. The defender, at least with the generic noise-based methods tested, cannot.

Why it matters for open-source models

XBreaking is aimed specifically at open-source models 2. That matters because the technique assumes access to model internals, which open-weight releases provide by design. Anyone who downloads the weights can run XAI analysis, compare activations, and modify specific layers. No API filter or server-side monitoring stands in the way.

This does not make open models uniquely dangerous. It does clarify what alignment on an open-weight model really is: a property of particular parameters that can be found and changed, not a lock. The finding that safety behavior is concentrated in identifiable layers 2 is a useful diagnostic for researchers. It is equally useful to anyone who wants to strip that behavior out.

The takeaway

My reading is that these papers together argue against treating either depth or alignment as a security boundary for models whose internals are accessible. The USENIX results show that deep layers still leak recoverable input information 1. XBreaking shows that safety training can be localized and neutralized without the usual cost to model quality 2.

The defenses tested so far, at least for the inversion problem, impose steep utility costs at the strengths needed to work 1. For developers releasing open-weight models, the practical lesson is that alignment applied to the weights should be treated as a default behavior, not a guarantee. Stronger safeguards will likely need to come from deployment-level controls, or from alignment methods that are spread more robustly across the network rather than concentrated in a few layers.