XBreaking Jailbreak Uses Explainable AI to Find LLM Safety Layers

A different kind of jailbreak

Most attempts to jailbreak large language models work like a locksmith with a ring of keys. Attackers write a prompt, see whether the model refuses, adjust, and try again. The researchers behind XBreaking describe this generate-and-test approach as the dominant strategy in existing jailbreak work, and their method takes a different path 1. Instead of probing the model from the outside, XBreaking uses explainable AI (XAI) techniques to look inside it and work out where its refusal behavior comes from 12.

The work comes from computer scientists at the University of Pavia, together with a collaborator at Cochin University of Science and Technology 2. The method compares two versions of a model: one that has been safety-aligned, or "censored," and one that has not. By analyzing how their internal behavior differs, the researchers identify alignment patterns that are unique to the censored version and can therefore be exploited 1. The attack then injects targeted noise into the parts of the model that carry those patterns, with the aim of disabling its safety constraints 1.

Three questions, three yeses

The study is published in Neural Computing and Applications and is also available as a preprint on arXiv. It is organized around three research questions, and it answers each one in the affirmative [2]:

• Can censored and uncensored models be told apart by their internal activity? According to the study, they can be distinguished reliably.
• Can specific layers be identified as the main carriers of safety behavior? The researchers report that they can.
• Can calibrated noise in those layers suppress refusals without breaking the model? The study finds that the refusal mechanism can be suppressed while the model's general abilities stay largely intact.

The last point carries the most weight. A crude attack that makes a model stop refusing but also leaves it incoherent would be of little practical concern. The reported result is more precise than that: the safety behavior can be removed while the model's usefulness is mostly preserved 2.

Same work, different emphasis

The two accounts of the project describe the same technique but frame it differently. The arXiv abstract presents the XAI analysis partly as a way to understand censoring mechanisms better, with the attack following from that understanding 1. Coverage at bioengineer.org leans into the offensive side, describing safety alignment as something that can be "surgically dismantled" and stressing that the method locates where censorship "lives" inside a transformer 2. Neither framing is wrong. The difference reflects a familiar tension in security research, where a diagnostic tool can also serve as a weapon.

The scope also matters. The secondary coverage specifies that the work concerns open-source models 2. That fits the method's requirements. Comparing internal activity and injecting noise into particular layers appears to require direct access to a model's weights, which closed, API-only systems do not provide. This is a white-box attack, not a clever prompt anyone could type into a chatbot.

Why it matters

In our reading, XBreaking's main contribution is evidence about how safety alignment is structured, more than any new capability for bad actors. People who download open-weight models have long been able to fine-tune away refusals. If the study's findings hold up, though, the safety behavior added during alignment is not spread throughout the network. It appears to sit in identifiable layers, where a small and targeted intervention can neutralize it. That suggests alignment in these models may be a fairly thin layer on top of the underlying capabilities, not something deeply built into them.

This has implications for both sides. For attackers with weight access, it points to a cheaper and more principled path than repeated prompt trial-and-error. For defenders, the same XAI analysis could serve as an auditing tool. Developers could check whether their alignment is concentrated in a few fragile spots and try training approaches that spread safety behavior more widely, so that it is harder to remove without also damaging the model. Whether that kind of distributed alignment is achievable is an open question that this study raises but does not settle.

It also adds to the ongoing debate over releasing open weights. Supporters of open models argue that transparency enables exactly this sort of scrutiny, and XBreaking is an example: researchers could not have mapped safety layers without inspecting the internals. Critics will point out that the same transparency makes safeguards easier to remove.

The takeaway

XBreaking turns interpretability tools, often presented as a route to safer AI, against the safety mechanisms themselves 12. The more useful lesson is less about this particular attack and more about what it shows. In the open-source models studied, refusal behavior appears to be localized, detectable, and removable without much collateral damage 2. Alignment that can be found this precisely can also be removed precisely, and developers who rely on it as a final safeguard should take that into account.