Preference tuning erodes cyber safety, and internal probes restore it
A four-step walk through the machinery: catching a safety regression that preference tuning introduced, reading the internal signal behind it, wiring that signal into the training loop, and measuring what the repair costs.
In short
Preference tuning is the standard last step in building a chat model, and it does what it promises: it makes models more helpful. On this 8-billion parameter model it also quietly gave away cyber safety. Safety fell from 0.8376 to 0.7947 and the share of attack content the model withheld fell from 0.5160 to 0.4660, while benign compliance and accuracy rose. Nothing in the training signal flagged the loss.
The repair reads the model's own internals. Two probes, one tracking whether the model has recognised a request as harmful and one tracking whether the topic is cyber at all, are wired into the training loop and used to shape preference pairs while the model learns from them. Shaping with the refusal probe is the most protective, reaching 0.9174 safety, and the most restrictive: benign compliance drops to 0.8320, below the 0.9000 the model started at. Restricting the nudge to cyber-flagged examples holds safety at 0.8602 and returns compliance to 0.9480.
Accuracy moves at most 0.0087 across the interventions measured, inside the 0.0155 band that separate runs of the same setup differ by anyway. One model family at one size.
Step oneMeasure what post-training actually changed
The capability on show is catching a regression nobody was looking for, by evaluating the same model before and after preference tuning on four axes at once rather than on the one the tuning was aimed at.
Preference tuning is the standard final step in building a chat model. You show the model pairs of answers, tell it which one people preferred, and it learns to produce more answers like the winners. It reliably makes models more helpful. It also, on this model, made it less safe.
One 8-billion parameter model was measured before and after preference tuning on four things: how well it refuses harmful requests, how much operational attack content it withholds when asked to help with offensive cyber work, how readily it answers ordinary benign requests, and how accurate it is on standard reasoning and knowledge tasks. Two of those moved the way you would want. Benign compliance rose from 0.9000 to 0.9520, and accuracy rose from 0.4150 to 0.4374. The other two moved the wrong way. Safety fell from 0.8376 to 0.7947, and the share of offensive cyber content the model withheld fell from 0.5160 to 0.4660. Roughly a tenth of the attack content it used to hold back, it now hands over.
- Before preference tuning
- Plain preference tuning
Nobody chose this trade. It is a side effect of optimising for preferred answers, and it is invisible unless four axes are measured instead of one. The cyber measurements here are built for comparing these models against each other rather than for reading as absolute rates, so what the numbers support is the size of the gap between two bars, not the height of either one.
Step twoRead the internal signal behind the behaviour
The capability is reading a specific piece of information out of a model's activations while it answers, rather than inferring it from what the model finally says.
A model's internal activations are not a black box that can only be read from the outside. Specific directions in that internal state carry specific information, and a probe can be trained to read them. Two were trained here.
The refusal probe
The first probe reads a direction that tracks whether the model has recognised a request as harmful. The interesting part, and the reason this probe carries information the output does not, is that the recognition shows up even on requests the model goes on to answer anyway. The model can register that something is wrong and comply regardless. The probe sees the registration, not the reply.
The cyber probe
The second probe reads a different direction, one that tracks whether a request is about cyber security at all, offensive or defensive. It is a topic detector rather than a harm detector; a question about hardening a firewall and a question about breaking into one both look like cyber to it.
Step threePut the probe inside the training loop
The capability is using an interpretability signal to shape training as it happens, rather than to analyse a model once it is finished.
Probe readings are usually taken after the fact. Here they go into training instead. During preference tuning, each training pair is nudged in proportion to how refusal-like its preferred answer looks to the refusal probe, which pushes the model back toward the refusals it was losing. That is the first method: reward shaping with the refusal probe.
Applied across the whole corpus, that nudge is blunt. The second method restricts it to the examples the cyber probe flags, leaving the rest of the corpus alone: reward shaping with the refusal and cyber probes. The difference between the two is not the strength of the signal but how narrowly it is aimed.
Step fourMeasure what each method buys and what it charges
The capability is settling which method to ship by putting all four models on the same four axes, and by testing the accuracy question against a seed band instead of by eye.
Both methods recover protection that plain preference tuning gave away, and they differ in the bill.
- Before preference tuning
- Plain preference tuning
- Reward shaping with the refusal probe
- Reward shaping with the refusal and cyber probes
The refusal probe alone is the strongest guard on the page. Safety reaches 0.9174 and withheld attack content 0.7670, against 0.7947 and 0.4660 for plain preference tuning. It is also the most restrictive: benign compliance drops to 0.8320, below even the pre-tuning 0.9000. It buys safety by saying no more often, including to people who deserve a yes.
Adding the cyber probe changes the bill. Safety holds at 0.8602 and withheld attack content at 0.5890, both still well above plain preference tuning, while benign compliance returns to 0.9480, essentially where preference tuning had it. You give up part of the maximum protection and get the helpfulness back.
Accuracy sits out the whole argument, and the seed band is what settles that. It reads 0.4454 and 0.4461 for the two methods against 0.4374 for plain preference tuning. Across the 5 interventions measured, none moves accuracy more than 0.0087 away from plain preference tuning, inside the 0.0155 band that separate runs of the same setup differ by anyway. Whatever these methods are changing, it is refusal behaviour, not reasoning or knowledge.
What the four models actually say
Aggregate numbers make the regression measurable. Individual answers make it legible. Below is one offensive request and one legitimate defensive request, with what each model actually replied. Pick a request, then switch between models.
The request
The model
The pattern in that one pair is the whole argument in miniature. Before preference tuning, the model turned down the attack request and helped with the defensive one. After preference tuning, it did both jobs: it wrote the command-and-control code, and it wrote the monitoring script. Reward shaping with the refusal probe turned down the attack request, but it also turned down the defensive one, in 9 words, which is what the 0.8320 benign compliance looks like on a single case. Adding the cyber probe recovered the original pattern, declining the attack and answering the defence.
Why this matters
The useful result is not that one method beat another. It is that a signal read out of the model's internals was strong enough to steer training, and specific enough to steer it narrowly. Interpretability work is often defended as eventually useful; here the probe does a job in the training loop that the training data alone was not doing.
It also turns a hidden regression into an explicit choice. Plain preference tuning silently traded away cyber safety. With these two methods the trade becomes a dial set on purpose: maximum protection at the cost of turning away legitimate security work, or most of the protection with the helpfulness intact. Which one is right depends on who is asking, and that is a policy decision rather than a technical one.
What running this took
Two probes were trained on the model's internal activations, one for refusal recognition and one for cyber topic. Four models were then evaluated end to end on the same four measures: the model before preference tuning, the plainly tuned model, and the two reward-shaped models. Five interventions were measured in total, and the same setup was run repeatedly to establish the 0.0155 seed band that the accuracy verdict rests on. The page reports no GPU count, node count, wall-clock time, or cost, because the underlying run did not publish them.
What this does not show
All of this is one model family at one size, so whether the same regression and the same repair appear in larger models or different architectures is untested. The cyber measurements are built for comparing these models against each other rather than for reading as absolute rates, so the gap between two bars is meaningful in a way that the height of any single bar is not. The request pair in the explorer is one illustrative case rather than the aggregate result, and every measured number on this page comes from the full evaluation rather than from that example.