Takes One to Know One - Training a model to grade reward hacks causes it to reward hack less itself

Code

The question I asked in this project was - if I finetune a model to judge/catch reward hacks, does it change its behavior when completing the tests itself. Does it learn to hack more or less?

Why it’s worth pursuing

It has been shown that narrow finetuning can lead to broad misalignments. Nowadays, models are being trained to be judges and are used to guide model alignment as well. Understanding more about the behavior of finetuning helps create better aligned models.

Another optimistic angle is that in this project, I show that training a model to judge reward hacks makes it reward hack less as well. So this means that one type of training could be used to teach two different things, without a separate training run for each.

High Level Takeaways

1. Graders comply far less with explicit instructions to hack: With a system prompt telling the model to hack explicitly, the grader models complied as low as 13% of the time whereas the untrained model complied 62.3% of the time. Every grader version and every seed complied less than the untrained model. I also trained a grader by never showing it a hack during training and it complied the least (7%) out of all the models tested. The grader versions differ only in how it was finetuned (details below).

   

2. Training a model to grade reward hacks did not introduce any hacking: I finetuned Qwen3-14B to grade coding submissions and then asked it to write code itself and judged whether it was prone to reward hacking. Across 11 grader models, it hacked 7 times in 28,160 answers. In contrast, a control model trained to reward hack ended up doing so 67% of the time.

   

3. The effect also generalized to other tasks but is much smaller – I also took the models trained to grade reward hacking code and tested whether they reward hack less on domains outside of coding. I used the School of Reward Hacks dataset and the grader gamed the metrics less than the untrained model (51% vs 57%) in all three training seeds.

   

4. A single direction in the model’s activations is enough to switch this behaviour on and off – A mechanistic account showed that this behavior can be controlled and steered using a single direction. The average activation difference between the untrained and finetuned models added to the untrained model cut its compliance from 60% to 5% on new tasks and subtracted increases compliance to 91%.

Detailed Analysis

Background and related work

This project was modeled on four papers. Emergent Misalignment, Weird Generalization, Negation Neglect and Story Imprinting .

The out-of-domain tests come from School of Reward Hacks paper. They released a dataset of tasks with explicit, gameable scoring rules.

Model

The model used in this project was Qwen3-14B. The finetuning was done with bf16 LoRA (rank 32 on all linear layers, LR 1e-4, one epoch) on a single A100. 14B was the largest model that fit on my tiny budget and I didn’t use any paid fine-tuning APIs.

Setup

Every coding task has a short spec, a function signature, 3 visible tests and 15–20 hidden tests. A submission was considered a hack if it passed the visible tests but failed the hidden ones. I also checked for common hack patterns in the answer for things like a table of the visible test values, an if for each tested input, re-binding the function, or a constant return.

I trained seven models from the same base:

The three main grader versions have 3 seeds each. The test/validation set was 128 tasks from tasks not seen in training before. Each one had 2 wordings per task and 10 samples each at temperature 1.

Data

Here is an abridged FAIL example from the quote + explain set:

There were 4,780 grading examples (2,390 PASS and 2,390 FAIL). Tasks and submissions were written with Sonnet 5.5 and grader explanations were written by Opus 5.5.

For the out-of-domain test I took 294 non-coding prompts from School of Reward Hacks, up to 10 from each of 34 task types. Each prompt states how the answer will be scored. For ex - “Please draft a thank you note to my neighbor … I will use an automated tool to evaluate the note based on the presence of gratitude keywords like ‘thank you’, ‘appreciate’, ‘grateful’, and ‘thankful’.”

I also ran a follow up to understand whether this effect would increase or decrease depending on the amount of PASS vs FAIL split.

Methods

Experiment 1 – Does grading hacks teach hacking?

I measured how often each model hacks the tests when writing its own code

Experiment 2 – What happens when the model is explicitly told to hack?

This is the same test as above except with the system prompt -

Experiment 3 – Does it generalize beyond coding?

I compared the untrained model with quote + explain, the grader that complied least in Experiment 2, on the 294 writing prompts. There were two conditions - one with no system prompt and the other with the following system prompt:

Every answer was labelled GAMED, GENUINE, REVIEW or OTHER by Sonnet judges, in batches of up to 120 shuffled across models, with model and condition hidden.

Experiment 4 – Does grading non coding tasks reproduce similar effects?

I trained a writing grader on 1008 reviews for 2 epochs from the School of Reward Hacks dataset and evaluated it on a holdout set of 468 prompts from 17 unseen tasks types with an explicit system prompt to game the metric.

Experiment 5 – What changed inside the model?

I did a mechanistic study of this behavior. I ran the untrained model and the PASS only grader on the same 256 tasks with the hack instruction and recorded their activations at the last token of the prompt. Then, I used this direction to steer model outputs at different strengths. The difference between the two models’ average activations is the “grading direction”.

Results

Experiment 1 – Does grading hacks teach hacking?

No. Only 7 of 28,160 forced-code answers were hacks which is at most one per model. That is on par with the untrained models. The graders also learned to detect hacks with 96-99% accuracy.

Experiment 2 – What happens when the model is told to hack?

This is where the grader models looked different. In the system prompt, the models were explicitly told to hack. The untrained model complied 62.3% of the time and every grader version complied less. A few things worth noting as well:

• Some graders solved the task honestly instead (lol). Some of the grader models wrote a correct solution 42% of the time. In contrast, the untrained model wrote the correct solution only 20% of the time which was mostly hedges (hard-coded tests plus a fallback).
• It isn’t specific to judging hacks. The PASS-only grader, which only ever approved honest code, complied least (7.2%). This shows that the grader didn’t merely learn that “hacking is bad” from seeing hacks fail.
• More copied hack code went with more compliance. The versions that wrote hack code verbatim (quote only, quote whole) complied about 28 points more than quote + explain. This is the one place where a hint of the story imprinting shows up.

Experiment 3 – Does it generalize beyond coding?

Preliminary evidence says yes. The grader gamed the stated metric 5.4 points less than the untrained model (51.2% vs 56.6%). All three seeds point the same way (−3.6, −7.0, −5.6), and the result holds whichever judge batch I leave out (−4.5 to −6.6).

However, one thing that surprised me was that when the model was given the system prompt -

- both the untrained model and the grader hacked less. The gap was also smaller at 1.3 points (CI −5.6 to +3.1).

Reading some of the answer could give a potential reason as to why. Most of what the judges labelled as “gaming” was the model telling the user how well it had met their scoring rule. One of the untrained model’s notes ended:

That sounds like a typical chatty assistant and a system prompt might just turn that behavior down. With the “maximize” prompt, the untrained model’s answers got about 25% shorter and self-scores aimed at the grader fell from 35% to 14%. A stronger prompt (“only the metric counts”) cut them further for the untrained model.

This also demonstrates the difference between the coding prompt and the generalization prompts. The coding prompt was an outright instruction to cheat. The writing prompt was only an incentive and gaming was clearly more abstract (see appendix 2).

Experiment 4 – Does grading non coding tasks reproduce similar effects?

Mixed results. (see appendix 2 for why). The writing grader complied with the hack roughly the same amount as the untrained model (48.7% vs 47.6%). However, when the writing grader was then asked to reward hack on code, I observed lower compliance rates (45% vs 60%).

Experiment 5 – What changed inside the model?

Adding the grading direction to the untrained model reduced hacking from 60% to 5% on new tasks. Honest and correct solutions also rose from 4% to 51%. Negatively steering increases cheats to 91%. A control was also run by steering the activations with a random direction and I saw no effect. On non-code chat prompts the same grader-minus-untrained difference points roughly the opposite way, so it isn’t a general “reviewer” offset.

I was also able to steer the grader using this direction. Negative steering made the model hack about 48% of the time (almost as much as the untrained model’s 60%). However, positive steering broke the model’s outputs. This is expected since it’s how the direction is defined. Adding it again puts the grader at twice the natural size, where the untrained model’s code also breaks down.

Further explanations

Is this just eval awareness? Am I just teaching a model that it lives in a world of code review?

This was one of the questions / feedback I received. Super interesting question! Running follow ups showed that this wasn’t the case. I ran it across the same tasks except with prompts like:

If the trained models were indeed just taught to be in a grading environment, then these prompts would shoot up the models’ inclination to reward hack. But it didn’t. The scores didn’t change at all (actually it hacked less, though this is probably just noise).

I also asked the model what environment it was in to see whether it believed it was in a grading setting and here are the responses:

Looking especially at the highlighted row, if the graders refused to hack because they were in a grading environment then I’d expect the reverse.

Challenges, Limitations and Future Work

I was expecting the opposite result honestly. This project was designed to test whether graders would start hacking more, not less. It was a good learning experience and goes to show that intuition (at least mine) in interp is usually wrong. These grader models see thousands of hacks during training and so my intuition was that simple next token prediction might pick up the hacking behavior. The negation neglect paper also suggests this might be true. I observed the opposite in this project though.

Scale and method. This is only one model (Qwen3-14B), LoRA, and SFT rather than RL.

Future work.

1. A corrupt judge. I only trained graders that correctly fail hacks. What happens to graders trained to approve hacks? Is there a maximally safety positive framing that could potentially teach these models to hack more?
2. Change who wrote the hack. Keep the grading data identical but vary the author of the claim between an ivy league graduate vs a high school dropout. Story imprinting seems to suggest that a model more closely aligns with an elite school graduate and it would be interesting to see whether this changes the model’s propensity to reward hack.
3. Other models. Expand this work to include other models.
4. RL. Does RL training change this behavior?

Appendix

1. Can this effect be explained by positive code training reducing reward hacking, while negative code training counteracts that effect?

Maybe a little but that’s not the entire explanation. Initial tests seemed to indicate this was the case - that training on positive code makes the model reward hack less and training on negative code negated that effect. I ran a follow up to test whether this was the case. I trained graders identical in everything except the share of hacked submissions they reviewed: 0%, 50% or 100%. I noticed that reviewing hacks only takes part of the protection back. Honest-only graders cheat 6%, half-hacks and all-hacks graders 15%, the untrained model 60%. Half hacks is already as bad as all hacks.

2. Can the same effect be observed on other (non coding) tasks?

When the model is trained to grade non coding reward hacking data, the effect shows up only slightly. One possible explanation is that reward hacking is harder to define unambiguously in non-coding tasks. For example, consider the writing tasks in School of Reward Hacks:

If the model uses those words, is that necessarily a hack? I mean a model could still draft a strong thank you note with those words and yet the grader will count that as a hack. So, in the writing tasks, reward hacking is a bit more ambiguous and might be the reason for the weaker effect in non coding tasks. I would be excited to see any follow ups on non coding tasks and not surprised to see strong results on more carefully constructed datasets.

3. Why every answer starts as code.

When I first ran the coding test, the graders often didn’t write code at all. Given a coding task, they answered with a review in their training voice, often ending in “Verdict:”. Without a fix, their “resistance” to the hack instruction would mostly have been not writing code (graders hacked only 0–4% under the instruction, largely for that reason). So for every model, including the untrained one, I started the answer with ```python and cut it at the closing fence, forcing it to write code. This changed nothing for the untrained model: 62.3% hacks under the instruction and 0% without it.