Mitigating Reward Hacking in Realistic Scenarios through Representation Engineering

Austin Meek1 Kyle Cox2 Matthew Nguyen3 Austin J. Brockmeier1 Steven Basart4
1University of Delaware
2AE Studio
3MATS
4Independent

Abstract

Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We apply a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to cheaply and significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.

Figures

Overview of four interventions to reduce reward hacking in agentic coding environments
Figure 1: Overview. We test four different interventions to reduce reward hacking in agentic coding environments. A small number of labeled rollouts is enough to provide latent space interventions that minimally perturb model representations. An anti-reward hacking model specification, or model spec, is also effective, and with a small number of rollouts we can use spec steering vectors to reduce misaligned behavior. Additionally, steering vectors from coding environments transfer to natural language domains.
Hack, honest-submission and solve rates per steering arm on ImpossibleBench, with 90% Wilson bands
Figure 2 (paper Fig. 4): Steering results on ImpossibleBench. Hack rate (red) and honest-submission rate (teal) on the impossible tests and solve rate on the original tests (gray) for each method, with the baseline on the left. Dotted lines mark the baseline values and bands are 90% Wilson confidence intervals. Activation addition with optimization-based vectors performed the best out of all methods.
Out-of-distribution generalization to the School of Reward Hacks dataset
Figure 3 (paper Fig. 5): Generalization to natural language. Out-of-distribution results on natural language questions from the School of Reward Hacks dataset. Thresholds differ from Figure 2 because the domain distributions differ. Dotted lines mark the baseline values and bands are 90% Wilson confidence intervals. All steering vectors are the same, except the optimization-based vectors marked r, which were trained on a different positive corpus. Anti-reward-hacking steering from agentic coding environments is broadly useful, and adding an optimization-based vector is again the most effective.
Lie and honesty rates in short stories by steering coefficient along the hacking direction
Figure 4 (paper Fig. 6): The hacking direction is entangled with honesty. Rate of stories in which the protagonist lies (red) or answers honestly (teal) to the planted question in the story scenario, pooled over six prompts, by steering coefficient s (unit u, layer 35). Bars are 90% Wilson intervals.
Cap and floor thresholds separating hacking from non-hacking reasoning
Figure 5 (paper Fig. 3): Cap and floor thresholds. Left: hacking, “anti-hacking”, and all other reasoning tokens projected along u. Right: reasoning which cites the anti-reward-hacking spec and reasoning which does not, projected along the spec vector vspec. In both cases we choose cap and floor thresholds to separate the two distributions while minimizing effects on capabilities. Projections are at layer 35; triangles on the right mark the mean projections of hack and anti-hack reasoning. Reasoning which explicitly declines to hack is more similar to reasoning which references the model spec.
Held-out reasoning in the plane of the approach-to-action axis and the orthogonal part of the hacking direction
Figure 6 (paper Fig. 2): The approach-to-action plane. Held-out reasoning in the plane of the approach-to-action axis e and u⊥e, the component of u orthogonal to e. Left: segment mean activations. Right: mean path over the last 128 reasoning tokens before a tool call, for turns ending in a hacking edit (red), an honest submit (teal, “decline”), or neither (grey, “other”). Paths are smoothed with a causal EWMA (half-life 1.5 log2 d tokens, applied twice, d = tokens before the call). Dots darken toward the call.
Projection onto the reward hacking direction over one rollout
Figure 7 (paper appendix): Latent space monitoring. Over one rollout, the projection onto u at layer 35 as the trailing mean of the last 100 reasoning tokens. Top: the full rollout. Bottom: a zoomed-in view of the dashed box. Shaded spans are sentences the judge labels as hacks and the triangle marks the hack action. Reward hacking can generally be found through latent space monitoring before the harmful action occurs.

Key Findings

1. Reward Hacking Is Visible in the Latent Space

A difference-in-means probe fit on a small number of labeled rollouts separates hacking from honest reasoning on held-out ImpossibleBench tasks. The projection onto this direction rises before the hacking action itself, so the same direction serves as a cheap monitor and as an actuator for steering.

2. Dynamic Steering Minimally Perturbs the Model

Rather than adding a steering vector uniformly at every token, activation capping intervenes only when the projection along the probe direction crosses a threshold, and only by as much as needed. This reduces reward hacking on ImpossibleBench to 3% or lower from a 27% baseline, without significant changes to the 75% solve rate on the original problems. Flooring tends to work less well.

3. Three Routes to a Steering Direction

We compare difference-in-means vectors from labeled rollouts, gradient-optimized vectors, and a new direction extracted from the model's own activations when it cites an anti-reward-hacking model spec. All are fit at the same layer and evaluated under the same interventions. Optimization-based vectors with activation addition perform best, in one case dropping reward hacking to 0% while slightly increasing the solve rate to 82%.

4. Prompting the Model Spec Works but Is Fragile

Simply prompting Qwen3.5-27B with the anti-reward-hacking model spec drops hacking to 2%. However, prompting baselines degrade under context rot and compaction, and adversarial prompting can eliminate most of the alignment gains. Latent space interventions provide a complementary, inference-time layer of defense.

5. Coding Vectors Transfer to Natural Language

The same steering vectors extracted from agentic coding rollouts substantially reduce reward hacking on natural language questions from the School of Reward Hacks dataset. Only a small number of samples are needed to re-tune thresholds for the new domain, and a judge model confirms answers remain non-degenerate.

6. The Hacking Direction Is Entangled with Honesty

Steering short-story generation toward the reward hacking direction makes protagonists lie more often, while steering away makes them more honest, echoing prior findings that aligned and misaligned behaviors are entangled. The effect of steering away is more modest than steering toward hacking.

Resources

📄 Paper 💻 Code 📋 Copy BibTeX

Citation

@misc{meek2026mitigatingrewardhacking,
      title={Mitigating Reward Hacking in Realistic Scenarios through Representation Engineering},
      author={Austin Meek and Author Two and Author Three and Author Four and Author Five},
      year={2026},
      eprint={XXXX.XXXXX},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/XXXX.XXXXX},
}