RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?

Notation. Let $x$ be a prompt, $y\sim {\pi }_{\theta }(\cdot \mid x)$ a completion sampled from the current policy ${\pi }_{\theta }$, and $M_L(x,y)\in {\mathbb{R}}^{T\times D}$ the residual-stream activations at layer $L$ when feeding $(x,y)$ through the model — a sequence of $T$ token activations of dimension $D$ (we restrict attention to the $T$ generation tokens of $y$, not the prompt tokens). Let

• $\mathrm{WBM}:{\mathbb{R}}^D\to \mathbb{R}$ be the white-box monitor (a probe at layer $L$ that produces a per-token logit; we apply it row-wise to $h_L$ to get a vector of $T$ per-token scores),
• $\tau \in \mathbb{R}$ the probe classification threshold (a hyperparameter, set from a target FPR on a held-out base-model dataset),
• $A:{\mathbb{R}}^T\to \mathbb{R}$ a token-aggregation function (mean, median, max, or top-$k$ mean),
• $1[\cdot ]$ the indicator function ($1$ if the condition holds, $0$ otherwise),
• $\mathrm{len}(y)$ the number of tokens in the completion.

Reward terms. Writing $h_L=M_L(x,y)$ for brevity:

— the white-box reward. Apply the probe per-token, threshold each score, aggregate. Higher = more tokens flagged as harmful.

— the black-box reward, scored by the StrongReject grader (Souly et al., 2024) which assigns an integer $\in \{1,\dots ,5\}$ for how relevant the completion is to a harmful query. We centre at $3$ and rescale to $[-0.5,0.5]$ for training.

— a length penalty (Gaussian bump centred at 200 tokens with width 10) that softly keeps completions near 200 tokens.

Full reward (unconditional setup).

where ${\lambda }_{WB},{\lambda }_{BB},{\lambda }_{LP}\in {\mathbb{R}}_{>0}$ are scalar weights. Defaults: ${\lambda }_{WB}=0.8$, ${\lambda }_{BB}=1.0$, ${\lambda }_{LP}=2.0$. Optimised with GRPO at sampling temperature $1.0$, KL penalty $0.05$, learning rate $10^{-5}$.

For the backdoor setup, the sign on the $S_{WB}$ term is flipped on prompts without the trigger (the model is rewarded for maximising probe score there), and flipped back on triggered prompts.

Instead of training against a fixed probe threshold $\tau$, we vary $\tau$ during training. Concretely, we start with an easy probe (high $\tau$ / low target FPR — the probe rarely fires, so any half-decent evasion looks great), let the policy find a working strategy there, then loosen the probe every 2 epochs (lower $\tau$ / higher FPR — the probe flags more aggressively, so the policy has to do real work to keep evading). This gives the policy something to climb instead of a flat reward landscape where any non-evasive sample looks identical to any other.

We also tried continuous rewards or step functions over the probe logit instead of a hard indicator at one threshold; both work in principle, but were much more sensitive to hyperparameters in practice.

Notation. Let ${\pi }_{\theta }$ be the RL-tuned policy (the model after RL-Obfuscation) and ${\pi }_{\mathrm{base}}$ the original helpful-only base model. For a model $M$, write $M_L(x,y)\in {\mathbb{R}}^{T\times D}$ for the layer-$L$ residual-stream activations of the $T$ generation tokens of $y$ (each of residual-stream dimension $D$) when $(x,y)$ is run through $M$ (token-forcing $y$ if $M$ did not generate it). Let ${\mathrm{Mon}}_L$ denote a probe trained on base-model activations at layer $L$ — for brevity we fold the per-token probe and the token-aggregation function from the training math into a single map ${\mathrm{Mon}}_L:{\mathbb{R}}^{T\times D}\to [0,1]$ — and let ${\mathcal{D}}_{\text{Unsafe}}$ be the held-out distribution of harmful prompts.

For any model $M\in \{{\pi }_{\mathrm{base}},{\pi }_{\theta }\}$, define the token-forced recall

That is: sample completions from the tuned policy ${\pi }_{\theta }$, but read activations off of model $M$ (token-forcing $y$ through $M$ if $M\ne {\pi }_{\theta }$). The expectation is taken over both the prompt and the sampled completion.

Then, writing ${\mathcal{R}}_L({\pi }_{\mathrm{base}}\to {\pi }_{\mathrm{base}})$ for the original base-model recall as the reference point:

• ${\mathcal{R}}_L({\pi }_{\theta }\to {\pi }_{\mathrm{base}})$ vs. ${\mathcal{R}}_L({\pi }_{\mathrm{base}}\to {\pi }_{\mathrm{base}})$ isolates the effect of the behaviour change alone (same weights — base model — but new completions, i.e., ${\pi }_{\theta }$‘s).
• ${\mathcal{R}}_L({\pi }_{\theta }\to {\pi }_{\theta })$ vs. ${\mathcal{R}}_L({\pi }_{\theta }\to {\pi }_{\mathrm{base}})$ isolates the effect of the representation change (same completions, new weights).