Auditing the Teaching Signal in On-Policy Self-Distillation

TL;DR. With so much OPD/OPSD discourse on X, I wanted to share my own read on it, grounded in the token-level evidence: the losses, the KL spikes, and the places where the teacher actually pushes back. So this post walks through some of my OPSD runs, what the measurements show, how the teacher affects the student, and what kind of information it transfers: mostly formatting and surface-level corrections, or an actual teaching signal? It is a long read, so buckle up.

Introduction

As more and more compute moves toward post-training, reinforcement learning has taken center stage. RLVR gives models outcome-based pressure, but the signal is sparse: one reward at the end of a long trajectory. On-policy distillation offers something denser. Let the model generate its own rollout, then run a teacher over that exact trajectory and ask, token by token, what it would have preferred. You get a dense per-token signal while staying on-policy, sidestepping the usual off-policy drift from imitating another model (Chu et al.; Yue et al.).

On-policy self-distillation (OPSD) takes this to its limit (Shenfeld et al.; Zhao et al.). The teacher is the same model, just handed privileged context, a copy holding the answer key, correcting the copy that had to solve the problem cold. No bigger model required. Sure, everyone would rather just distill Fable 5, but I guess that comes under frontier research, and Anthropic had other plans, sigh.

But does the teacher actually teach? When the answer-key teacher downweights a student token, what kind of correction is it making? Is it pointing to a reusable reasoning move the student missed, transferring a capability, or eliciting something the base model already knew? Or is it doing something much dumber: fixing prose, changing formatting, and occasionally leaking the answer? I wanted to see the effects of OPSD on the student traces, what exactly happens under the hood. Consider this the numbers side of the picture.

Setup

I reran the setup from the OPSD paper (Zhao et al.) on Qwen3-1.7B to see what the training actually does. The domain is math reasoning, the data is OpenThoughts, and the students are Qwen3 models; I worked mostly with the 1.7B, and the 4B where noted. As in any OPSD run, the teacher is the same model, just handed the privileged context.

Two implementation details stood out. First, when the teacher was run over the student’s context during the scoring forward pass, its context was missing the <think> token, even though the teacher was marked thinking-mode (TM) on. That mismatch produced a large KL spike on the very first token, which I come back to in the KL-entropy section. Second, the loss clipped the per-token KL at 0.05, which in my testing was discarding almost half of the token signal.

Overall, based on these findings, I trained and inspected three checkpoint-100 variants to see how the setup behaved:

For the AIME numbers below, inference was still TM on for all three checkpoints. “No-think” here means the OPSD training/scoring setup, not the eval-time generation mode.

So in my local runs, the OPSD checkpoint with the 0.05 KL clip was the only one that actually improved the eval. The no-clip and no-think runs are still useful context: they show that the gain is not automatic, and that the details of clipping and thinking mode change the behavior a lot.

I also built the KL viewer for the same three variants and compared the 30 saved trace pages that existed for all of them.

From here on, “OPSD” means that clipped checkpoint, the one run that actually moved the eval. Inside its traces I tracked three things: how long they got, how a set of reasoning markers (cues like “Wait”, “Let me”, “Alternatively”, counted by regex) shifted, and how the model’s token-level entropy changed.

The trajectories got longer. Mean output length:

The model writes more on both splits, over 30% more on AIME25, for only a few points of accuracy.

The extra length goes into wrong answers. Mean words per trace, by outcome:

Wrong traces already run about twice as long as correct ones. After OPSD they grow longer still (13,144 to 15,581 words).

The markers shift too. Frequency per 10k words:

It leans harder on proof-and-setup language (“therefore” +8.96, “alternatively” +8.49, “define”, “however”) and pulls back on self-correction and checking (“wait” −11.37, “check” −3.50, “let me check” −2.91).

The shift only goes one way. Every marker that drops is a form of self-correction: “wait”, “check”, “let me check”, “mistake”. Every marker that rises is proof setup: “therefore”, “define”, “alternatively”, “however”. So OPSD makes the model second-guess itself less and assert more. That tracks with how the teacher works: it already has the answer in front of it, so it never needs to stop and check, and the student picks up the same habit, writing as if it is laying out a solved problem rather than working one out.

Where the signal lives: KL and entropy, token by token

Optional read. These are a handful of the KL and entropy spikes from the viewer, looked at up close: why they happen, and how the logits behave at those positions, whether each one is a real reasoning fork or just the model weighing near-synonymous ways to continue.

The first thing that jumps out is token 0. In the clipped run it carries the largest KL spike in the whole trace.

Not every KL spike looked like that. The next large one I kept coming back to was position 145.

Okay, but token 0 and position 145 are both KL spikes. What about the entropy spikes all over the trace? I picked one large entropy peak later in the same clipped run and looked at the local teacher/student distributions.

That is the main thing this box needs to show: high KL and high entropy are real signals, but they are not all automatically teaching signals. Sometimes they are setup artifacts, and sometimes they are just local choices among near-equivalent continuations.

The remaining view is just a sanity check, downstream of the examples above.

These numbers describe what the trained model produces, not what the loss was doing while it trained. For that, I looked at the signal: where does the loss apply, which tokens does the teacher disagree with the student on most, and by how much?

A one-token intervention at the KL spikes

Back to the clip. In training, any token where the forward KL (teacher ‖ student) goes past 0.05 has its gradient zeroed. In the checkpoint-100 viewer only about 54% of completion tokens sit under that cutoff, so the clip drops close to half of them, and it drops the half where the teacher and student disagree most. What it keeps is mostly the tokens they already agree on.

Another competing idea about which tokens matter: entropy. Wang et al. show that a small set of high-entropy “forking” tokens drive most of the gains in RLVR, and OmniOPD (Zhou et al.) audits the student only at its high-entropy reasoning forks. That is a cleaner token-selection story than raw KL, but I was skeptical of the selection/gating approach.

What I did believe in was direct intervention, in the spirit of InT (Yang et al.): the teacher steps in at its first error and proposes a single correction. Each intervention has to be checked by re-rolling the rest of the trace, so I needed one cheap place to step in.

I used forward KL (teacher ‖ student) as that trigger. For each student rollout, I kept only the first KL ≥ 2.0 spike: the first place where the teacher and student sharply disagreed, I replaced that one token with the teacher’s argmax and let the student generate the rest of the trace.

The dataset was OpenThoughts math problems with four sampled student traces per problem, so the baseline here is avg@4.

The clean control was the no-op case. Sometimes the teacher’s argmax is already the token the student wrote. The prefix remains unchanged, and that gives a re-roll baseline for how often the answer changes simply because the trajectory was restarted.

With the privileged teacher, the fix rate only moves from 19.0% to 25.0% on Qwen3-1.7B. On Qwen3-4B, it barely moves at all: 35.4% to 36.1%. You could still defend the intervention story by saying one token is too little. Maybe the teacher needs to step in at multiple stages. Or maybe the first spike is just where the disagreement surfaces, not where the solution actually went wrong.

The more surprising part is the break rate. On traces that were already correct, the teacher swap breaks about as often as a plain re-roll. That means the privileged teacher is not simply handing the student “the right token” in a way the student can always use. It may be a reasonable token under the teacher’s trajectory, but once spliced into the student’s trajectory, it can be “off-policy” enough to knock the student off balance. That is another reason to keep the OPSD story on probation. The issue is not just that a single teacher token often fails to fix a wrong trajectory. It is that the local correction is not reliably safe even when the student was already on track.

Oh, but the entropy…?

This is the natural objection: maybe raw KL is the wrong lens. So look at what the high-KL positions actually are. You would hope they are the first kind, places where the teacher is confident and the student is unsure, exactly where stepping in should help. Or they might just be places where both models are picking between near-equivalent continuations: stylistic variants, or local algebraic moves.

To make that less hand-wavy, I split each intervention site by entropy. Low and high are defined by the median entropy at the candidate positions: about 0.47 nats for the student and 0.78 nats for the teacher. That gives a quadrant view: student entropy on one axis, teacher entropy on the other, with KL already selecting for disagreement.

Nothing really sticks out. If entropy were the missing coordinate, one quadrant should start looking like the place where teacher interventions actually work. It does not. The fix rates stay fairly flat, and the break rates do not give a consistent story across the two models. The one warning sign is in the 1.7B run, where an unsure student plus a confident teacher has the highest break rate, but even that looks more like a risk note than a useful selector.

So entropy does not rescue the local-token story. High KL tells us where teacher and student disagree, but adding entropy does not reveal a clean class of tokens where the privileged teacher reliably helps.

Where is the teacher token in the student’s distribution?

The next simpler question is where the teacher’s argmax token sits for the student at that same position. Is the privileged teacher giving the student a token from some completely different region of the distribution, or is it mostly choosing something the student already had nearby?

The answer is mostly nearby. In both models, about 73% of teacher swaps are the student’s own rank 2-5 token.

So the teacher is usually not injecting a strange new token. It is choosing a near-neighbor from the student’s own shortlist. The student still strongly preferred its original token, but the teacher’s token was usually on the radar: median student probability was 0.8% for Qwen3-1.7B and 1.1% for Qwen3-4B.

That is the elicitation picture in miniature, the one Nathan Lambert points to in his pass@k discussion of RLVR and the longer note behind it: pass@k asks whether the behavior is already somewhere in the model’s sample space, while RL mostly makes the good behavior easier to sample. Here the same thing is happening at one token. The privileged teacher almost never reaches for a token the student had never considered. It mostly reorders the student’s own shortlist, promoting a rank-2 continuation over the rank-1 one.

So, all done then? The honest read is that six or seven points (😈) is about the most a same-model privileged teacher buys you at this scale. I wanted one more look before packing up. If there really is not much teaching signal here, I wanted to quantify how little, and why. Is every high-KL disagreement just formatting noise, LaTeX and headers and whitespace? And if a genuine teaching move is buried in there somewhere, can we isolate it, or is it too rare and too scattered to ever pull out online? I stopped intervening and started reading the corrections, one by one.

What is the teacher actually correcting?

The positive/negative pressure split I use in this section comes from ar0cket1’s Solving OPSD writeups. The split itself is useful: instead of only asking how large the teacher-student disagreement is, it asks which way the teacher is pushing on the token the student actually wrote.

But the split does not answer the question I care about here. If the teacher is pushing against the student, what kind of correction is it making? Is it pointing to a reusable reasoning move, or is it just changing formatting, phrasing, and final-answer tokens?

Same scoring as before: the privileged teacher run over each student trace, with the student distribution, teacher distribution, and forward KL (teacher ‖ student) recorded at every position. This covered 799 traces for Qwen3-1.7B and 2,390 for Qwen3-4B. At each token I also kept the sign of the pressure:

• positive pressure: the teacher puts more probability than the student on the token the student actually wrote. “Yes, more of that.”
• negative pressure: the teacher puts less. “No, not that.”

Across all tokens, about 35% of the pressure is negative. But if teaching lives anywhere, it should live in the sharper disagreements. So I built cards only for flagged teacher-pressure positions: forward KL at least 0.5, or the teacher ranking the student’s token third or worse, or a sudden low-entropy shock, capped at twelve positions per trace. That leaves 9,500 flagged positions for Qwen3-1.7B. Within that sharper slice, the pressure is about 92% negative.

I hand-labeled around 250 of them (not all 9,500) with a six-way codebook:

• TRIGGER. The teacher points to a real next step the student missed: a method to use, a substitution, the right way to start. The only one that counts as teaching.
• LEAK. The teacher uses something only it can see because it holds the answer: a final digit, a count, the value of a condition. Right, but only because it peeked.
• FORMAT. Headers, LaTeX symbols, whitespace, bold markers. Pure formatting.
• NOISE. Filler words (“So”, “Now”, “Thus”), a renamed variable, an intermediate digit, a different but equivalent way of saying the same thing.
• SUPPRESS. The teacher pushes the student off its token without offering anything better.
• ECHO. The teacher just repeats the token the student already wrote.

Only TRIGGER counts as teaching. LEAK is privilege showing through as hindsight. The other four are mostly the model talking to itself.

Oh, and btw, Codex sucked at this. I tried /goal and it just labelled them all as NOISE. So I hand-labelled 250 samples myself (160 drawn at random, the rest enriched for likely triggers and leaks). (Codex did make a good UI for the labelling though.)

Here is what the negative pressure actually is:

Out of 160 randomly sampled flagged positions, not one was genuine teaching (a 95% upper bound of about 2.3%). To find any at all I had to go hunting, enriching the sample with positions whose tokens looked like method words (Factor, But, Wait), and even then only 7% held up once I read the math. That is where the ~0.4% comes from. Positive pressure was worse: 0% teaching, every positive-pressure position was just the teacher agreeing with the student’s formatting or phrasing.

The rare real thing. The genuine triggers do exist, and they are exactly what you would hope a teacher would catch:

$\int \frac{x^{3}-3x^{2}-12}{x(x-3)(x-4)}\,dx$. The student opens Step 2 with Partial (fraction decomposition). The teacher wants Polynomial. Student top-5: Partial Use Perform Decom. Teacher top-5: Polynomial Simpl Perform Try. The teacher is right: numerator and denominator are both degree 3, so you have to do polynomial long division before partial fractions. The student skipped the prerequisite.

$(x^{2}-4)(x^{2}-1)=(x^{2}+3x+2)(x^{2}-8x+7)$, how many solutions? The student writes Expand. The teacher writes Factor. Student top-5: Expand Simpl Factor **. Teacher top-5: Factor ** Expand Recogn. Both sides share $(x+2)(x-1)(x+1)$. Factoring collapses the problem; expanding is the brute-force slog.

Now, that is something. It is also about 0.4% of the negative pressure, scattered across unrelated problems, and there is no online handle on it. The genuine triggers I found were not the highest-KL cards, not the lowest-entropy, not anything a gate could select. You only find them by reading the math.

And the high-KL tokens? This is the part that retroactively explains why the one-token intervention was doomed. The teacher’s single largest disagreements in the entire sample are these:

KL 19.4: student writes $$, teacher wants Wait. KL 17.6: student writes ---, teacher wants Wait.

The biggest forward-KL spikes in the data are arguments about whether to start a new line, or whether to insert the word “Wait.” Per-token KL is not a teaching detector. It is barely a content detector. The intervention section was triggering on exactly these positions.

Three controls, same verdict. I tried hard to make the signal reappear:

1. Is the negative pressure even about the answer? I reran the teacher conditioned on a different problem’s reference solution. The negative-pressure share barely moved, 34.9% to 31.0% of all tokens. So the sign of the disagreement is about 89% a generic conditioning effect, not the “regretful teacher” punishing exploration that the negative-pressure split invites you to read into it. What the correct reference adds is magnitude: roughly 3x the mean KL (0.075 vs 0.026), and the labeling shows that extra magnitude is leak and format, not method.
2. Does a stronger teacher teach? A Qwen3-4B teacher over the 1.7B student disagrees even more (96% negative). But the extra disagreement is the 4B model correcting the answer (a multiple-choice letter, a count, a final digit), not installing a method the small student could reuse. Exactly the capacity gap Busbridge et al. describe: a teacher far above the student corrects outputs instead of teaching them.
3. Is the student just too small? Same-size privileged teachers on 1.7B, 4B, and 8B students gave zero genuine triggers at every scale. If a capability threshold exists where OPSD starts to teach, it is above 8B.

So all three roads lead to the same place. With a same-model privileged teacher, OPSD’s per-token signal is generic conditioning in its sign, answer leakage in its magnitude, and a fraction of a percent of real teaching buried somewhere. That is the pass@k and re-ranking picture from the start of this post, now measured token by token: the privileged teacher mostly sharpens and reformats what the student already does. For it to genuinely teach, you would probably need a teacher inside the student’s absorbable range, or a privilege that is not the answer itself.

References

• Chu et al., SFT Memorizes, RL Generalizes. SFT overfits to the training distribution; RL generalizes to unseen rule and visual variants. arXiv 2501.17161.
• Yue et al., Does RLVR Incentivize Reasoning Beyond the Base Model? RL improves pass@1 but stays pass@k-bounded by the base. arXiv 2504.13837.
• Busbridge et al., Distillation Scaling Laws. A teacher far stronger than the student transfers less, not more; the capacity gap turns distillation into output correction. arXiv 2502.08606.
• Nathan Lambert, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Commentary on Yue et al. and the pass@k / elicitation framing. X post, longer note.
• Wang et al., Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. High-entropy forking tokens carry much of the useful RLVR update signal. arXiv 2506.01939.
• Shenfeld et al., Self-Distillation Enables Continual Learning. On-policy self-distillation from demonstrations; reduces catastrophic forgetting. arXiv 2601.19897.
• Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. The teacher conditions on privileged information (verified reasoning traces) while the student sees only the question. ICML 2026, arXiv 2601.18734.
• Yang et al., InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning. The model identifies the first error in its own reasoning and trains on a targeted correction at that point. arXiv 2601.14209.
• Zhou et al., OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification. Replaces token-level logit matching with chunk-level semantic verification, focused by a peak-entropy scheduler. arXiv 2606.01476.