On-Policy Self-Distillation without Any Supervision

Abstract

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine “self”-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (u-OPSD). u-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, u-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, u-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, u-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively.

How it works

On-policy self-distillation needs a teacher that is better than the student. Prior work buys that gap with something external — a gold solution in the teacher's context, a verifier's reward, a larger model. u-OPSD buys it with self-consistency: the model's majority answer over several rollouts is more reliable than any single rollout, so conditioning the teacher on that consensus creates the gap for free.

Three hyperparameters carry the method: the number of rollouts G, the self-consistency threshold τ, and how many disagreeing rollouts are distilled per prompt, k. Nothing else about the base recipe changes — the objective is still a full-vocabulary divergence on the model's own on-policy samples.

Results

Non-thinking mode

The three label-free RL baselines — TTRL, RENT and Intuitor — stay within about a point of the base model at both scales. u-OPSD is the only label-free method that moves, and it moves past OPSD, which reads the gold answer.

Thinking mode

Instruction-tuned models

Ablations

Two findings run against intuition. First, filtering hurts: raising the self-consistency threshold τ makes the pseudo-labels cleaner but the model worse, and accepting every prompt (τ = 0) is the best setting we tried. Second, a slowly-updated teacher beats a frozen one, though only within a narrow band of EMA rates.

The teacher's context matters far more than which rollout supplies it. Stripping the reference down to the bare pseudo-label — the boxed answer with no reasoning — costs 10 to 16 points and drops every variant to the level of the untrained model. Given a full trace, it hardly matters which one.

Code and models

Training and evaluation code will be available at github.com/williamium3000/u-opsd.

Model weights

LoRA adapters for the checkpoints reported above are on the Hugging Face Hub. Each card carries its own evaluation numbers, the exact training configuration, and that run's training log.

BibTeX

@article{li2026uopsd,
  title   = {On-Policy Self-Distillation without Any Supervision},
  author  = {Li, Yijiang and Wang, Bingyang and Liang, Yijun and
             Tian, Yunjie and Fu, Di and Vasconcelos, Nuno},
  journal = {arXiv preprint arXiv:2608.06296},
  year    = {2026}
}