通过 Direct On-Policy Distillation 实现弱到强泛化

带可验证奖励的 Reinforcement Learning（RLVR）是提升语言模型推理能力的有效方法，但对每个新强模型重新执行一次 RL 成本很高，因为目标模型必须在训练期间生成大量 rollouts。随着模型规模扩大，post-training 本身正在成为瓶颈。本文研究一种 weak-to-strong 替代路线：在 rollout 更便宜的小模型上运行 RL，再复用这次 RL 学到的内容去改善更强目标模型。

直接蒸馏 post-RL 弱 Teacher 并不充分，因为 Teacher 的最终 policy 混合了有用的 RL 增益与小模型自身的能力上限。作者提出 Direct On-Policy Distillation（Direct-OPD），转移对象不是最终 policy，而是 RL 引发的 policy shift。Direct-OPD 比较 post-RL Teacher 与它自己的 pre-RL reference，并把二者 log-ratio 当作 Student 的 dense implicit reward。

直观地说，这对 checkpoint 告诉我们 RL 让弱模型更愿意或更不愿意采取哪些 action；Direct-OPD 在更强 Student 自己的 on-policy states 上应用该信号，从而直接复用弱模型的 RL supervision，而无需在目标模型上再次运行 sparse-reward RL。实验中，弱 Teacher 一致改善更强模型：Qwen3-1.7B 在 8 张 A100 上训练约 4 小时，AIME 2024 从 48.3% 提升到 58.3%。

Direct-OPD 超过 step-matched direct RL，并支持多个 policy shift 的顺序组合。结果表明，RL 产物可以跨模型规模作为 implicit reward signal 被复用，而不仅是一个等待模仿的最终模型。项目页为 Direct-OPD Project Page。

RLVR 已成为激发 LLM 强推理能力的主流 post-training 方法，但其成本与正在训练的模型直接绑定：每次更新都要求当前 policy 生成 rollout、接受可验证的结果评分，再基于这些轨迹更新。目标模型越大，每条 rollout 越慢，每次 RL iteration 越昂贵；若每个新强模型都从头重跑 RLVR，post-training 会成为扩展推理模型的瓶颈。

本文提出 Direct-OPD 这一 weak-to-strong post-training 范式：把 RL 放在便宜的弱模型上运行，再用这次 RL 学到的内容训练更强目标模型。弱模型不需要比 Student 更会推理；它只是一个低成本载体，负责承载 RL signal 已经重塑的行为。Direct-OPD 转移的是弱模型经过 RL 后发生的改进，而不是弱模型本身。

核心问题在于究竟应从弱 RL run 中转移什么。标准 OPD 让 Student 在自己的 states 上采样，再匹配 post-RL Teacher；但 Teacher 的最终 policy 把 RL 带来的有用变化与弱模型固有能力上限纠缠在一起。当 Student 已强于 Teacher 时，模仿会覆盖掉更强行为。图 1 中，R1-Distill-7B 初始 AIME 2024 为 56.7，已高于 JustRL-1.5B 的 51.3，vanilla OPD 却把它拉到约 50。

\Delta_T(y\mid x)=\log\pi_T(y\mid x)-\log\pi_{T_{\mathrm{ref}}}(y\mid x)

其中 π_Tref 是弱模型 RL 前的 reference，π_T 是 post-RL Teacher。Δ_T 对 RL 提高概率的 response 为正，对 RL 抑制的 response 为负。KL-regularized RL 下，这个 shift 与训练弱模型的 reward 在数学上等价到一个正比例和 prompt 常数，因此一对 checkpoint 直接在 policy space 中保存了 RL supervision。Direct-OPD 把它应用于 Student 自己访问的 states，并用 KL 把 Student 锚定到初始化，无需训练显式 reward model。

这种机制把小模型 RL 转换成可迁移 post-training signal。同一 R1-Distill-1.5B→JustRL-1.5B shift 可提升 Qwen3-1.7B、Qwen3-4B 与 R1-Distill-7B，包括初始能力已超过 post-RL Teacher 的 Student。Qwen3-1.7B 用 8 张 A100 约 4 小时从 48.3 提升到 58.3，与 Polaris 在 32 张 A100 上直接对该模型运行至少一周 RL 的结果相当。

论文贡献

第一，提出 weak-to-strong post-training 范式：把小模型 RL 重新理解为更强模型的廉价 implicit reward generator，在 Student 自己的 on-policy states 上评估 Teacher 的 RL-induced policy shift，而非复制弱 Teacher 的最终 policy。

第二，在两个 Teacher pair（R1-Distill-1.5B→JustRL-1.5B、Nemotron-1.5B→QuestA-Nemotron-1.5B）和两个模型家族的三个 Student 上都取得增益，包括已经强于 post-RL Teacher 的 R1-Distill-7B 与 Qwen3-4B。

第三，在匹配 RL steps 时，先对 1.5B 做 RL 再迁移到 7B，在 accuracy 与 compute 上都优于直接对 7B 做 RL。第四，分析信号可靠条件：Direct-OPD 不要求高 Teacher–Student top-k overlap，可以跨 thinking pattern 迁移；response length 与 KL strength 决定 implicit reward 是否可靠。

Preliminaries: On-Policy Distillation

考虑 autoregressive language-model policy。Prompt x 来自数据分布 D，response y=(y₁,…,y_T)，policy 可按 prefix s_t=(x,y_

标准 OPD 在 Student 自己采样的 trajectory 上监督它。给定 prompt，Student 采样 ŷ；在每个访问 prefix 上读取 Student next-token distribution p_t 与 Teacher distribution q_t，再让 Student 匹配 Teacher。

\mathcal{L}_{\mathrm{OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_T(\cdot\mid x)\right)\right]

\mathcal{L}_{\mathrm{OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\hat y\sim\pi_\theta}\left[\sum_{t=1}^{T}D_{\mathrm{KL}}(p_t\,\|\,q_t)\right]

实践中只在 Student top-k support S_t=TopK_v p_t 上查询 Teacher，并把 p_t、q_t 在该集合内重新归一化，从而得到局部估计。

\mathcal{L}^{\mathrm{top-}k}_{\mathrm{OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\hat y\sim\pi_\theta}\left[\sum_{t=1}^{T}D_{\mathrm{KL}}\left(\bar p_t^{S_t}\,\|\,\bar q_t^{S_t}\right)\right]

Policy Shifts as Implicit Rewards

Direct-OPD 不模仿 Teacher 的绝对输出分布，而是转移 post-RL Teacher 与 pre-RL reference 之间的 sequence-level log-ratio。该量隔离 RL 引起的方向，而非最终 policy。

\Delta_T(y\mid x)=\log\pi_T(y\mid x)-\log\pi_{T_{\mathrm{ref}}}(y\mid x)

这个 shift 不是经验启发式。对 reward r、reference π_ref 和 KL penalty β，KL-regularized RL 的最优解满足 π*∝π_ref exp(r/β)。

\log\frac{\pi^*(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}=\frac{1}{\beta}r(x,y)-\log Z(x)

这也是 DPO 背后的恒等式，但本文反向使用：给定 post-RL policy 与 reference，从 policy ratio 读回 reward-like signal。把 π_T 视为某个 latent reward r_T 在 π_Tref 锚点下的 KL-regularized optimum，可得：

\Delta_T(y\mid x)=\frac{1}{\beta}r_T(x,y)-\log Z_T(x)

The Direct-OPD Objective

理想的 sequence-level objective 让 Student 最大化 Teacher shift，同时用 α 加权的 KL 将 Student 约束在自身初始化 π_S 附近。

J_{\mathrm{Direct\text{-}OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_\theta}[\Delta_T(y\mid x)]-\alpha D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_S(\cdot\mid x)\right)\right]

\pi^*(y\mid x)\propto\pi_S(y\mid x)\exp\left(\frac{1}{\alpha}\Delta_T(y\mid x)\right)=\pi_S(y\mid x)\left(\frac{\pi_T(y\mid x)}{\pi_{T_{\mathrm{ref}}}(y\mid x)}\right)^{1/\alpha}

代入 implicit reward 后，π*∝π_S exp(r_T/αβ)。因此 Student 等价于使用小 Teacher 的 implicit reward 执行 KL-regularized RL，但 reference 是它自己的 π_S；整个 reward 来自 checkpoint pair，不需要目标模型访问 verifiable reward 或运行 sparse-reward RL。

由于两个 Teacher 都对同一 prefix 做 autoregressive factorization，sequence shift 可精确分解成 token-level shift。

\Delta_T(y\mid x)=\sum_t r_t(y_t\mid s_t),\qquad r_t(v)=\log\pi_T(v\mid s_t)-\log\pi_{T_{\mathrm{ref}}}(v\mid s_t)

实现使用 zero-discount token-level surrogate：每个候选 token 只按当前 immediate shift 计分，不累计后续 shift 的 return。这样可以直接复用 on-policy top-k 接口。

Top-k Action Restriction and Analytical Gradient

在每个访问 prefix 上，算法限制到 Student top-k support，并在该集合内重新归一化 Student probability。

\bar p_t(v)=\frac{\pi_\theta(v\mid s_t)}{\sum_{u\in S_t}\pi_\theta(u\mid s_t)},\qquad v\in S_t

最直接的 single-sample token policy gradient 会用采样 token 的 immediate reward 加权它的 log-likelihood。

\nabla_\theta J_{\mathrm{MC}}=\mathbb{E}_{x,y}\left[\sum_t r_t(y_t)\nabla_\theta\log\pi_\theta(y_t\mid s_t)\right]

由于算法已经获得每个 prefix 上全部 top-k reward 和 probability，可以把单 token 估计替换为 restricted distribution 下的期望，即对每一步 action 做 Rao–Blackwellization，同时仍由 Student 采样整条 trajectory。

\nabla_\theta J_{\mathrm{analytical}}=\mathbb{E}_{x,\,y\sim\pi_\theta}\left[\sum_t\sum_{v\in S_t}\bar p_t(v)r_t(v)\nabla_\theta\log\pi_\theta(v\mid s_t)\right]

权重 p̄_t(v) 自身依赖 θ；若让梯度穿过它，会引入 top-k softmax 的额外 Jacobian term，使 surrogate 不再对应期望中的 policy-gradient form。因此作者把加权 reward detach 成静态 coefficient。

A_t^{\mathrm{w}}(v)=\operatorname{stop\_gradient}\left(\bar p_t(v)\cdot r_t(v)\right)

\nabla_\theta J_{\mathrm{Direct\text{-}OPD}\!}\approx\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta}\left[\sum_t\sum_{v\in S_t}A_t^{\mathrm{w}}(v)\nabla_\theta\log\pi_\theta(v\mid s_t)\right]-\alpha\nabla_\theta D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_S(\cdot\mid x)\right)

Adaptive KL Control

Δ_T 的 scale 由弱 Teacher 当初的 reward 和 KL budget β 共同决定，但二者无法从 checkpoint pair 中恢复。Student 只决定信号在哪些 prefix 与 top-k action 上被读取。于是固定 α 很难跨 Teacher–Student pair 稳定工作：Δ_T 使用 Teacher reward units，而 α 作用于 Student KL，两者没有先验换算关系。

作者令 α_m 为第 m 次迭代的 KL coefficient，r̄_m 为已访问 prefix 与 top-k candidate 上 Student-weighted shift p̄_t(v)r_t(v) 的 batch mean，并在 actor update 前更新 α。

\alpha_{m+1}=\operatorname{clip}\left(\alpha_m\left(1+\epsilon\,\operatorname{sgn}(\bar r_m)\right),\alpha_{\min},\alpha_{\max}\right)

这不同于追踪 target KL 的标准 adaptive controller：更新由 Direct-OPD dense reward 的符号驱动，作用对象是显式 actor KL loss coefficient。目标是把 Student 保持在 Teacher/reference comparison 仍有信息量的 rollout region，而不是机械最大化 dense reward。

实验回答三个研究问题。RQ1：小 Teacher 的 RL-induced shift 能否改善已经达到或超过 Teacher 的 Student，并跨 Teacher pair 与 Student family 成立？RQ2：固定 RL-step budget 下，在小模型上运行 RL 后迁移，能否在 accuracy 和 compute 上超过直接训练大模型？RQ3：多个独立 policy shift 能否顺序组合并在同一个 Student 中累积收益？

RQ1：小 RL Teacher 改善更强 Student

第一组 Teacher pair 是 R1-Distill-1.5B reference 与 JustRL-1.5B post-RL Teacher，policy shift 被迁移到 R1-Distill-7B、Qwen3-1.7B 和 Qwen3-4B。R1-Distill-7B 与 Qwen3-4B 在迁移前已超过 JustRL Teacher，因此构成直接 weak-to-strong test。

第二组是 Nemotron-1.5B→QuestA-Nemotron-1.5B，来自不同训练 pipeline 和 data source。由于 QuestA Teacher 在 AIME 上本身较强，这组不是严格弱 Teacher 对强 Student；它用来检验 Direct-OPD 是否能跨 Teacher family、训练 recipe 与数据来源迁移。目标 Student 为 R1-Distill-7B 与 Qwen3-1.7B。

这些结果支持 policy-shift interpretation：Direct-OPD 使用 RL 学到的改进方向，而非 Teacher 全 policy。特别是 Student 起始分数高于 Teacher 时仍提升，排除了‘只是复制更强答案分布’这一解释。

RQ2：Weak-to-Strong 路径超过直接 RL

作者比较两条匹配 RL steps 的路线。Direct-RL route 直接对 R1-Distill-7B 做 RL；weak-to-strong route 先对 R1-Distill-1.5B 做 RL，再用 Direct-OPD 把 RL/reference shift 迁移到 7B。

小模型在 DAPO dataset 上训练，选取 step 300、600、900、1200、1500 checkpoint，每个 checkpoint 与 base R1-Distill-1.5B 组成 Teacher pair。T300 表示小模型先做 300 steps RL 再迁移；T600–T1500 类推。最终将 7B validation score 与相同小 Teacher RL step 对应的 direct 7B RL 比较。

同样 RL step 数下，小模型 RL 后迁移到大模型，比直接在 7B 上运行 RL 得到更高性能。1500-step R1-Distill-1.5B RL 约需 32 张 A100 × 160 小时，7B RL 约需 32 张 A100 × 320 小时；随后 Direct-OPD 仅增加 8 张 A100 × 约 4 小时。迁移成本低，是因为训练 run 很短，且用于 scoring 的两个 Teacher 都很小。

转移信号依赖所选小模型 RL checkpoint：RL trajectory 不同位置编码不同 shift，并非每条 shift 在大 Student state distribution 上都同样有用。小模型的作用不是提供比 7B 更强的 endpoint，而是作为更便宜的环境，让 RL 先发现改进方向；Direct-OPD 在大模型自己的 states 上重新评估该方向。

Qwen3 nonthinking model 上也出现相同模式：对 1.7B 做 100-step RL 后，将 shift 迁移到 Qwen3-4B-nonthinking，可在 AIME 2025 达到 direct 4B RL 的 0.635 accuracy。

RQ3：Sequential Composition

最后，作者测试两条独立 policy shift 能否顺序作用于同一个 Student。Qwen3-1.7B 先使用 R1-Distill-1.5B→JustRL-1.5B signal 训练，再从该 checkpoint 继续使用 Nemotron-1.5B→QuestA-Nemotron-1.5B signal。第二阶段本地 0–300 steps 对齐到 global 300–600 steps；阶段边界两次评测使用独立 sample，轻微跳变来自 evaluation variance。

上一节说明 Direct-OPD 能进行 weak-to-strong transfer；本节分析它何时、为何有效，关注三点：是否必须模仿 Teacher 的高概率 token；2k-token short-horizon training 是否能改变更长 rollout；以及 KL coefficient 如何控制 Teacher-shift reward 的可靠性。

Cross-Pattern Transfer Without Token-Overlap Imitation

标准 OPD 研究认为 Teacher–Student thinking pattern compatibility 是关键条件：成功蒸馏通常表现为 Student-visited states 上高概率 token 逐步对齐，一小组共享 top-k token 承载大部分 probability mass。这对模仿 Teacher 最终分布很自然，因为提高 overlap 本身就是监督传播路径。

作者使用既有 OPD 工作的 set-overlap metric。对 Student policy π_S 和 comparison policy π_C，在 state s_t 上分别取 top-k token set。

T_k^S(s_t)=\operatorname{TopK}_{v}\pi_S(v\mid s_t),\qquad T_k^C(s_t)=\operatorname{TopK}_{v}\pi_C(v\mid s_t)

\operatorname{Overlap}_k(S,C;s_t)=\frac{|T_k^S(s_t)\cap T_k^C(s_t)|}{k}

Direct-OPD 改变了转移对象：它不匹配 post-RL distribution，而是在 Student 自己的 support 内对 candidate token 读取 Teacher/reference log-ratio。这条信号可以在 Student action space 内重新排序，因此有用迁移不要求 Student 进入 Teacher 的高-overlap regime。

Aligned case 会进入更高 post-RL overlap；cross-pattern case 则与 post-RL Teacher overlap 保持较低，与 reference overlap 也没有补偿性上升。因此图 2 的增益更符合‘在 Student 自己访问的 states 上使用 reference→post-RL direction’，而不是模仿某个 Teacher。

作者进一步检查低 overlap 是否只是 degenerate sharpening。如果 Direct-OPD 让 actor collapse 成低 entropy distribution，overlap 结论就不可靠。

Short-Horizon Training Changes Longer Rollouts

受 on-policy prefix distillation 启发，作者训练时故意只使用 2k-token response horizon，但测试时允许长生成，检验 policy change 是否超出直接监督 prefix。自然担忧是短训练只改变 early prefix，后续推理行为保持不变。

为直接测量长 rollout 行为，作者在固定长序列的每个 prefix x_≤t 上取 actor top-16 set K_t，计算 actor-probability-weighted Teacher gap。

g_t=\sum_{a\in K_t}\pi_{\mathrm{actor}}(a\mid x_{\le t})\left[\log\pi_{\mathrm{JustRL}}(a\mid x_{\le t})-\log\pi_{\mathrm{R1}}(a\mid x_{\le t})\right]

G_T=\sum_{t=1}^{T}g_t

40 steps、2k response 后，actor 沿更长固定 rollout 都朝 Teacher-shift direction 移动。6k 在 diagnostic 上移动得更远，却以 45.6 低于 2k 的 48.8；一种解释是 late prefix 上小 Teacher pair 越来越 off-distribution，log-ratio 大但不可靠。6k 会过度驱动 actor，2k 避开噪声却已捕获并泛化可靠 early-prefix direction。

KL Controls Reward Reliability

Teacher/reference log-ratio 只有在 Teacher shift 对 Student 所访问 state 有意义时才是可靠 dense reward。因此 KL 不只是常规 regularization coefficient，它还控制 Student 去哪些 rollout states，从而决定 reward 是否仍可信。KL 太弱，Student 会漂到 Teacher 与 reference 都给低概率的区域，log-ratio 成为 noisy target；KL 太强，Student 又无法跟随 policy shift。

Fixed-KL sweep 表明不存在通用最佳值；dense reward 也没有跨 pair 通用的单调规则。在一个设置中更大的正 reward 可能伴随更差 validation，在另一个设置中却追随最佳曲线。因此 reward 必须结合 rollout distribution 解读，不能脱离 Student 当前采样区域单独最大化。

Adaptive-KL run 在初始修正后让 mean dense reward 接近零。持续大 reward 可能意味着 Student 正在采样 Teacher 与 reference 强烈分歧、但两者都低 support 的 token，此时 log-ratio 数值大却不再是可信改进方向。接近零的 batch mean 代表更平衡的 regime：局部 token shift 仍可排序 action，但没有整体漂移。

On-Policy Distillation of Reasoning Models

RLVR 已成为训练强 reasoning model 的标准范式，knowledge distillation 则用于把能力扩散到其他模型。对 reasoning model 而言，Student 在自己采样的 states 上接受 Teacher supervision 的 OPD 已成为主流，并出现与 RL 结合的 hybrid 方法。

已有分析把 OPD 成功与 Student-visited states 上的 Teacher–Student top-k overlap 联系起来，也指出小 Student 难以复制强 reasoner，并将 OPD 解释为 dense KL-constrained RL。它们都在模仿或超越 Teacher 最终 policy；当 Teacher 是比目标更弱的小 post-RL model 时，这也会引入容量上限。Direct-OPD 只蒸馏 log π_T−log π_Tref，丢弃 Teacher 的绝对 policy。

Weak-to-Strong Generalization

Weak-to-strong generalization 研究弱 supervisor 能否激发更强模型的能力，承接 semi-supervised learning、noisy-label learning、scalable oversight、eliciting latent knowledge 与 easy-to-hard generalization。多数方法仍用弱模型 label 或 distribution 监督，因此 Student 容易被限制在 supervisor 附近。

与本文最接近的 weak-to-strong preference optimization 会复用弱 aligned model 的 implicit reward 去引导强模型。Direct-OPD 的 supervisor 则是一对小 RL checkpoint：它转移二者差值隐含的 reward，并证明这种 reward 可以改善已经超过 post-RL Teacher 的 Student。

Implicit Rewards and Reusing Trained Models

Direct-OPD 基于 policy-as-reward identity：KL-regularized RL 中 policy/reference log-ratio 可在常数意义下恢复 reward。DPO 等方法用它从 preference 直接拟合 policy；本文反向从 post-RL checkpoint 与 reference 读取 dense implicit reward。这与 process reward、per-token credit assignment 相关，同时避免显式 reward model 的 over-optimization。

另一类工作在 weight space 中复用模型，如 task arithmetic、model merging 与 proxy tuning。Direct-OPD 的不同在于信号是 behavioral log-ratio 而非 weight delta，并在 Student 自己的 on-policy states 上应用，因此可跨 model family 与 scale。

Direct-OPD 表明，小 RL Teacher 学到的 policy shift 可以作为 weak-to-strong generalization 的有效 dense reward。可迁移对象不是 post-RL Teacher policy，而是它相对 pre-RL reference 的 log-ratio，并在 Student-visited token 上评估；这解释了为什么更小、更弱的 Teacher 仍能改善强 Student。

实验显示该方向可跨 Teacher pair 与 Student family 迁移，以更低 compute 超过 step-matched direct RL，并可顺序组合多个 Teacher shift。论文因此把 RL outcome 重新理解为可复用 improvement signal，而不是只能模仿的最终模型。

Appendix A：Experimental Details

最终 Direct-OPD run 使用 Skywork-OR1-RL-Data 的 math subset，并采用 DAPO-style math prompt。Prompt 要求逐步解题，最后一行单独写成 ‘Answer: $Answer’。该模板同时用于训练 rollout 与 evaluation。

该 prompt 不同于 Teacher RL 使用的 boxed-answer prompt；实验中 DAPO-style prompt 略微提升迁移。作者没有把 prompt-template divergence 作为研究重点，因此统一采用该设置。将 Skywork training data 换成 DAPO-Math-17K 且保持 prompt 不变时也观察到类似趋势，说明结果并非依赖单一训练集。

Evaluation Protocol

Direct-OPD Training Hyperparameters

RL Teacher and Baseline Settings

Teacher construction 与 direct-RL baseline 都使用 GRPO。R1-Distill-1.5B/7B 的 train batch size 为 512，Qwen3-1.7B/4B-nonthinking 为 128；PPO mini batch size 均为 128，rollout n=8，maximum prompt/response length=2,048/16,384，temperature=1.0，learning rate=1×10⁻⁶，KL coefficient=0。R1 run 另外使用 clip high=0.28、clip low=0.2。

Appendix B：Additional Entropy Diagnostics

Appendix C：Additional Results

参考文献保留英文题名和原始 bibliographic 信息，不翻译文献条目。以下仅列出正文与附录实际引用的文献，并按论文首次引用顺序排列。

1. DeepSeek-AI; Guo, Daya; Yang, Dejian; Zhang, Haowei; Song, Junxiao; Wang, Peiyi; Zhu, Qihao; Xu, Runxin; Zhang, Ruoyu; Ma, Shirong; Bi, Xiao; others (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature.
2. Shao, Zhihong; Wang, Peiyi; Zhu, Qihao; Xu, Runxin; Song, Junxiao; Bi, Xiao; Zhang, Haowei; Zhang, Mingchuan; Li, Y. K.; Wu, Y.; Guo, Daya (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.
3. Schulman, John; Wolski, Filip; Dhariwal, Prafulla; Radford, Alec; Klimov, Oleg (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.
4. Kimi Team (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599.
5. Qwen Team (2025). QwQ-32B: Embracing the Power of Reinforcement Learning. https://qwenlm.github.io/blog/qwq-32b/.
6. Hu, Jingcheng; Zhang, Yinmin; Han, Qi; Jiang, Daxin; Zhang, Xiangyu; Shum, Heung-Yeung (2025). Open-Reasoner-Zero: An Open Source Approach to Scaling Reinforcement Learning on the Base Model. https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero.
7. Yu, Qiying; Zhang, Zheng; Zhu, Ruofei; Yuan, Yufeng; Zuo, Xiaochen; Yue, Yu; Fan, Tiantian; Liu, Gaohong; Liu, Lingjun; Liu, Xin; others (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476.
8. He, Bingxiang; Qu, Zekai; Liu, Zeyuan; Chen, Yinghao; Zuo, Yuxin; Qian, Cheng; Zhang, Kaiyan; Chen, Weize; Xiao, Chaojun; Cui, Ganqu; Ding, Ning; Liu, Zhiyuan (2025). JustRL: Scaling a 1.5B LLM with a Simple RL Recipe. arXiv preprint arXiv:2512.16649.
9. An, Chenxin; others (2025). POLARIS: A Post-Training Recipe for Scaling Reinforcement Learning on Reasoning Models. https://github.com/ChenxinAn-fdu/POLARIS.
10. Li, Jiazheng; Lin, Hongzhou; Lu, Hong; Wen, Kaiyue; Yang, Zaiwen; Gao, Jiaxuan; Wu, Yi; Zhang, Jingzhao (2025). QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. arXiv preprint arXiv:2507.13266.
11. Jujie He; Jiacai Liu; Chris~Yuhao Liu; Rui Yan; Chaojie Wang; Peng Cheng; Xiaoyu Zhang; Fuxiang Zhang; Jiacheng Xu; Wei Shen; others (2025). Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312.
12. Zhiwei He; Tian Liang; Jiahao Xu; Qiuzhi Liu; Xingyu Chen; Yue Wang; Linfeng Song; Dian Yu; Zhenwen Liang; Wenxuan Wang; others (2025). Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456.
13. Bingxiang He; Yuxin Zuo; Zeyuan Liu; Shangziqi Zhao; Zixuan Fu; Junlin Yang; Cheng Qian; Kaiyan Zhang; Yuchen Fan; Ganqu Cui; others (2026). How far can unsupervised rlvr scale llm training?. arXiv preprint arXiv:2603.08660.
14. Yang, An; Li, Anfeng; Yang, Baosong; Zhang, Beichen; Hui, Binyuan; Zheng, Bo; Yu, Bowen; Gao, Chang; Huang, Chengen; Lv, Chenxu; others (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.
15. DeepSeek-AI (2026). DeepSeek V4 Preview Release. https://api-docs.deepseek.com/news/news260424.
16. Z.ai (2026). GLM-5.2: Built for Long-Horizon Tasks. https://z.ai/blog/glm-5.2.
17. Moshkov, Ivan; Hanley, Darragh; Sorokin, Ivan; Toshniwal, Shubham; Henkel, Christof; Schifferer, Benedikt; Du, Wei; Gitman, Igor (2025). AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning Dataset. arXiv preprint arXiv:2504.16891.
18. Etash Guha; Ryan Marten; Sedrick Keh; Negin Raoof; Georgios Smyrnis; Hritik Bansal; Marianna Nezhurina; Jean Mercat; Trung Vu; Zayne Sprague; others (2025). Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178.
19. Jia Li; Edward Beeching; Lewis Tunstall; Ben Lipkin; Roman Soletskyi; Shengyi Huang; Kashif Rasul; Longhui Yu; Albert~Q Jiang; Ziju Shen; others (2024). Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.
20. Geoffrey Hinton; Oriol Vinyals; Jeff Dean (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
21. Kim, Yoon; Rush, Alexander M (2016). Sequence-Level Knowledge Distillation. Proceedings of EMNLP.
22. Victor Sanh; Lysandre Debut; Julien Chaumond; Thomas Wolf (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
23. Xiaoqi Jiao; Yichun Yin; Lifeng Shang; Xin Jiang; Xiao Chen; Linlin Li; Fang Wang; Qun Liu (2020). Tinybert: Distilling bert for natural language understanding.
24. Wenhui Wang; Furu Wei; Li~Dong; Hangbo Bao; Nan Yang; Ming Zhou (2020). Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
25. Seyed~Iman Mirzadeh; Mehrdad Farajtabar; Ang Li; Nir Levine; Akihiro Matsukawa; Hassan Ghasemzadeh (2020). Improved knowledge distillation via teacher assistant.
26. Jang~Hyun Cho; Bharath Hariharan (2019). On the efficacy of knowledge distillation.
27. Busbridge, Dan; Shidani, Amitis; Weers, Floris; Ramapuram, Jason; Littwin, Etai; Webb, Russ (2025). Distillation Scaling Laws. arXiv preprint arXiv:2502.08606.
28. Gu, Yuxian; Dong, Li; Wei, Furu; Huang, Minlie (2023). MiniLLM: Knowledge Distillation of Large Language Models. arXiv preprint arXiv:2306.08543.
29. Yuqiao Wen; Zichao Li; Wenyu Du; Lili Mou (2023). f-Divergence Minimization for Sequence-Level Knowledge Distillation. Proceedings of ACL.
30. Qihuang Zhong; Liang Ding; Li Shen; Juhua Liu; Bo Du; Dacheng Tao (2024). Revisiting Knowledge Distillation for Autoregressive Language Models. Proceedings of ACL.
31. Taiqiang Wu; Chaofan Tao; Jiahao Wang; Runming Yang; Zhe Zhao; Ngai Wong (2025). Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. Proceedings of COLING.
32. Jongwoo Ko; Sungnyun Kim; Tianyi Chen; Se-Young Yun (2024). DistiLLM: Towards Streamlined Distillation for Large Language Models. Proceedings of ICML.
33. Jongwoo Ko; Tianyi Chen; Sungnyun Kim; Tianyu Ding; Luming Liang; Ilya Zharkov; Se-Young Yun (2025). DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. Proceedings of ICML.
34. Yuxian Gu; Hao Zhou; Fandong Meng; Jie Zhou; Minlie Huang (2025). MiniPLM: Knowledge Distillation for Pre-Training Language Models. Proceedings of ICLR.
35. Yongchao Zhou; Kaifeng Lyu; Ankit Singh Rawat; Aditya Krishna Menon; Afshin Rostamizadeh; Sanjiv Kumar; Jean-Fran\ccois Kagy; Rishabh Agarwal (2024). DistillSpec: Improving Speculative Decoding via Knowledge Distillation. Proceedings of ICLR.
36. Wenda Xu; Rujun Han; Zifeng Wang; Long T. Le; Dhruv Madeka; Lei Li; William Yang Wang; Rishabh Agarwal; Chen-Yu Lee; Tomas Pfister (2025). Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. Proceedings of ICLR.
37. Agarwal, Rishabh; Vieillard, Nino; Zhou, Yongchao; Stanczyk, Piotr; Ramos, Sabela; Geist, Matthieu; Bachem, Olivier (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. arXiv preprint arXiv:2306.13649.
38. Kevin Lu; Thinking~Machines Lab (2025). On-policy distillation.
39. Li, Yaxuan; Zuo, Yuxin; He, Bingxiang; Zhang, Jinqian; Xiao, Chaojun; Qian, Cheng; Yu, Tianyu; Gao, Huan-ang; Yang, Wenkai; Liu, Zhiyuan; Ding, Ning (2026). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv preprint arXiv:2604.13016.
40. Song, Mingyang; Zheng, Mao (2026). A Survey of On-Policy Distillation for Large Language Models. arXiv preprint arXiv:2604.00626.
41. Yuqian Fu; Haohuan Huang; Kaiwen Jiang; Yuanheng Zhu; Dongbin Zhao (2026). Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562.
42. Woogyeol Jin; Taywon Min; Yongjin Yang; Swanand~Ravindra Kadhe; Yi~Zhou; Dennis Wei; Nathalie Baracaldo; Kimin Lee (2026). Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079.
43. Ijun Jang; Jewon Yeom; Juan Yeo; Hyunggu Lim; Taesup Kim (2026). Stable on-policy distillation through adaptive target reformulation. arXiv preprint arXiv:2601.07155.
44. Gengsheng Li; Tianyu Yang; Junfeng Fang; Mingyang Song; Mao Zheng; Haiyun Guo; Dan Zhang; Jinqiao Wang; Tat-Seng Chua (2026). Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288.
45. Tianzhu Ye; Li~Dong; Xun Wu; Shaohan Huang; Furu Wei (2026). On-policy context distillation for language models. arXiv preprint arXiv:2602.12275.
46. Tianzhu Ye; Li~Dong; Qingxiu Dong; Xun Wu; Shaohan Huang; Furu Wei (2026). Online experiential learning for language models. arXiv preprint arXiv:2603.16856.
47. Siyan Zhao; Zhihui Xie; Mengchen Liu; Jing Huang; Guan Pang; Feiyu Chen; Aditya Grover (2026). Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734.
48. Guoliang Zhao; Ruobing Xie; An~Wang; Shuaipeng Li; Huaibing Xie; Xingwu Sun (2026). Self-distillation for multi-token prediction, 2026. arXiv preprint arXiv:2603.23911.
49. Jonas Hübotter; Frederike Lübeck; Lejs Behric; Anton Baumann; Marco Bagatella; Daniel Marta; Ido Hakimi; Idan Shenfeld; Thomas Kleine Buening; Carlos Guestrin; Andreas Krause (2026). Reinforcement Learning via Self-Distillation. arXiv preprint arXiv:2601.20802.
50. Idan Shenfeld; Mehul Damani; Jonas Hübotter; Pulkit Agrawal (2026). Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897.
51. Zhang, Dongxu; Yang, Zhichao; Janghorbani, Sepehr; Han, Jun; Ressler II, Andrew; Qian, Qian; Lyng, Gregory D.; Batra, Sanjit Singh; Tillman, Robert E. (2026). Fast and Effective On-policy Distillation from Reasoning Prefixes. Findings of the Association for Computational Linguistics: ACL 2026.
52. Yuanda Xu; Hejian Sang; Zhengze Zhou; Ran He; Zhipeng Wang (2026). PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence. arXiv preprint arXiv:2603.11178.
53. Jongwoo Ko; Sara Abdali; Young~Jin Kim; Tianyi Chen; Pashmina Cameron (2026). Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137.
54. Jeonghye Kim; Xufang Luo; Minbeom Kim; Sangmook Lee; Dohyung Kim; Jiwon Jeon; Dongsheng Li; Yuqing Yang (2026). Why does self-distillation (sometimes) degrade the reasoning capability of llms?. arXiv preprint arXiv:2603.24472.
55. Tianzhu Ye; Li Dong; Zewen Chi; Xun Wu; Shaohan Huang; Furu Wei (2025). Black-Box On-Policy Distillation of Large Language Models. arXiv preprint arXiv:2511.10643.
56. Bangjun Xiao; Bingquan Xia; Bo~Yang; Bofei Gao; Bowen Shen; Chen Zhang; Chenhong He; Chiheng Lou; Fuli Luo; Gang Wang; others (2026). Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780.
57. Shicheng Xu; Liang Pang; Yunchang Zhu; Jia Gu; Zihao Wei; Jingcheng Deng; Feiyang Pan; Huawei Shen; Xueqi Cheng (2025). RLKD: Distilling LLMs' Reasoning via Reinforcement Learning. arXiv preprint arXiv:2505.16142.
58. Hongling Xu; Qi Zhu; Heyuan Deng; Jinpeng Li; Lu Hou; Yasheng Wang; Lifeng Shang; Ruifeng Xu; Fei Mi (2025). KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning. arXiv preprint arXiv:2506.02208.
59. Zhaoyang Zhang; Shuli Jiang; Yantao Shen; Yuting Zhang; Dhananjay Ram; Shuo Yang; Zhuowen Tu; Wei Xia; Stefano Soatto (2026). Reinforcement-aware Knowledge Distillation for LLM Reasoning. arXiv preprint arXiv:2602.22495.
60. Songming Zhang; Xue Zhang; Tong Zhang; Bojie Hu; Yufeng Chen; Jinan Xu (2025). AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation. Proceedings of ACL.
61. Chenxu Yang; Chuanyu Qin; Qingyi Si; Minghui Chen; Naibin Gu; Dingyu Yao; Zheng Lin; Weiping Wang; Jiaqi Wang; Nan Duan (2026). Self-distilled rlvr. arXiv preprint arXiv:2604.03128.
62. Yuetai Li; Xiang Yue; Zhangchen Xu; Fengqing Jiang; Luyao Niu; Bill~Yuchen Lin; Bhaskar Ramasubramanian; Radha Poovendran (2025). Small models struggle to learn from strong reasoners.
63. Yang, Wenkai; Liu, Weijie; Xie, Ruobing; Yang, Kai; Yang, Saiyong; Lin, Yankai (2026). Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv preprint arXiv:2602.12125.
64. Burns, Collin; Izmailov, Pavel; Kirchner, Jan Hendrik; Baker, Bowen; Gao, Leo; Aschenbrenner, Leopold; Chen, Yining; Ecoffet, Adrien; Joglekar, Manas; Leike, Jan; Sutskever, Ilya; Wu, Jeff (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision. arXiv preprint arXiv:2312.09390.
65. Leike, Jan; Sutskever, Ilya (2023). Introducing Superalignment. OpenAI Blog.
66. Grandvalet, Yves; Bengio, Yoshua (2004). Semi-supervised learning by entropy minimization. Advances in neural information processing systems.
67. Dai, Andrew M; Le, Quoc V (2015). Semi-supervised sequence learning. Advances in neural information processing systems.
68. Laine, Samuli; Aila, Timo (2016). Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242.
69. Athiwaratkun, Ben; Finzi, Marc; Izmailov, Pavel; Wilson, Andrew Gordon (2018). There are many consistent explanations of unlabeled data: Why you should average. arXiv preprint arXiv:1806.05594.
70. Han, Bo; Yao, Quanming; Yu, Xingrui; Niu, Gang; Xu, Miao; Hu, Weihua; Tsang, Ivor; Sugiyama, Masashi (2018). Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems.
71. Berthelot, David; Carlini, Nicholas; Goodfellow, Ian; Papernot, Nicolas; Oliver, Avital; Raffel, Colin A (2019). Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems.
72. Li, Junnan; Socher, Richard; Hoi, Steven CH (2020). Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394.
73. Chen, Ting; Kornblith, Simon; Swersky, Kevin; Norouzi, Mohammad; Hinton, Geoffrey (2020). Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems.
74. Chen, Yining; Wei, Colin; Kumar, Ananya; Ma, Tengyu (2020). Self-training avoids using spurious features under domain shift. Advances in Neural Information Processing Systems.
75. Chen, Baixu; Jiang, Junguang; Wang, Ximei; Wan, Pengfei; Wang, Jianmin; Long, Mingsheng (2022). Debiased self-training for semi-supervised learning. Advances in Neural Information Processing Systems.
76. Christiano, Paul; Shlegeris, Buck; Amodei, Dario (2018). Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575.
77. Leike, Jan; Krueger, David; Everitt, Tom; Martic, Miljan; Maini, Vishal; Legg, Shane (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871.
78. Irving, Geoffrey; Christiano, Paul; Amodei, Dario (2018). AI Safety via Debate. arXiv preprint arXiv:1805.00899.
79. Bowman, Samuel R; Hyun, Jeeyoon; Perez, Ethan; Chen, Edwin; Pettit, Craig; Heiner, Scott; Lukosuite, Kamile; Askell, Amanda; Jones, Andy; Chen, Anna; others (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv preprint arXiv:2211.03540.
80. Bowman, Sam (2022). Artificial Sandwiching: When can we test scalable alignment protocols without humans?. AI Alignment Forum.
81. Bai, Yuntao; Kadavath, Saurav; Kundu, Sandipan; Askell, Amanda; Kernion, Jackson; Jones, Andy; Chen, Anna; Goldie, Anna; Mirhoseini, Azalia; McKinnon, Cameron; others (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073.
82. Christiano, Paul; Cotra, Ajeya; Xu, Mark (2022). Eliciting latent knowledge.
83. Burns, Collin; Ye, Haotian; Klein, Dan; Steinhardt, Jacob (2022). Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827.
84. Schwarzschild, Avi; Borgnia, Eitan; Gupta, Arjun; Huang, Furong; Vishkin, Uzi; Goldblum, Micah; Goldstein, Tom (2021). Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems.
85. Schwarzschild, Avi; Borgnia, Eitan; Gupta, Arjun; Bansal, Arpit; Emam, Zeyad; Huang, Furong; Goldblum, Micah; Goldstein, Tom (2021). Datasets for studying generalization from easy to hard examples. arXiv preprint arXiv:2108.06011.
86. Boxi Cao; Keming Lu; Xinyu Lu; Jiawei Chen; Mengjie Ren; Hao Xiang; Peilin Liu; Yaojie Lu; Ben He; Xianpei Han; Le Sun; Hongyu Lin; others (2024). Towards Scalable Automated Alignment of LLMs: A Survey. arXiv preprint arXiv:2406.01252.
87. Wei Yao; Wenkai Yang; Ziqiao Wang; Yankai Lin; Yong Liu (2025). Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL. arXiv preprint arXiv:2502.11107.
88. Wei Yao; Gengze Xu; Huayi Tang; Wenkai Yang; Donglin Di; Ziqiao Wang; Yong Liu (2025). On Weak-to-Strong Generalization and f-Divergence. arXiv preprint arXiv:2506.03109.
89. Jitao Sang; Yuhang Wang; Jing Zhang; Yanxu Zhu; Chao Kong; Junhong Ye; Shuyu Wei; Jinlin Xiao (2024). Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning. arXiv preprint arXiv:2402.00667.
90. Yige Yuan; Teng Xiao; Shuchang Tao; Xue Wang; Jinyang Gao; Bolin Ding; Bingbing Xu (2025). Incentivizing Strong Reasoning from Weak Supervision. arXiv preprint arXiv:2505.20072.
91. Yuan, Weizhe; Pang, Richard Yuanzhe; Cho, Kyunghyun; Sukhbaatar, Sainbayar; Xu, Jing; Weston, Jason (2024). Self-Rewarding Language Models. arXiv preprint arXiv:2401.10020.
92. Zhu, Wenhong; He, Zhiwei; Wang, Xiaofeng; Liu, Pengfei; Wang, Rui (2025). Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model. arXiv preprint arXiv:2410.18640.
93. Rafailov, Rafael; Sharma, Archit; Mitchell, Eric; Ermon, Stefano; Manning, Christopher D.; Finn, Chelsea (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290.
94. Azar, Mohammad Gheshlaghi; Rowland, Mark; Piot, Bilal; Guo, Daniel; Calandriello, Daniele; Valko, Michal; Munos, Rémi (2024). A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS.
95. Ethayarajh, Kawin; Xu, Winnie; Muennighoff, Niklas; Jurafsky, Dan; Kiela, Douwe (2024). KTO: Model Alignment as Prospect Theoretic Optimization. arXiv preprint arXiv:2402.01306.
96. Meng, Yu; Xia, Mengzhou; Chen, Danqi (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS.
97. Lightman, Hunter; Kosaraju, Vineet; Burda, Yura; Edwards, Harri; Baker, Bowen; Lee, Teddy; Leike, Jan; Schulman, John; Sutskever, Ilya; Cobbe, Karl (2023). Let's Verify Step by Step. arXiv preprint arXiv:2305.20050.
98. Wang, Peiyi; Li, Lei; Shao, Zhihong; Xu, RX; Dai, Damai; Li, Yifei; Chen, Deli; Wu, Y; Sui, Zhifang (2023). Math-Shepherd: A Label-Free Step-by-Step Verifier for LLMs in Mathematical Reasoning. arXiv preprint arXiv:2312.08935.
99. Cui, Ganqu; Yuan, Lifan; Wang, Zefan; Wang, Hanbin; Li, Wendi; He, Bingxiang; Fan, Yuchen; Yu, Tianyu; Xu, Qixin; Chen, Weize; others (2025). Process Reinforcement through Implicit Rewards. arXiv preprint arXiv:2502.01456.
100. Kazemnejad, Amirhossein; Aghajohari, Milad; Portelance, Eva; Sordoni, Alessandro; Reddy, Siva; Courville, Aaron; Le Roux, Nicolas (2024). VinePPO: Unlocking RL Potential for LLM Reasoning through Refined Credit Assignment. arXiv preprint arXiv:2410.01679.
101. Zhang, Chenchen; others (2026). From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models. arXiv preprint arXiv:2604.09459.
102. Gao, Leo; Schulman, John; Hilton, Jacob (2023). Scaling Laws for Reward Model Overoptimization. ICML.
103. Ilharco, Gabriel; Ribeiro, Marco Tulio; Wortsman, Mitchell; Schmidt, Ludwig; Hajishirzi, Hannaneh; Farhadi, Ali (2023). Editing Models with Task Arithmetic. ICLR.
104. Wortsman, Mitchell; Ilharco, Gabriel; Gadre, Samir Yitzhak; Roelofs, Rebecca; Gontijo-Lopes, Raphael; Morcos, Ari S; Namkoong, Hongseok; Farhadi, Ali; Carmon, Yair; Kornblith, Simon; others (2022). Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time. ICML.
105. Liu, Alisa; Han, Xiaochuang; Wang, Yizhong; Tsvetkov, Yulia; Choi, Yejin; Smith, Noah A. (2024). Tuning Language Models by Proxy. arXiv preprint arXiv:2401.08565.