How Much of Post-Training Is an Input?

A sentence in the prompt can change a language model’s output completely. I correct my AI agents this way every day, which raises a question: is a correction in words the same kind of change as post-training, only cheaper? We measured it. For a stage that turns a model P into a model T, the share of the stage that an input c to P gives back is

\[R(c) = 1 - \frac{\mathrm{KL}(T\,\Vert\,P_c)}{\mathrm{KL}(T\,\Vert\,P)},\]

with both KL divergences taken on T’s own answers to held-out prompts. R = 1 means the input reproduces the stage, 0 means it does nothing, and below 0 means it moves P away from T.

No text gives a DPO stage back; one learned vector gives back half. On the DPO stages of OLMo 2 at 1B and 32B and Tulu 3 at 8B, we tried hand-written instructions, 96 prompts a 27B model wrote while seeing their scores, tokens optimized by gradient, and T’s own answers as examples. Learned inputs were fitted on T’s answers to 3,000 other prompts.

	1B	8B	32B
Best text	−0.04	−0.15	0.02
One learned vector in the system prompt	0.51	–	–
Sixteen learned vectors	0.64	0.66	0.55
LoRA, rank 64 (what the data supports)	0.89	0.74	–

At 8B, the sixteen vectors bring the reward model’s score and IFEval, GSM8K and MATH accuracy to T’s level; the text that does best moves the reward a third of the way. Scaling a text’s effect up or down does not help, because the change it makes is close to orthogonal to the stage’s.

Even when a sentence produced the preferences, training does not land on the sentence. Maybe what DPO learns from human or AI preferences just cannot be said. To test that, we built stages whose answer is a sentence w. For each prompt, an answer from P with w in the system prompt is preferred over an answer from P without it. DPO on these pairs has \(P_w\), P with the sentence, as its exact optimum at β = 1, and so do fine-tuning on the preferred answers and distilling \(P_w\) token by token. We ran all three on the same 2,800 prompts with the same adapter and steps:

Three runs toward the same target, P with the text (star), on the same answers. Each point is a checkpoint, placed by how far the model has moved from P and how much of that move the text gives back at its best strength.

Distillation heads for the sentence. Fine-tuning gets halfway and then moves past it. DPO puts about a fifth of its change along the sentence from its first steps, while moving about as far as the others.

All three start in the same expected direction. At the start, the rejected answers in DPO add nothing on average, and distillation’s gradient is fine-tuning’s averaged over every possible next token instead of the one sampled. What differs is how much of the target each example carries: whole next-token distributions, one sampled answer, or one sampled answer minus another that is pure noise in any single pair. The less of the target an example carries, the further training wanders from it.

A correction in words and a training step have the same form but carry different amounts. Adding w to the prompt gives exactly the optimum of KL-regularized RL with reward \(\log P_w - \log P\) (Korbak et al. 2022). But the sentence delivers the whole shift at once, while preferences and rewards deliver samples of it, and training on samples also moves the model along whatever is particular to them. A real DPO stage learns from preferences between sampled answers, so it is not P under any sentence, though it is close to P under one learned vector. To put a sentence into the weights, distill it; DPO on pairs built from it, as in Open Character Training, mostly goes elsewhere.

These are small models: most runs are at 1B, the sentence stages use two texts, one learning rate and one answer per prompt, and “no text” means none we found.