a visual guide to model post-training

Pretraining gives a language model broad capabilities. Post-training decides which capabilities appear, what behavior counts as good, and how strongly the model should move toward it. The field's alphabet soup—SFT, RLHF, RLVR, PPO, GRPO, DAPO, GSPO, and DPO—becomes manageable once we separate the data, the learning signal, and the optimizer.

The post-training landscape

Select a route below. Each uses different data and supervision, but all are trying to make desirable responses more likely.

data

prompt + ideal response

→

learning signal

next-token targets

→

policy change

imitate the demonstrated response

Typical result: instruction following and response format.

A modern model rarely uses only one route. A common sequence is instruction tuning, followed by verifiable-reward training for reasoning, rejection sampling on successful trajectories, and preference tuning for qualities that cannot be mechanically checked.

Stage 1: instruction fine-tuning teaches the interface

Instruction fine-tuning—often called IFT or SFT—uses the same next-token prediction objective as pretraining, but on demonstrations of the behavior we want. Given a prompt and a high-quality response, the model learns to imitate that response token by token.

L_{\mathrm{SFT}}(\theta) = -\sum_{t=1}^{T}\log \pi_\theta(y_t \mid x,y_{<t})

(1)

SFT is especially good at teaching chat templates, response structure, tool-call syntax, tone, and task conventions. It can elicit capabilities already latent in the base model, but imitation alone does not answer every alignment question. There may be many valid responses, and the best one may be difficult to specify as a single target.

Data matters most

High-quality demonstrations should resemble the prompts and behaviors expected downstream.

Mask the prompt

Training commonly applies loss to assistant responses rather than teaching the model to predict user messages.

Use gentle optimization

Smaller batches and learning rates help preserve the useful behavior acquired during pretraining.

Expect diminishing returns

More demonstrations are not automatically better; curation and task match often matter more than raw volume.

Stage 2: define what “good” means

Post-training can learn from subjective preferences or objective checks. These are different reward sources, not different policy optimizers.

Signal	What supplies it?	Best suited for	Main limitation
Preference reward	Human or AI comparisons	Helpfulness, tone, style, safety	A learned proxy can be exploited
Outcome verifier	Answer checker or environment	Math, code, tools, structured output	Does not validate the reasoning path
Process reward	Step judge or process reward model	Intermediate reasoning quality	Expensive, detailed supervision
Composite reward	Weighted checks	Correctness plus format or style	Auxiliary signals invite reward hacking

Preference reward models

Classic RLHF asks annotators to choose between two responses. A Bradley–Terry reward model gives each response a scalar score and learns that the chosen response should score higher:

P(y_c > y_r \mid x) = \sigma\left(r_\phi(x,y_c)-r_\phi(x,y_r)\right)

(2)

Only the score difference matters. The reward model does not need an absolute definition of “8.7 quality”; it needs to rank the preferred completion above the rejected one.

Verifiable rewards

Reinforcement learning with verifiable rewards, or RLVR, replaces the learned preference model with a programmatic check:

R_{\mathrm{verify}}(x,y)=\begin{cases}1,&\operatorname{verify}(x,\operatorname{extract}(y))\\0,&\text{otherwise}\end{cases}

(3)

A math checker compares the extracted answer, code runs against tests, and a tool-use environment checks whether the requested state actually changed. The verifier supplies the reward; PPO, RLOO, GRPO, or another algorithm decides how that reward changes the model.

Outcome, process, and token granularity

Even after choosing a reward source, we must decide where feedback attaches. Use the same response below to compare one final score, step-by-step supervision, and illustrative token feedback.

One score for the response

The verifier sees the correct final answer and returns +1. Earlier mistakes receive the same signal.

7 × 8 = 54. Wait — let me recheck. 7 × 8 = 56. Final answer: 56.

reward +1.0

Illustrative rewards, not model-generated labels

Outcome supervision is common because it is cheap and reliable when the final result is checkable. It reinforces the entire trajectory associated with success, including any accidental or invalid reasoning inside it. Process supervision improves credit assignment but requires a trustworthy judge for intermediate steps.

Stage 3: turn reward into a policy update

Reward says how good the sampled response was. Advantage says whether it was better or worse than a baseline. The policy update then changes the probability of the sampled behavior.

1 · reward

Was the sampled response good?

2 · advantage

Was it better than its baseline?

3 · update

How much should its probability move?

All the policy-gradient methods below share the same basic direction:

\nabla_\theta J(\theta)\approx\mathbb{E}\left[\sum_t A_t\nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t})\right]

(4)

Positive advantage makes a sampled action or completion more likely; negative advantage makes it less likely. The methods differ mainly in how they construct the baseline and constrain update size.

Method	Baseline	Value model	Update constraint
PPO	Learned expected return	Yes	Token policy-ratio clipping
RLOO	Mean of the other completions	No	Usually REINFORCE-style
GRPO	Normalized group mean	No	Token policy-ratio clipping

GRPO: compare answers within a group

GRPO samples several responses for the same prompt and normalizes each reward relative to the group:

A_i^{\mathrm{GRPO}} = \frac{R_i-\operatorname{mean}(R)}{\operatorname{std}(R)}

(5)

Click individual answers below or try the presets. Mixed outcomes create a ranking signal. Uniform outcomes do not.

group mean0.50

group std0.50

learning signalrelative contrast

GRPO can rank these answers. Positive advantages increase probability; negative advantages decrease it.

This is why useful RLVR prompts live near the model's capability frontier: it should sometimes succeed and sometimes fail. Prompts solved zero or one hundred percent of the time give a group-relative method no direction.

DAPO: make the GRPO recipe survive long reasoning

DAPO retains the group-relative foundation and adds four practical changes aimed at long chain-of-thought training. The comparison below keeps each intervention on its own axis so “token-level” does not get confused with token-level reward.

1 · Clip-higher

exploration

Decouple the lower and upper bounds, then give useful rare tokens more room to grow.

GRPO1.20×

DAPO1.28×

2 · Dynamic sampling

useful batches

Keep mixed groups with contrast; replace groups whose rewards are all identical.

1 1 0 0train

1 1 1 1resample

0 0 0 0resample

3 · Token-level loss

aggregation

GRPO gives responses equal weight. DAPO gives generated tokens equal weight.

GRPO50 / 50

DAPO25 / 75

4 · Overlong shaping

stable length

Introduce a soft penalty zone before truncation instead of one abrupt reward cliff.

full rewardsoft penaltylimit

GSPO: move the clipping decision to the response

GRPO and DAPO ask how far each token moved from the rollout policy. GSPO computes one length-normalized importance ratio for the complete response:

\rho_i = \exp\left(\frac{1}{T_i}\sum_t \log \frac{\pi_\theta(y_{i,t})}{\pi_{\mathrm{old}}(y_{i,t})}\right)

(6)

Move the rare-token ratio below. GRPO clips that token once it crosses its ceiling. DAPO's higher ceiling preserves more exploration. GSPO averages movement across the response and makes one coherent decision.

This control isolates the clipping difference. DAPO's sampling, loss-aggregation, and length changes are shown in the four-part recipe above.

Probability ratio for the token “56”1.65×

Positive-advantage response

GRPO · one clipping decision per token

upper clip 1.20×

Check1.04×update

70.98×update

×1.05×update

80.96×update

=1.02×update

561.65×clipped

.1.03×update

DAPO · token decisions with a higher ceiling

illustrative upper clip 1.28×

Check1.04×update

70.98×update

×1.05×update

80.96×update

=1.02×update

561.65×clipped

.1.03×update

GSPO · one clipping decision for the response

geometric mean

Check 7 × 8 = 56 .1.09×whole response updating

KL regularization: improve without drifting too far

Reward optimization can exploit weaknesses in the judge or push the model away from generally useful behavior. A frozen reference policy, usually the SFT checkpoint, supplies an anchor:

R_{\mathrm{shaped}}(x,y)=r_\phi(x,y)-\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}

(7)

Larger $\beta$ charges the policy more for behavior drift. Smaller $\beta$ permits more aggressive reward optimization.

KL pressure β0.50

Bold response

Higher judge score, farther from the reference

1.20

1.80 reward − 0.50 × 1.20 drift

Familiar response

Lower judge score, closer to the reference

1.20

1.25 reward − 0.50 × 0.10 drift

Higher β increasingly favors behavior that stays close to the reference policy. Values are illustrative.

Direct preference optimization takes an offline route

Online RL repeatedly samples from the current policy, evaluates fresh responses, and updates from those rollouts. Direct alignment methods such as DPO instead train directly on a static dataset of chosen and rejected responses. They avoid a separately deployed reward model and a full online rollout loop.

Online RL

Generate → score → estimate advantages → update → generate again. Powerful for exploration and verifiable environments, but operationally complex.

Direct preference

Load chosen/rejected pairs → optimize preference likelihood. Simpler and offline, but limited by what the static dataset already covers.

Rejection sampling sits between these worlds: sample many candidates, keep the highest-scoring ones, and fine-tune on them as supervised targets. It converts a reward function into a cleaner imitation dataset.

Inference-time scaling is not training

Post-training changes model weights. Inference-time scaling spends more computation after training: produce longer reasoning, sample more candidates, search, or rerank with a verifier. The two interact—RLVR can teach productive reasoning behaviors—but extra tokens are useful only when they improve the final result.

A practical hybrid recipe

01Cold-start SFT: teach the format, task interface, and a small set of strong reasoning demonstrations.
02RLVR: optimize on math, code, logic, tools, or other tasks with reliable checks.
03Rejection sampling: collect successful trajectories and turn them into new supervised data.
04Preference tuning: refine helpfulness, safety, tone, and qualities that verifiers cannot capture.

For smaller models, distilling successful reasoning traces from a stronger model can be more sample-efficient than asking RL to discover every behavior from scratch.

The durable mental model

Keep the axes separate. SFT versus preference data describes the supervision. A reward model versus verifier describes who judges. Outcome versus process describes where feedback attaches. PPO, RLOO, GRPO, DAPO, and GSPO describe how online reward changes the policy. DPO describes a direct offline route. KL controls how far the result may move.

In one line: curate the right behavior, define a trustworthy signal, make better responses more likely, and keep the policy from learning the wrong lesson.