Fine-Tuning & Model Training

The decision comes first

Fine-tuning is the most over-reached-for tool in applied AI. It is also genuinely the right answer sometimes. Getting the decision right matters more than executing it well.

The line that resolves most cases:

Fine-tuning teaches behaviour. Retrieval supplies knowledge.

Fine-tuning on your product catalogue does not give a reliable product catalogue. It gives a model fluent in the style of your catalogue, which will confidently invent plausible products. Every fact that changes becomes a retraining run.

The two genuinely strong cases, both about economics rather than capability:

• Prompt distillation. A 3,000-token system prompt sent millions of times, folded into weights. Real, large, ongoing savings.
• Small-model specialisation. A small tuned model matching a frontier model on one narrow task, at a fraction of the cost and latency. This is where most production value lives.

LoRA

Full fine-tuning updates every parameter — prohibitive in memory and compute, and it produces a full model copy per task.

LoRA starts from the observation that the update needed to adapt a model has low intrinsic rank. So freeze the original weight $W$ and learn a low-rank correction:

$W' = W + BA$

with $W \in \mathbb{R}^{d \times k}$, $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$.

For $d = k = 4096$ and $r = 16$, that is 16.8M parameters replaced by 131K — about 0.8%. Typical results reach most of full fine-tuning quality on well-scoped tasks.

Three practical consequences that matter more than the mathematics:

• Adapters are small and swappable. Megabytes, not gigabytes. Serve one base model with many adapters loaded per request.
• The base model is untouched, so catastrophic forgetting is bounded and you can always fall back.
• Rank is the main dial. $r = 8$–16 for style and format; 32–64 for substantial behavioural change. Higher is not automatically better — it costs memory and increases overfitting risk on small datasets.

Apply adapters to the attention projections at minimum; including the feed-forward layers helps for larger behavioural shifts, at more memory.

QLoRA goes further: quantise the frozen base to 4-bit and train adapters on top in higher precision. This brings large-model fine-tuning within reach of a single GPU. The quality cost is small; the accessibility gain is enormous.

The dataset is the project

Allocate your effort accordingly: most of the work is data, not training. A clean set of 500 examples beats a noisy 5,000 reliably.

Format is conversational, matching inference:

{"messages": [
  {"role": "system", "content": "..."},
  {"role": "user", "content": "..."},
  {"role": "assistant", "content": "..."}
]}

Quantity, roughly: 50–100 examples for a narrow format or style change with LoRA; 500–5,000 for substantive behavioural change; more only with strong evidence it helps.

Quality criteria, in order of how badly violating them hurts:

1. Consistency. Every example must follow the same conventions. Contradictory examples teach the model that the mapping is arbitrary — actively worse than no training.
2. Correctness. Errors are learned faithfully.
3. Diversity. Cover the real input distribution, not just the easy centre.
4. Difficulty. Examples the base model already handles perfectly teach nothing. Mine your failures.
5. Distributional match. Training examples should look like production inputs. Clean curated prose fine-tuned for messy real input is a common and quiet failure.

The system prompt should match inference. Train with the system prompt you will deploy with, or the model learns behaviour conditioned on a context it will never see.

Holdout discipline

Split before you start: train, validation, and a test set you look at once. Fine-tuning has a strong pull towards iterating against the validation set until it means nothing. The test set exists to tell you the truth at the end.

Alignment

Supervised fine-tuning teaches a good answer. Preference methods teach which of two answers is better — useful for tone, helpfulness and safety, where quality is comparative rather than absolute.

RLHF collects human preferences, trains a reward model, then optimises the policy against it with reinforcement learning. Powerful, and operationally heavy: three models in play, and PPO is notoriously sensitive to hyperparameters.

DPO shows the reward model can be skipped. The optimal policy under the RLHF objective has a closed form, which turns preference learning into a direct objective over preference pairs:

$\mathcal{L}_{\text{DPO}} = -\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta\log\frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$

where $y_w$ is preferred, $y_l$ rejected, $\pi_{\text{ref}}$ the frozen reference policy, and $\beta$ controls divergence from it.

Simpler, more stable, no reward model, no RL loop. It is the default for most teams doing preference tuning below frontier-lab scale.

Most teams do not need either. Preference tuning is for when you have real preference data — user thumbs, expert rankings — and supervised fine-tuning has plateaued. Reach for it third, not first.

Evaluation

Everything from lesson 7 applies, plus specifics.

Baseline before training. Run your eval set on the base model. Without it you cannot claim an improvement, and teams routinely ship fine-tunes that are worse.

Watch for catastrophic forgetting. Narrow training erodes unrelated capability. Include general cases outside your fine-tuning distribution in the eval set — this failure is invisible without them, and it is the most common way a fine-tune quietly damages a product.

Compare against the honest alternative. Not against nothing — against a well-engineered prompt on the base model. Often that wins, and finding out after the training run is an expensive lesson.

Perplexity is a training diagnostic, not a quality metric. It tells you the model fits the distribution. It does not tell you the outputs are good. BLEU and ROUGE are similarly weak for open-ended generation.

Operations

• Version datasets like code. You will iterate many times, and “which data produced this checkpoint?” must be answerable.
• Log the whole configuration — base model, rank, learning rate, epochs, data version.
• Two to three epochs is typical. More usually overfits; watch validation loss diverge from training loss.
• Plan for the base model being deprecated. Your adapter is tied to it. Budget for retraining.
• Deploy behind a flag and A/B against the base. Real traffic finds what evals miss.

Practical order

1. Engineer the prompt properly. Measure.
2. Add few-shot examples. Measure.
3. Add retrieval if the gap is knowledge. Measure.
4. Still short, and the gap is behavioural? Build a dataset.
5. LoRA on the smallest capable base model.
6. Evaluate against the prompted baseline, including general cases.
7. Consider preference tuning only if SFT has plateaued.

Steps 1–3 resolve the large majority of cases. That is not a reason to skip fine-tuning; it is the reason it works when you do reach it, because by then you know precisely what gap you are closing.

Try this: Before any fine-tuning project, write down the exact eval metric that will move and by how much. If you cannot state it, you are not ready to train — you are hoping. That sentence has saved more compute budget than any optimisation.