Fine-tuning

Fine-tuning takes a model that was pretrained on broad data and continues training it, at a lower learning rate, on data for a specific task. The model starts with useful general representations and adjusts them to the target problem, which is why fine-tuning reaches strong results with far less data and compute than training from scratch.

You can fine-tune the whole model or just part of it, and parameter-efficient methods update only a small set of added weights to save cost.

Starting from pretrained weights, the model is trained further on the target dataset, usually with a small learning rate so it refines rather than overwrites what it already knows. You might update all layers, freeze early layers and tune later ones, or attach lightweight adapter parameters and train only those. The right choice depends on how much target data you have and how close it is to the pretraining data.

Fine-tuning is how foundation models become useful in specific settings, and it is often the highest-leverage step in a pipeline. Because its results hinge on the quality of the task-specific data, fine-tuning also makes data curation and clean labels directly responsible for final model quality.

Usually far less than training from scratch, since the model already carries general knowledge and only needs to specialize.

It updates only a small number of added or selected weights rather than the whole model, which lowers memory and compute costs while retaining most of the benefit.

Transfer learning, Self-supervised learning, Knowledge distillation, Foundation model labeling.