What is data lineage?
Data lineage is the traceable history of a dataset: where each piece of data originated, what transformations it underwent, and how it arrived in its current form. It answers questions like which source a sample came from, which cleaning steps touched it, and which annotation pass produced its label.
Where versioning captures the states of a dataset, lineage captures the path between them.
Key takeaways
- Lineage traces data's origin and its transformation history.
- It answers where a sample and its label came from.
- It underpins debugging, reproducibility, and compliance.
How it works
Lineage is captured by recording metadata at each pipeline stage, from ingestion through cleaning, labeling, and augmentation, and linking those records to the samples they affect. The result is a traceable graph you can follow backward from a final training sample to its raw source and every step in between.
Why it matters
When a model behaves unexpectedly, lineage lets you trace the problem to a specific source or processing step rather than guessing. It is also increasingly required for governance and compliance, where you must prove exactly what data was used and how it was handled.
Frequently asked questions
What questions does data lineage answer?
Where a sample originated, what transformations it went through, and which processing or labeling steps produced its current state.
Why is lineage important for compliance?
Regulations often require showing where data came from and how it was processed, and lineage provides that auditable trail.
Related terms