Wiki · Concept · Last reviewed July 10, 2026

OpenLineage

OpenLineage is an open standard for collecting lineage metadata from data pipelines: jobs, runs, datasets, events, and facets that reconstruct how data moved and changed.

Category: Concept Published: July 10, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: OpenLineage, data lineage, MLOps, AI governance, provenance

Definition

OpenLineage is an open framework for collecting and analyzing data lineage. Its project site describes it as tracking metadata about datasets, jobs, and runs, while the documentation defines the standard as an extensible way for systems to interoperate with lineage metadata. Pipeline components such as schedulers, warehouses, analysis tools, and SQL engines can send lineage events to a compatible backend.

The project is not an AI model and not a model registry. It is evidence infrastructure for data movement and transformation. For AI systems, that matters because training sets, evaluation tables, embedding indexes, prompt corpora, logs, and monitoring datasets become harder to govern when nobody can say which job produced them and from which inputs.

LF AI & Data lists OpenLineage as a graduation-stage foundation project. The same project page says Datakin contributed it in May 2021 as an incubation-stage project and that it graduated in July 2023.

How It Works

The core OpenLineage model centers on three entities. A Job is a process that consumes or produces datasets. A Run is one occurrence of that job in time, identified by a run ID. A Dataset is an abstract representation of data, such as a database table, object-store path, file, stream, topic, or other discrete collection.

OpenLineage supports both runtime and design-time lineage. Its object-model documentation names RunEvent for job run state updates, JobEvent for job metadata updates, and DatasetEvent for dataset metadata updates. Runtime events commonly mark job state transitions such as start, completion, failure, or abort; design-time events can describe static job or dataset metadata without being attached to a particular run.

Facets are how the model becomes useful without becoming one giant fixed schema. The documentation defines a facet as an atomic piece of metadata that can be attached to a run, job, input dataset, or output dataset. Facets can describe source-code location, parent runs, SQL, schema, data source, lifecycle changes, dataset version, data quality, and output statistics.

The dataset-type facet is a useful boundary reminder. The documented values include TABLE, VIEW, FILE, TOPIC, STREAM, MODEL, and JOB_OUTPUT. That vocabulary helps distinguish a persisted table from a temporary job output, but it does not decide whether the artifact is legitimate.

AI Governance Context

AI governance often fails at the gap between a model card and the data system underneath it. The card may name a dataset, but the organization may not preserve the transformation chain: source tables, filters, joins, deduplication, labeling, redaction, feature extraction, vectorization, benchmark construction, and monitoring exports. OpenLineage gives those steps a portable event language.

That matters for agents. A data agent may read from a warehouse, write a derived table, update an embedding index, run a quality test, and trigger a report. A lineage event record can keep those steps connected to job names, run IDs, datasets, and producer metadata.

OpenLineage should sit beside AI Data Provenance, AI Audit Trails, AI System Inventory, and MLflow. Provenance names origin, audit trails preserve actions, inventories identify deployed systems, and MLflow records experiments and model artifacts. OpenLineage contributes the pipeline graph.

Minimum Record

A governance-grade OpenLineage record should identify the OpenLineage version, producer, backend, job namespace and name, run ID, event type, event time, source-code location where available, parent run, input datasets, output datasets, schema facets, dataset versions, data-quality facets, ownership or catalog links, and retention class. For AI pipelines, it should also connect to dataset licenses, evaluation-set status, model registry entries, prompt corpus version, and incident-review links.

The review question is not "does the team have a lineage graph?" It is whether the graph can answer contested questions: which job created the evaluation table, which sources fed a retrieval index, which run produced the served artifact, and which quality assertion failed before deployment.

Limits

OpenLineage is not an access-control layer, consent system, data catalog, privacy review, safety evaluation, or guarantee that a pipeline behaves as documented. It records lineage metadata emitted by instrumented systems. Missing instrumentation, stale schemas, omitted facets, or downstream neglect can make the graph incomplete.

Lineage can also become surveillance. Dataset names, job names, ownership facets, quality metrics, and run metadata may reveal sensitive business processes or regulated data flows. A lineage backend therefore needs its own access controls, redaction rules, retention limits, export policy, and incident playbook.

Source Discipline

Claims about OpenLineage should cite the official project site, current documentation, object model, facet specification, and LF AI & Data project page. Implementation claims should name the specific integration and version rather than saying a pipeline "uses OpenLineage."

Spiralist Reading

Spiralism reads OpenLineage as a memory format for the data factory.

The lineage graph says that a table, model artifact, metric, or index did not simply appear. It was made by a job, during a run, from named inputs, through a chain of transformations. That is modest, but powerful. A machine-mediated institution becomes more accountable when its outputs can be traced back to the work that made them.

Sources

Return to Wiki