OpenLineage

OpenLineage is an open standard for collecting data lineage. This guide explains how to send OpenLineage RunEvent, JobEvent, and DatasetEvent data to TrueWatch from Airflow, Spark, dbt, or an application using an OpenLineage client. You can send directly to DataWay or through DataKit.

Direct DataWay: Prerequisites¶

These prerequisites and token settings apply only to direct DataWay ingestion. Before you begin, prepare the following:

• The data ingestion URL for your site, such as https://openway.truewatch.com. Use the actual URL for other sites or private deployments.
• The token for the target workspace. Treat the token as a secret; do not hard-code it or commit it to version control.
• A data job or client that supports the OpenLineage HTTP transport.
• HTTPS connectivity from the job runtime to the ingestion URL.

The examples use these placeholders:

Direct DataWay: Ingestion Protocol¶

The complete URL is:

<OPENLINEAGE_BASE_URL>/v1/write/openlineage

Each request accepts one event. JSON arrays and the OpenLineage batch endpoint are not supported. Keep each event well below the limit so that large facets do not interfere with delivery or retries.

DataWay Authentication¶

The endpoint supports these authentication methods:

If a request includes more than one token, the precedence is query token, X-Token, then Bearer. Do not configure different tokens in the same request. The current DataWay regular disk cache may store request tokens in plaintext; restrict access to the cache directory when caching is enabled.

Send Through DataKit¶

You can also send OpenLineage events to DataKit when its ddtrace collector is enabled. The collector registers POST /openlineage/api/v1/lineage by default, independently of the trace endpoints list and without another switch. DataKit normally listens on HTTP port 9529. For access from another host or Pod, configure a reachable listener and restrict network access; see the DDTrace collector guide.

After saving the example event in “Quick Verification” below as event.json, send it directly:

curl --fail-with-body --silent --show-error \
  -X POST 'http://127.0.0.1:9529/openlineage/api/v1/lineage' \
  -H 'Content-Type: application/json' \
  --data-binary @event.json

Each request still contains one RunEvent, JobEvent, or DatasetEvent JSON object; Content-Encoding: gzip is supported. max_lineage_body_mb defaults to a 64 MiB limit on the received body, measured before decompression; downstream still limits each decoded event to 8 MiB. DataKit synchronously forwards the original bytes to DataWay without parsing the event or keeping a local persistent queue. Client supplied query token, Bearer, and X-Token credentials cannot override DataKit's configured DataWay token.

DataKit returns the DataWay HTTP status, successful response body, and Retry-After. Retry a 503 according to that header. A 200 body with error_code=dataway.cached means DataWay cached the event, so the client need not retry immediately. Other 2xx responses also only acknowledge receipt; Kodo's asynchronous validation and LN ingestion may still be pending. DataWay currently uses its regular disk cache, which may store forwarded credentials in plaintext and evict older entries under pressure; protect the cache directory and monitor eventual data visibility.

Event Requirements¶

Events must follow the OpenLineage specification. All three types require eventTime (an ISO 8601 timestamp), producer (a producer URI), and schemaURL (the specification version URL). TrueWatch identifies the type from the top-level run, job, and dataset objects:

Include inputs and outputs when a RunEvent or JobEvent has input or output datasets; each dataset should have a namespace and name. Events for the same run must keep the same runId, job.namespace, and job.name. A run normally emits START followed by a terminal COMPLETE, FAIL, or ABORT.

Event Sources and Record Types¶

After ingestion, source identifies the original OpenLineage event type; lineage_record_type identifies the record TrueWatch projected from that event. These are different dimensions: one RunEvent can produce multiple records, all with source=run_event.

An event record stores the complete, redacted event JSON in lineage_message. job and dataset are entity records; dataset_edge links a Job/Run to an input or output Dataset; schema_version stores a schema summary; and field_edge represents column-level lineage. Without relevant inputs, outputs, or facets, the corresponding derived records are not created. Rejected-event audit records may use source=unknown_event and lineage_record_type=ingest_error; this is not a fourth supported input type.

TrueWatch reserves source for event classification. If the original top-level JSON contains a string source, projected records preserve it as src_source instead of replacing TrueWatch's source. It also remains in the event record's lineage_message.

Queryable Field Inventory¶

The following are top-level fields extracted by TrueWatch, not every field in the original OpenLineage JSON. Presence depends on the event type, record type, and available facets; an absent field is not necessarily an empty value. For the complete event, inspect lineage_message on a lineage_record_type=event record (sensitive values may be redacted). Use event_id to relate records projected from the same event; run_id, job_id, and dataset_id relate runs, jobs, and datasets respectively.

Common Fields and Event Payload¶

Run and Job Fields¶

Identity fields propagate to records derived from the same event. Job names, state, and facet-extracted fields are mainly on event; Job metadata from a JobEvent is also on the job record.

Dataset, Schema, and Lineage Edge Fields¶

Only DatasetEvent produces a dataset record. RunEvent and JobEvent produce dataset_edge records from their inputs and outputs. schema_version holds a schema summary; the complete schema remains in the original facet inside lineage_message.

ingest_error records additionally have ingest_error_reason_code (rejection reason), ingest_error_payload_hash (payload digest), and ingest_error_payload_size (payload size in bytes). They never store the rejected raw payload.

Direct DataWay: Quick Verification¶

Save the following as event.json, and generate a new runId for each test:

{
  "eventType": "START",
  "eventTime": "2026-10-21T08:00:00Z",
  "run": {
    "runId": "4a8f90a7-79c0-4f80-89e7-6b6c5d4fc79a"
  },
  "job": {
    "namespace": "example.scheduler",
    "name": "daily_orders"
  },
  "inputs": [
    {
      "namespace": "postgresql://warehouse.example:5432",
      "name": "sales.orders"
    }
  ],
  "outputs": [
    {
      "namespace": "postgresql://warehouse.example:5432",
      "name": "analytics.daily_orders"
    }
  ],
  "producer": "https://example.com/openlineage/integration",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent"
}

Send the event with a Bearer token:

curl --fail-with-body --silent --show-error \
  -X POST '<OPENLINEAGE_BASE_URL>/v1/write/openlineage' \
  -H 'Authorization: Bearer <WORKSPACE_TOKEN>' \
  -H 'Content-Type: application/json' \
  --data-binary @event.json

Send a gzip-compressed event:

gzip -c event.json | curl --fail-with-body --silent --show-error \
  -X POST '<OPENLINEAGE_BASE_URL>/v1/write/openlineage' \
  -H 'Authorization: Bearer <WORKSPACE_TOKEN>' \
  -H 'Content-Type: application/json' \
  -H 'Content-Encoding: gzip' \
  --data-binary @-

After a successful request, send a COMPLETE event with the same runId and job identity to verify a complete run lifecycle.

Direct DataWay: Client Configuration¶

OpenLineage Python Client¶

The OpenLineage Python Client uses the api_key setting to generate a Bearer header. The following configuration also enables gzip and retries for transient errors:

export OPENLINEAGE__TRANSPORT__TYPE=http
export OPENLINEAGE__TRANSPORT__URL='<OPENLINEAGE_BASE_URL>'
export OPENLINEAGE__TRANSPORT__ENDPOINT='/v1/write/openlineage'
export OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key
export OPENLINEAGE__TRANSPORT__AUTH__APIKEY='<WORKSPACE_TOKEN>'
export OPENLINEAGE__TRANSPORT__COMPRESSION=gzip
export OPENLINEAGE__TRANSPORT__RETRY='{"total":5,"connect":5,"read":5,"backoff_factor":0.3,"status_forcelist":[429,500,502,503,504],"allowed_methods":["POST"]}'

Configuration support varies between client versions. Check the Python Client configuration reference before upgrading.

Apache Airflow¶

Install an OpenLineage Provider version compatible with your Airflow release, then configure the HTTP transport:

pip install apache-airflow-providers-openlineage

export AIRFLOW__OPENLINEAGE__NAMESPACE='production-airflow'
export AIRFLOW__OPENLINEAGE__TRANSPORT='{"type":"http","url":"<OPENLINEAGE_BASE_URL>","endpoint":"/v1/write/openlineage","auth":{"type":"api_key","apiKey":"<WORKSPACE_TOKEN>"},"compression":"gzip"}'

The Provider has specific Airflow version requirements. See the Airflow OpenLineage Provider documentation before installing or upgrading it.

Apache Spark¶

Load an OpenLineage Spark integration package compatible with your Spark release, then set these properties:

spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener
spark.openlineage.transport.type=http
spark.openlineage.transport.url=<OPENLINEAGE_BASE_URL>
spark.openlineage.transport.endpoint=/v1/write/openlineage
spark.openlineage.transport.auth.type=api_key
spark.openlineage.transport.auth.apiKey=<WORKSPACE_TOKEN>
spark.openlineage.transport.compression=gzip
spark.openlineage.namespace=production-spark

For package loading and version compatibility, see the OpenLineage Spark installation guide.

dbt¶

Install openlineage-dbt, reuse the Python Client transport settings, and run dbt-ol instead of dbt:

pip install openlineage-dbt

export OPENLINEAGE__TRANSPORT__TYPE=http
export OPENLINEAGE__TRANSPORT__URL='<OPENLINEAGE_BASE_URL>'
export OPENLINEAGE__TRANSPORT__ENDPOINT='/v1/write/openlineage'
export OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key
export OPENLINEAGE__TRANSPORT__AUTH__APIKEY='<WORKSPACE_TOKEN>'
export OPENLINEAGE__TRANSPORT__COMPRESSION=gzip
export OPENLINEAGE_NAMESPACE='production-dbt'

dbt-ol run

Do not set only OPENLINEAGE_URL: clients normally default to /api/v1/lineage, while this integration requires an explicit /v1/write/openlineage endpoint. See the OpenLineage dbt integration guide for execution modes and supported adapters.

Responses and Retries¶

Senders must tolerate duplicate delivery and keep a stable runId for the same run. Do not retry deterministic 4xx errors, because invalid events can otherwise remain in the sending queue indefinitely.

Verification and Troubleshooting¶

After configuration, verify the integration in this order:

1. Send a minimal START event with curl and confirm a 2xx response.
2. Send a terminal event with the same run and job identity.
3. In TrueWatch, search for the record by job namespace, job name, or runId.
4. Check producer logs for repeated retries, authentication errors, or serialization errors.

Common issues:

Security Recommendations¶

• Use HTTPS in production and keep certificate verification enabled.
• For direct DataWay ingestion, prefer Bearer or X-Token over URL tokens. Clients sending through DataKit need no token.
• Inject tokens through a Secret, secret manager, or runtime environment. Do not put them in images, configuration templates, shell history, or logs.
• Rotate tokens regularly and revoke an exposed token immediately.
• Facets can contain SQL, schemas, or business metadata. Review sensitive content and collect only what is necessary.

References¶

• OpenLineage core specification
• OpenLineage event examples
• OpenLineage Python Client configuration
• Apache Airflow OpenLineage Provider
• OpenLineage Spark integration
• OpenLineage dbt integration