Monitor Apache Airflow with OpenTelemetry

Monitor Apache Airflow data by configuring OpenTelemetry to send data to New Relic, where you can visualize tasks, operators, and DAG executions as metrics.

Prerequisites

Before enabling OpenTelemetry in Apache Airflow, you'll need to install the Airflow package with the otel extra. The installation method depends on your Airflow deployment approach:

Option 1: Installing from PyPi

1. Follow the installation instructions from Airflow's Documentation.

2. When installing with pip, add the otel extra to the command. For example:

   bash

   Copy

   $
   pip install "apache-airflow[otel]"

Option 2: Installing from Docker

1. Set up the Airflow Docker image using instructions from Airflow's documentation.

2. Extend the pre-built Docker image by using a Dockerfile to install the otel extra. You can replace the latest tag with your desired version of the image.

   FROM apache/airflow:latest
   RUN pip install --no-cache-dir "apache-airflow[otel]==$AIRFLOW_VERSION"

   Copy

Configuration

To send Airflow metrics to New Relic, configure the OpenTelemetry metrics to export data to an OpenTelemetry Collector, which will then forward the data to a New Relic OTLP endpoint using a license key.

Configure the OpenTelemetry Collector

1. Follow the basic Collector example to set up your OpenTelemetry Collector.
2. Configure the Collector with your appropriate OTLP endpoint, such as https://otlp.nr-data.net:4317.
3. For authentication, add your license key to the environment variable NEW_RELIC_LICENSE_KEY so that it populates the api-key header.
4. Ensure port 4318 on the Collector is reachable from the running Airflow instance. (For docker, you may need to use a docker network.)
5. Launch the Collector.

Configure Airflow metrics

Airflow sends metrics using OTLP over HTTP, which uses port 4318. Airflow has multiple methods of setting configuration options.

Choose one of the following methods to set the required options for Airflow.

1. Set the required options in the airflow.cfg file.

   [metrics]
   otel_on = True
   otel_host = localhost
   otel_port = 4318
   otel_ssl_active = False

   Copy

2. Or, set the required options as environment variables.

   bash

   Copy

   $
   export AIRFLOW__METRICS__OTEL_ON=True
   $
   export AIRFLOW__METRICS__OTEL_HOST=localhost
   $
   export AIRFLOW__METRICS__OTEL_PORT=4318
   $
   export AIRFLOW__METRICS__OTEL_SSL_ACTIVE=False

Validate data is sent to New Relic

To confirm New Relic is collecting your Airflow data, run a DAG or pipeline:

1. Login to Airflow.
2. Click the run button on one of the existing tutorial DAGs, or your own.
3. Wait for the pipeline to finish running.
4. Go to one.newrelic.com > All capabilities > APM & services > Services - OpenTelemetry > Airflow.
5. Click Metrics Explorer to visualize metrics for pipeline executions.

Building dashboards

With Airflow metrics, you can build dashboards around individual pipelines, overall performance, or view a comparison between different pipelines. Click here to learn more about querying your metrics.

This query retrieves a list of all reported metrics for Airflow:

SELECT uniques(metricName) FROM Metric WHERE entity.name = 'Airflow' 
AND metricName LIKE 'airflow.%' SINCE 30 MINUTES AGO LIMIT 100

Make sure to change the limit (100) if your metric names exceed it.

This query shows a comparison of different completion times for successful runs of different DAGs:

SELECT latest(airflow.dagrun.duration.success) FROM Metric 
FACET dag_id WHERE entity.name = 'Airflow' SINCE 30 minutes AGO TIMESERIES

This query shows counts of failed DAG runs, which can be used to build alerts for critical pipelines:

SELECT count(airflow.dagrun.duration.failed) FROM Metric 
FACET dag_id WHERE entity.name = 'Airflow' SINCE 30 minutes AGO TIMESERIES