Connect Apache Airflow

The Scheduler Knows Everything About Your Data. Nobody Governs the Scheduler.

Picture a failure any Airflow estate can produce. The revenue dashboard is wrong for nine days. Not down — wrong, which is worse, because down gets noticed. During a migration someone paused the DAG that rebuilds the orders file. The downstream DAG that feeds the dashboard keeps right on running, every run green, consuming a file that has quietly stopped changing. Airflow does exactly what it was told. That is the problem: nothing in the system is broken, and nothing in the system is accountable either.

Recipe: Cross-DAG lineage map

What Changes on an Ordinary Tuesday, Once Your Airflow Estate Is Governed

Governance tooling should be judged by what it changes about an ordinary working day — not the demo day, the Tuesday. So instead of listing features, here is a walk through the moments that go differently once your Airflow deployment has a governed model behind it: after you have extracted your DAG list, tasks, and data-aware scheduling assets from your own REST API, imported them into CoreModels, and wired the audit into CI.

Recipe: First governed Airflow import

Five Questions to Ask Before Any Tool Touches Your Airflow Metadata

Your orchestration metadata is a map of how your business actually runs — what moves, when, owned by whom, feeding what. Any tool that wants to import it should be interrogated first. Here are the five questions we think you should ask, and how our Apache Airflow connector answers them. We are publishing the answers because the posture *is* the product: a governance tool you cannot trust is worse than none.

Airflow Sits in the Middle of Your Stack. Govern It Without Wrapping It.

Draw a map of a typical modern data platform and Apache Airflow is rarely at the edge. It is the connective tissue: it triggers the ingestion jobs, kicks off the dbt build, loads the warehouse, refreshes the feature tables, and moves the files everything else depends on. An Apache Software Foundation project with pipelines defined as Python DAGs and a broad provider ecosystem, Airflow became, for many data teams, simply *the scheduler* — the thing all the other tools are arranged around.

Your AI Agent Shouldn't Have to Read Python to Understand Your Pipelines

Watch what a coding agent does when you ask it why a table is stale. It opens the DAG repository, reads some Python, greps for table names, follows a few imports, and produces a fluent explanation of what it *thinks* the pipelines do. The prose is confident. The structure underneath it is guessed — and orchestration is one of the worst places in your stack to guess.

Zero to First Audit: Bringing an Apache Airflow Deployment under CoreModels Governance

Your Airflow deployment can already describe itself. The stable REST API v1 will tell you every DAG, every task, and — if you use data-aware scheduling — every asset your pipelines produce and consume. What it will not tell you is whether any of that still matches what your organization *thinks* it runs: who owns each pipeline, which assets are consumed but produced by nothing, which consumers are quietly running on stale data because every producer is paused.

Recipe: Blank Apache Airflow project · First governed Airflow import

The Airflow Integration API, End to End: Every Route, Role, and Payload

Every vendor integration in CoreModels answers the same first question the same way. Ask the platform what it knows how to govern:

Governing Airflow with an AI Agent: The CoreModels MCP Tools in Practice

"Which of our pipelines have no accountable owner, and is anything downstream of a paused DAG?" That is a governance question, and until recently answering it meant a human clicking through the Airflow UI and cross-referencing a wiki. With CoreModels, an AI agent answers it directly: the same import/audit machinery we expose over HTTP is exposed over the Model Context Protocol, so an agent connected to your CoreModels server can inspect connector capabilities, run a full drift-and-hygiene audit of an Airflow deployment, and read back a human-quality report — all through typed tool calls.

Catch Pipeline Drift Before It Ships: A CI Gate and Drift Loop for Airflow

Orchestration drift is the quiet kind of failure. Nobody deletes a pipeline; a schedule is flipped to manual during an incident and never flipped back, and for three weeks a downstream table renders yesterday's world with perfect confidence. Nobody removes a dataset from governance; a DAG is renamed in a refactor and the governed model now describes a pipeline that no longer exists. None of this throws an exception — which is exactly why it belongs in CI, where a machine checks it on every change.

Recipe: CI Drift Gate for Airflow

Lineage First: How CoreModels Maps an Airflow Estate into the Governed Graph

Airflow is not a schema estate. A warehouse table has columns with types; a dbt model has a contract; an Avro subject has fields. A DAG has none of those — it has tasks, a schedule, an owner, and (with data-aware scheduling) declared relationships to the data it produces and consumes. So when we built the Apache Airflow connector for CoreModels, the design question was not "how do we pretend DAGs are tables?" but "what is orchestration's actual governance value?" The answer we committed to: the DAG-to-asset dependency graph, and pipeline hygiene. This connector is lineage-first, and every mapping decision below follows from that.

Recipe: Cross-DAG lineage map