Connect AWS Glue

The Catalog Knows Your Tables. It Doesn't Know What They Mean.

A crawler ran over the weekend. On Monday, a column in a table your finance dashboards depend on is no longer a `bigint` — a few malformed rows in a new S3 partition convinced the crawler it was looking at strings. Nothing failed. Nothing alerted. The Glue Data Catalog did exactly what it is designed to do: it recorded what the files look like now. The first person to notice is an analyst three days later, staring at a join that silently returns nothing.

Recipe: Lake Catalog Governance

After the Import: What a Governed Glue Catalog Changes Day to Day

The honest test of any governance tool is not the demo — it is what an ordinary Tuesday looks like six weeks after you adopted it. So instead of walking through features, let us walk through a week with a Glue Data Catalog that has a governed CoreModels project behind it, and contrast each moment with how the same moment used to go.

Recipe: CI Drift Gate for Glue · First Governed Import for Glue · Generate Athena DDL from the Governed Model · Lake Catalog Governance

Read-Only by Design: How We Earn Trust With Your Glue Catalog

"What does it touch?" is the first question a platform team asks about any tool that wants to look at their data lake, and it is the right question. Our answer for the AWS Glue integration is unusually short: nothing in your AWS account, ever. CoreModels never holds AWS credentials and never opens a connection to AWS. You run `aws glue get-tables` yourself, with your own credentials, inside your own perimeter, and upload the resulting JSON. A live-sync mode is a capability we have deliberately deferred rather than quietly shipped, precisely because "we'll be careful with your keys" is a weaker promise than "we never have them."

Glue Is the Metastore. It Shouldn't Have to Be the Meaning.

Few pieces of infrastructure are load-bearing for as many systems as the AWS Glue Data Catalog. It began as a managed, Hive-compatible metastore and became the connective tissue of the AWS analytics stack: crawlers and Glue ETL jobs populate it, Athena resolves every query through it, EMR and Redshift Spectrum read it, Lake Formation layers permissions over it. If your data lives in S3 and you query it with SQL, the catalog is almost certainly in the path.

Agents Shouldn't Guess at the Data Lake

Give an AI agent access to Athena and ask it a business question — "how many participants enrolled last quarter" — and you can watch it improvise. It lists tables and picks the one whose name sounds right. It infers that `status = 'active'` probably means enrolled, because what else would it mean? It writes a query that scans every partition because nothing told it the table was partitioned by month. The SQL is fluent, the answer arrives with confidence, and every load-bearing assumption in it was a guess.

Ten Minutes to a Governed Glue Catalog

Open a terminal. Everything in this tutorial is one AWS CLI command and two HTTP calls, and at the end of it your AWS Glue Data Catalog has a governed twin in CoreModels plus a first drift audit you can read line by line. Nothing gets installed in your AWS account, no IAM role is granted to us, and no credential of yours travels anywhere: you run the CLI, you upload the JSON it printed.

Recipe: Blank Glue project · First Governed Import for Glue · Generate Athena DDL from the Governed Model

The AWS Glue Integration API, Route by Route

CoreModels models vendor governance as a small, fixed set of verbs and applies them uniformly to every connector. For AWS Glue Data Catalog — vendor key `glue` — that is eight core routes on the interactive surface and two on the machine-to-machine surface. (A newer sync-plan surface — `sync/propose`, plan fetch, and the sync ledger — sits beside these and is not covered here.) This is the reference: the exact paths, the role each one enforces, the request bodies as the API defines them, and the responses you actually get back.

Handing the Glue Catalog to an Agent: CoreModels over MCP

An agent that can answer *"did our lake drift?"* needs three things: a way to reach the governed model, a way to hand it a fresh catalog export, and a guarantee that asking the question cannot change the answer. The CoreModels MCP server provides all three. The same vendor-integration engine behind the HTTP routes is exposed as MCP tools, with the read verbs at Viewer role and the write verbs behind an admin endpoint and real project membership.

Automating Glue Drift: The Gate, the Trail, and the Heartbeat

Nothing in your repository changes when a crawler retypes a column. That is the awkward fact about lake governance: the estate you need to watch does not live in git, so the usual "run it on pull requests" reflex leaves the interesting drift entirely unobserved. A Glue drift gate has to be driven by a clock, not by a commit — and once it is, three more mechanisms fall out of it almost for free: a rolling trail, a status badge, and a server-side heartbeat that watches the *other* direction of drift.

Recipe: CI Drift Gate for Glue

What Happens to a Glue Table on Its Way into the Graph

Take one entry out of an `aws glue get-tables` response — a table called `events` in database `lake`, a few columns, one partition key, a `classification` parameter, an S3 location — and follow it into CoreModels. By the end it is a governed Type with Elements, a vendor identity, a metadata mixin, an entry in an estate snapshot, and a set of audit rules watching it. This is that journey, plus the honest edges: what is approximated, what is deliberately not invented, and which behaviors will surprise you.

Recipe: Lake Catalog Governance