Why we built an autonomous agent for the lakehouse

Every data platform team we know is running the same playbook in 2026. Connect a metrics tool. Build dashboards. Set alerts. When the alert fires, somebody jumps on Slack, opens a runbook, runs a few commands, and resolves it. By Friday, there are 14 unresolved tickets in the backlog and nobody can remember which cluster has the broken auto-termination.

The dashboards aren't wrong. The alerts aren't wrong. The tools aren't wrong. The posture is wrong.

Observability stops where it gets interesting

For a decade, observability has meant see the problem, route it to a human, hope the human has time. That posture made sense when fixes were nuanced, judgement-heavy, or risky. A senior engineer was the right interface to platform state.

Most lakehouse waste isn't nuanced. A cluster has been idle for 47 hours — terminate it. A warehouse has no auto-stop — set it to 10 minutes. A Delta table is fragmented into 12,000 small files — run OPTIMIZE. None of that needs senior engineering judgement. It needs a teammate who reads the dashboard at 3am, makes the call, and writes a note explaining what they did.

Agents finally make a different posture possible

Three things changed in the last 18 months:

1. LLMs got cheap and reliable enough to reason over telemetry, weigh trade-offs, and explain decisions in audit-grade English.
2. Tool-use protocols matured — the Anthropic Model Context Protocol means agent actions are first-class, structured, and auditable, not screen-scraped or shell-piped.
3. Closed-loop verification got cheap. With a programmatic apply path, you can also programmatically verify the outcome. The agent checks its own work.

Stack those three and you get something that doesn't exist in legacy observability tools: a teammate. It senses, it reasons, it acts within the guardrails you set, it verifies, and it tells you exactly what it did and why.

What we're not building

This isn't a "fully autonomous data platform" pitch. We don't believe humans should be removed from the loop — we believe humans should be removed from the boring parts of the loop. Three explicit design decisions:

• Per-strategy autonomy. Switch on auto-apply for cluster auto-termination. Keep small-files OPTIMIZE on recommend-only. The agent operates inside the budget you set, type-by-type. If you change your mind tomorrow, flip the toggle back to MANUAL.
• Blast-radius caps. Every action carries a dollar estimate. The agent will not exceed the cap you set on a strategy, full stop. No "I had a great reason" overrides.
• Auditable everything. The strategy version. The MCP request. The MCP response. The verifier outcome. The autonomy level at apply time. Every action carries enough provenance to debug or roll back six months later.

The dollar question

Investors and customers ask the same question with different words: does it actually save me money? The honest answer is that lakehouse waste is structurally underestimated. A typical mid-size Databricks deployment leaks $60K–$180K/year in idle DBUs, missing auto-stop, photon-eligible workloads on standard runtime, and OPTIMIZE-overdue tables. Half of that is fixable in your sleep. None of it gets fixed because nobody owns it.

Omnitrace owns it. The agent finds the leak, quantifies the dollar impact, and (if you let it) plugs it. Then verifies the plug worked.

Where we're going

Today the agent supports Databricks. Tomorrow it supports every compute platform that lakehouse teams care about — EMR, Starburst, Snowflake compute. Today it ships five autonomous strategies. Tomorrow it ships fifty. Today it senses what Databricks system tables expose. Tomorrow it senses live Spark JVM telemetry through a Prometheus integration we're building right now.

The bet is that observability that ends in a dashboard is yesterday's product. The next decade belongs to systems that close the loop.

If that resonates, come talk to us. We're picking design partners now.