An agent loop in production: lessons from the first hundred actions

We've now run our autonomous agent against real Databricks workspaces for long enough to have opinions. This is an engineering post about what we got right, what we got wrong, and what we'd tell anyone else building an agent that takes destructive action on production systems.

The first lesson: tiers, not toggles

Our v0 had a single global "auto-apply" toggle. It was the obvious thing to ship. It was also wrong. Customers who would happily auto-apply autotermination_minutes=30 on idle clusters would absolutely not auto-apply OPTIMIZE on production tables (warehouse compute, possibly disrupts queries). Bundling them killed onboarding.

v1 introduced per-strategy autonomy levels: MANUAL · AUTO_LOW_RISK · AUTO_FULL, set independently per recommendation type. This let teams turn on the safe stuff first and earn confidence. Adoption picked up immediately.

The lesson: autonomy is not a binary, it's a coordinate system. Risk varies by strategy. Trust earned on one strategy doesn't transfer to another. Build for that.

The second lesson: verify everything

Our v0 verifier ran on a cron, every 15 minutes, by reading the next regular cluster snapshot. That meant the latency from "agent applied" to "agent verified" was up to 15 minutes. During that window, the UI showed "APPLIED" which felt anticlimactic — and worse, sometimes the change didn't actually stick (Databricks API ack vs. settle is not the same thing).

v1 introduced two-tier verification:

• Tier 1 (polling) — runs every minute, reads the next available snapshot. Cheap, eventually consistent.
• Tier 2 (targeted) — fires once per APPLIED action, ~3 minutes after apply, makes a targeted MCP read of the specific resource and persists the result with a is_targeted_check=true flag.

Now apply→verify is consistently 3-5 minutes for most strategies, and the agent has a reliable signal even on terminated/idle clusters where the regular collector hasn't run lately.

The lesson: "the API returned 200" is not the same as "the change happened." Always read it back.

The third lesson: dollar-quantified, every time

We initially shipped recommendations with a severity tag (CRITICAL/HIGH/MEDIUM/LOW). Severity is what every observability tool ships. Customers ignored it. Severity has no signal — every team's CRITICAL is somebody else's whatever.

v1 added estimated_savings_usd on every recommendation, expressed annually. Suddenly the conversation changed. "Should I let the agent auto-fix this?" became "$4,000/year vs. risk of disrupting one query — yes, fine." Decisions shrank from days to minutes.

The lesson: dollars are the only severity score that scales. If you can't quantify in dollars, you don't have a recommendation, you have a complaint.

The fourth lesson: audit, audit, audit

Six months in, a customer asked us "wait, when you applied this fix three weeks ago, was the autonomy level AUTO_LOW_RISK or AUTO_FULL?" We had to tell them we didn't know because we'd designed the autonomy table as a current-state record (mutable). They could've changed it Tuesday and the action was Friday.

That mistake cost us trust. We now persist autonomy_level_at_apply on every databricks_agent_actions row — a point-in-time snapshot, immutable. Same for action_strategy_version, mcp_request, mcp_response, verifier_outcome, autonomy_budget_at_apply. Six months from now you can ask "why did the agent do this?" and get the full provenance.

The lesson: autonomous systems need forensic-grade audit trails by default, not as an enterprise add-on. You can't reconstruct it after the fact.

The fifth lesson: regression is more important than success

v0 had two terminal states: VERIFIED and VERIFY_FAILED. Real-world has a third: REGRESSED — the change was successfully applied, but later somebody undid it. A user manually set autotermination_minutes back to 0. A redeploy clobbered the policy. A different tool changed the warehouse spec.

The agent has to tell you that. We added the REGRESSED transition that fires anytime a previously VERIFIED outcome reverts. It triggers a different email template ("you might want to know") and shows up red on the Detector Performance dashboard.

The lesson: autonomy lives in a world it doesn't fully control. Plan for the change to come back. Detect it and notify the human.

What's next

We're a month into shipping the loop and there's a long list of things we still want to learn. Some near-term experiments:

• Letting the agent propose blast-radius cap increases when its track record on a strategy crosses a threshold (current "trust ramp" is manual).
• Adding richer telemetry sources (live Spark JVM via Prometheus integration, shipping Q3) so the agent can act on signals that don't show up in system tables yet.
• Multi-step strategies — investigate → propose → apply → verify, where the investigation step itself uses MCP tools to gather evidence before deciding what to do.

If you're building something similar and any of this rhymes with your own experience, we'd love to compare notes. hello@omnitrace.io.