What system tables don't tell you about Databricks cost

If you've ever opened system.billing.usage on a Databricks workspace and tried to reconcile it with your AWS bill, you know the gap. The numbers don't match. They never match. They're not even close.

This post is about why, and what to do about it. We built Omnitrace's cost intelligence layer specifically because every team we talked to was stitching this together by hand — usually badly, usually behind by a quarter.

What system tables actually report

Databricks system tables are excellent at one thing: DBU consumption per cluster, per warehouse, per job, per user, per day. That's it. Every row in system.billing.usage is a DBU charge tied to a cluster_id (or warehouse_id), with attribution metadata.

You can answer questions like:

• How many DBUs did this cluster consume yesterday?
• What's the most expensive job by DBU spend in the last week?
• Which user ran the most warehouse queries?

Useful, but not the whole picture.

What system tables don't report

Pretty much everything that costs you real cloud money outside the DBU multiplier:

• EC2 compute. The DBU price is a multiplier on Databricks platform value. The actual EC2 instances that run your clusters are billed separately by AWS — to your account, not Databricks's. System tables don't see this.
• EBS storage. Every cluster has root volumes plus task-temp volumes. They're billed per GB-hour by AWS. System tables don't see this either.
• NAT Gateway data transfer. If your clusters are in a private subnet (BYO-VPC), every byte they pull from S3 / external APIs goes through a NAT gateway. NAT data-transfer costs surprise everyone.
• S3 GET / PUT charges. Especially painful with small-files-heavy Delta tables — millions of GETs per query.
• Cross-AZ data transfer. If your driver and workers end up in different AZs, every shuffle byte costs money.
• API call costs. Cost Explorer itself charges per request, ironically.

Together these can be 30–60% of total Databricks-related cloud spend on top of DBUs. They're invisible to platform teams who only look at DBU billing.

Why "BYO-VPC" makes this worse

Most production Databricks deployments use BYO-VPC — Databricks provisions compute into your AWS account, in a VPC you own. That's the right architectural choice for security and compliance, but it has a side effect: your AWS bill now includes line items that Databricks doesn't see and Databricks's bill includes line items your AWS account doesn't see.

Reconciling them requires three things:

1. The Databricks side — system tables, REST API, cluster events. DBU consumption tied to compute IDs.
2. The AWS side — Cost Explorer line items grouped by tag, by service (EC2, EBS, NAT, S3, transfer), by usage type.
3. A join key — usually a tag (databricks-instance-group-id), but only if your platform team consistently applies it. Spoiler: they don't.

How Omnitrace does it (three tiers)

We took the position that no single attribution strategy works for every team, so we run three tiers and reconcile across them:

Tier 1: System tables

Standard DBU consumption, idle ratio, per-cluster spend. Cheap, fast, accurate for what it covers. This is the baseline.

Tier 2: AWS Cost Explorer (account-wide)

Strict-mode integration with AWS Cost Explorer API. We pull account-wide cost grouped by service, region, usage-type — including the API call charge metadata. This catches the EC2 / EBS / NAT / S3 / transfer line-items system tables miss.

We added a budget guard so the agent never blows a Cost Explorer API budget — the integration self-throttles when it sees daily-cap pressure.

Tier 3: BYO-VPC infrastructure discovery

The agent walks your AWS account and discovers EC2 instances, EBS volumes, and NAT gateways tagged as Databricks-related. It builds a per-resource cost model so you can attribute infrastructure cost back to the workspace, even when tags are inconsistent. This is the magic tier — it gives you per-workspace, per-team, per-project chargeback that no other tool delivers.

What this looks like in practice

A typical mid-size deployment we audit:

• Tier 1 reports: $288,000/year in DBU charges
• Tier 2 catches: $138,000/year in EC2 + EBS + NAT + S3 they didn't know was Databricks-attributable
• Tier 3 attribution: 62% of "untagged AWS spend" turns out to be Databricks workspaces in disguise

That last number is the killer. Most FinOps teams give up on Databricks attribution at 70-80% accuracy because the last mile is too painful. We close it.

Then the agent acts on the gap

Knowing where the cost is hiding is half the battle. The other half is doing something about it. The Omnitrace agent uses cost intelligence as one of its inputs — it weighs blast-radius decisions in actual dollars, not abstract severity scores. When it terminates an idle cluster, it knows the EC2 + EBS + DBU savings to the dollar and writes that into the audit log.

If you're stitching this together with manual queries against Cost Explorer and system tables every quarter, come talk to us. There's a better way.