Databricks cost optimization means cutting your Databricks Unit (DBU) consumption without cutting query throughput, and in 2026 most teams can pull 40-60% out of their bill with four moves: shift interactive work off all-purpose clusters onto job clusters or serverless SQL warehouses, enable Photon only where the 2x DBU multiplier actually pays back, enforce cheap defaults through cluster policies, and put Delta Lake housekeeping on a schedule so OPTIMIZE and VACUUM stop billing you for cold bytes. The techniques below are the ones I keep pulling out of my toolkit when I walk into a workspace that has grown past $200K/month.
All-purpose clusters cost 2.5-4x more per DBU than job clusters for the same workload. Moving scheduled jobs off shared interactive clusters is usually the single biggest win.
Serverless SQL warehouses bill per second with sub-10s starts, so they beat classic warehouses whenever your BI traffic is bursty or off-hours. For steady 24/7 dashboards, classic + Photon is still cheaper.
Photon multiplies DBUs by roughly 2x but delivers 3-8x speedups on Parquet/Delta scans. Enable it for ETL and SQL; skip it for lightweight streaming and Python UDF-heavy workloads.
Cluster policies let admins pin cheap defaults (max workers, spot bid, auto-terminate) so users cannot spin up $50/hour clusters by accident. This is the single highest-leverage governance control.
Delta Lake OPTIMIZE, Z-ORDER, and VACUUM cut both storage and compute cost by shrinking file counts and pruning stale data. Scheduling matters here, since OPTIMIZE itself burns DBUs.
Unity Catalog system tables (system.billing.usage) give you cost attribution by workspace, cluster, user, and tag. Once you learn the schema it beats any third-party dashboard.
How Databricks pricing actually works: DBUs, SKUs, and the compute markup
Before you can cut a Databricks bill you have to read it. Every Databricks invoice has two components stapled together: the Databricks Unit (DBU) charge that Databricks itself collects, and the underlying cloud compute (EC2, Azure VMs, GCE) that your cloud provider bills separately. The DBU rate depends on which SKU you picked (Jobs Compute, All-Purpose Compute, SQL Compute, or the Serverless variants), and each SKU has its own per-DBU price that also varies by tier (Standard, Premium, Enterprise). As of 2026, All-Purpose Compute on the Premium tier runs about $0.55 per DBU on AWS, while Jobs Compute on the same tier runs about $0.15 per DBU. That 3.7x SKU gap is the single most expensive thing most teams get wrong.
A DBU is a normalized unit of processing capability per hour, and Databricks publishes a DBU rating for every instance type. An i3.xlarge on AWS runs at 1 DBU/hour, an r5d.4xlarge at 4 DBU/hour, and a memory-optimized r5d.24xlarge at 24 DBU/hour. Multiply that by wall-clock hours and by the SKU price, and you get the Databricks-side charge. Then add the underlying EC2 On-Demand price (or Reservation-discounted price) to get the full hourly cost. The official Databricks pricing page is the source of truth here, and honestly, it's worth reading in full at least once a year because the SKU catalog keeps expanding.
The compound effect matters. A cluster running an r5d.4xlarge worker at $1.20/hour of EC2 plus 4 DBU/hour of All-Purpose Premium is really costing you $1.20 + (4 * $0.55) = $3.40/hour per worker. Ten workers, twelve hours a day, twenty-two workdays: $8,976/month for one team's interactive cluster. Move that same workload to Jobs Compute and the DBU line drops to $1.20 + (4 * $0.15) = $1.80/hour, or $4,752/month, which is a 47% cut before you touch anything else.
Job clusters vs all-purpose clusters: the biggest single lever
All-purpose clusters exist for one reason: humans need a place to run notebooks interactively with autocomplete, dashboards, and REPL state. Everything else (scheduled ETL, batch scoring, ML training runs, dbt jobs) belongs on a job cluster. Job clusters spin up on demand, run the workload, and terminate. They don't share compute across notebooks, they can't be attached to interactively, and they're billed at roughly 27% of the All-Purpose DBU rate on Premium tier. The math is not subtle.
So why do teams end up with expensive shared clusters? Inertia, mostly. Someone spun one up eighteen months ago, everyone attached their notebooks, and now nobody wants to be the person who breaks the workspace. The fix is a migration playbook, not an argument. First, tag every all-purpose cluster and pull a 30-day DBU report grouped by job_run_id versus interactive_notebook attach. Anything running unattended goes into a Workflows job with its own job cluster. Second, for the scheduled notebooks that are truly interactive-only, convert them into Databricks Workflows tasks with the target cluster set to a fresh Jobs Compute cluster (the notebook code itself doesn't change).
Here's a Databricks REST API snippet that lists your top interactive clusters by uptime, which is usually the fastest way to find the biggest offenders:
# List all-purpose clusters and rank by wall-clock uptime last 30 days.
# Requires a workspace PAT with Can View on clusters.
import os, requests, datetime as dt
HOST = os.environ["DATABRICKS_HOST"] # https://acme.cloud.databricks.com
TOKEN = os.environ["DATABRICKS_TOKEN"]
headers = {"Authorization": f"Bearer {TOKEN}"}
resp = requests.get(f"{HOST}/api/2.1/clusters/list", headers=headers).json()
clusters = [c for c in resp.get("clusters", []) if c.get("cluster_source") == "UI"]
now = dt.datetime.utcnow().timestamp() * 1000
for c in sorted(clusters, key=lambda c: c.get("start_time", 0)):
uptime_h = (now - c.get("start_time", now)) / 3_600_000
print(f"{c['cluster_name']:40s} {uptime_h:7.1f}h {c.get('num_workers', 0)} workers")
Cluster names that show 200+ hours of uptime and more than two workers are the ones sitting idle overnight and on weekends. Auto-termination on those clusters is almost always disabled or set to 24 hours. Cut the auto-terminate down to 30-60 minutes and you'll reclaim more DBUs in a week than any query tuning will find in a quarter.
Serverless SQL warehouses: when they're cheaper and when they're not
Databricks SQL warehouses come in three flavors as of 2026: Classic, Pro, and Serverless. Serverless is the one people ask about, because the pitch is compelling: sub-10-second query starts, no VM management, per-second billing, and no cold-cluster wait. But Serverless SQL DBUs are priced at roughly $0.70/DBU on Premium, versus $0.22/DBU for Classic SQL. That's roughly 3.2x per DBU, offset by the fact that you're only billed while queries are actively running.
The breakeven is simple. If your warehouse is actively serving queries less than about 30% of the wall-clock hours it's otherwise up, Serverless wins. Bursty BI dashboards, ad-hoc analyst queries, off-hours scheduled reporting: all better on Serverless. Steady 24/7 dashboards with high concurrent user counts (>10 QPS sustained) are usually cheaper on Classic + Photon because you amortize the always-on cost over enough queries to bring effective DBU/query down. Do the math on your own workload with the query history API before defaulting to either.
The knob that gets missed most often is warehouse sizing. Databricks SQL warehouses scale from 2X-Small to 4X-Large, and each step up roughly doubles DBU consumption. Most teams over-size warehouses because someone benchmarked a slow query once and jumped from Small to Large without checking that the query was slow because of a missing Z-ORDER, not insufficient compute. Run each warehouse at the smallest size that hits your P95 latency target, and let autoscaling add clusters horizontally for concurrency spikes instead of jumping the base size.
Photon acceleration: is the 2x DBU multiplier worth it?
Photon is Databricks' vectorized C++ query engine, and enabling it roughly doubles the DBU rate for any cluster that uses it. In exchange, it delivers 3-8x speedups on typical SQL and Delta ETL workloads. The math should be obvious (4x faster at 2x cost is net 50% cheaper), but Photon isn't a free win everywhere. The Photon compatibility documentation is the authoritative list of what accelerates and what doesn't.
Photon accelerates: SQL aggregations, joins, window functions, string operations, Delta MERGE and DELETE, and Parquet scans. Photon does not accelerate: Python UDFs (they still fall back to the JVM), Pandas UDFs of moderate size, RDD operations, arbitrary Scala UDFs, and structured streaming with complex stateful operators. A cluster that spends 70% of its time in a Python UDF is going to get almost none of Photon's speedup, but you'll still pay the 2x DBU multiplier. Cost goes up, not down.
The decision rule I use with clients: profile the top 10 jobs by DBU consumption, and check the Spark UI for time spent in "PhotonScan" and "PhotonHashAggregate" versus "PythonUDFRunner" or "MapPartitionsRDD". If more than 60% of task time is Photon-eligible, turn Photon on for that job cluster. Otherwise, leave Photon off and refactor the UDF into a Photon-compatible expression instead. Rewriting one Python UDF into a native Spark function is often worth a full engineer-day, because it unlocks Photon for that job forever.
Photon on Serverless SQL warehouses is on by default and you can't disable it. That's fine. Serverless SQL workloads are almost always Photon-friendly since they're pure SQL. It's on your interactive and job clusters where the choice matters.
Cluster policies and instance pools: enforce the cheap defaults
Cluster policies are the highest-leverage governance control in Databricks. A policy is a JSON document that constrains what users can and can't pick when creating a cluster: instance types, max worker count, DBU tier, autoscaling range, auto-termination window, and dozens of other fields. When a policy is applied, the UI grays out disallowed options and the API rejects them. This is the tool that stops your data scientists from spinning up a 20-worker r5d.24xlarge cluster to load a 200MB CSV. (Yes, I've seen it happen. More than once.)
A minimal cost-safe policy for a "standard analyst" persona looks like this:
Notice what this policy does. It caps workers at six, forces auto-termination between 15 and 60 minutes, requires spot with on-demand fallback, and demands a valid cost_center tag on every cluster. Users can't escape any of these constraints, and the tag requirement is what makes downstream chargeback actually work. The official cluster policies documentation lists every attribute you can constrain.
Pair policies with instance pools. A pool pre-warms a set of idle VMs so cluster starts drop from 4-6 minutes to about 30 seconds, and Databricks doesn't charge DBUs on idle pool instances (only the underlying EC2/Azure/GCE cost). For teams with lots of small job clusters, pools save real DBU by eliminating the wasted first minutes of every job while workers boot.
Spot fleets, autoscaling, and auto-termination
Every worker on a Databricks cluster can be a spot instance, which cuts the EC2/Azure/GCE side of the bill by 60-90%. The DBU side is unchanged. Spot workers on Databricks are safer than they sound because the driver stays on-demand. A lost spot worker just means a task retry, not a lost cluster. Configure spot with fallback to on-demand so the cluster doesn't stall waiting for spot capacity.
On AWS, set aws_attributes.availability = SPOT_WITH_FALLBACK and first_on_demand = 1 (which keeps the driver on-demand and puts every worker on spot). On Azure, use azure_attributes.availability = SPOT_WITH_FALLBACK_AZURE and set spot_bid_max_price to -1 to bid the on-demand price. For a broader view of when spot works and when it doesn't across all three clouds, see our spot instances guide.
Autoscaling is table stakes. Enable it on every cluster with a reasonable min-to-max range (I use 1-8 for interactive, 2-16 for job clusters). Databricks scales aggressively down when tasks finish, which is exactly what you want for cost. What you don't want is a fixed-size cluster provisioned for peak load. That's throwing away 40-70% of the compute you pay for.
Auto-termination is the setting that saves the most money and gets configured the worst. The default in the UI is 120 minutes, which for interactive clusters means every notebook session leaves an idle cluster running for two hours after the analyst closes their laptop. Set it to 20-30 minutes for interactive clusters. Job clusters auto-terminate as soon as the job finishes, so this setting doesn't apply to them.
Delta Lake OPTIMIZE, Z-ORDER, and VACUUM economics
Delta Lake tables accumulate small files, dead versions, and stale statistics. Left alone, this bloat means every query reads more bytes than it should, every MERGE rewrites more partitions than necessary, and every write burns more DBUs than the workload actually requires. The three commands that fix this (OPTIMIZE, Z-ORDER, and VACUUM) aren't free, but they pay for themselves many times over on any table that gets read frequently.
OPTIMIZE compacts small files into larger ones (target size ~1GB). For a table with 50,000 files averaging 4MB each, OPTIMIZE will collapse those into ~200 files of 1GB, and downstream queries that scan the table will see 250x fewer file open operations. Schedule OPTIMIZE weekly on high-write tables, monthly on stable ones. Z-ORDER (either the classic multi-column clustering or the newer Liquid Clustering feature) sorts data by frequently-filtered columns so that predicate pushdown skips more files. Applied to tables where 80% of queries filter by the same 1-3 columns, Z-ORDER can cut scan bytes by 5-20x.
VACUUM deletes old file versions after the retention window expires (default 7 days). For tables that get frequent MERGE or UPDATE operations, VACUUM reclaims storage that would otherwise pile up indefinitely. Reducing storage indirectly cuts cloud storage costs, and for tables that hit multi-terabyte scale, this isn't a rounding error.
-- Weekly maintenance for a high-write fact table.
-- Runs on a small job cluster (2x m5d.xlarge, Photon on, auto-terminate 15m).
OPTIMIZE analytics.fact_orders
WHERE order_date >= current_date() - INTERVAL 30 DAYS
ZORDER BY (customer_id, product_id);
-- Reduce retention to 3 days if you don't need time travel further back.
-- SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM analytics.fact_orders RETAIN 72 HOURS;
-- Refresh table statistics so the CBO plans efficient joins.
ANALYZE TABLE analytics.fact_orders COMPUTE STATISTICS FOR ALL COLUMNS;
One trap worth flagging: OPTIMIZE itself is expensive on a large table. A 10TB Delta table can burn thousands of DBUs on a full OPTIMIZE run. Scope it with a WHERE predicate on the partition column so you only compact recent partitions where the small-file problem actually lives. Older partitions that haven't been written to since the last OPTIMIZE don't need to be touched. (I learned this after wasting about $900 on an "innocent" nightly maintenance job that was re-compacting five years of history.)
Tracking DBU spend with Unity Catalog system tables
You can't optimize what you can't measure. Unity Catalog's system tables, specifically system.billing.usage and system.compute.clusters, give you a queryable, joinable record of every DBU consumed in the workspace, tagged with account_id, workspace_id, sku_name, usage_date, usage_unit, and any custom_tags you attached at cluster creation. The Databricks system tables billing schema is worth memorizing.
Here's a query I run weekly on every workspace to find the top DBU consumers by tagged cost center:
-- Top cost centers by DBU spend, last 30 days.
-- Assumes clusters are tagged with custom_tags.cost_center.
SELECT
usage_metadata.cluster_id AS cluster_id,
custom_tags.cost_center AS cost_center,
sku_name AS sku,
SUM(usage_quantity) AS dbu_hours,
ROUND(SUM(usage_quantity * list_price.pricing.default), 2) AS list_price_usd
FROM system.billing.usage AS u
LEFT JOIN system.billing.list_prices AS list_price
ON u.sku_name = list_price.sku_name
AND u.usage_start_time BETWEEN list_price.price_start_time
AND COALESCE(list_price.price_end_time, current_timestamp())
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY 1, 2, 3
ORDER BY list_price_usd DESC
LIMIT 50;
Pipe the result into a scheduled Databricks SQL alert that emails Slack when any single cluster exceeds a weekly DBU threshold, and you have anomaly detection for free. Combine this with automated tagging enforcement from cluster policies and you get real chargeback, not the estimated kind, the kind where the analytics team's monthly invoice is a query result they can drill into themselves.
For teams running Databricks alongside other data warehouses, unify these tags with your broader cost allocation strategy. Our guide on cloud cost tagging with Terraform covers the discipline of getting cost_center, environment, and owner tags applied consistently across every resource, which is the prerequisite for any of this reporting to hold up.
Frequently Asked Questions
What is a DBU in Databricks and how much does it cost?
A Databricks Unit (DBU) is a normalized unit of processing capability per hour. As of 2026, DBUs cost roughly $0.15 (Jobs Compute Premium) to $0.70 (Serverless SQL Premium) on AWS, and each cluster's DBU rate is set by its instance type multiplied by its SKU price. Underlying EC2/Azure/GCE compute is billed separately by your cloud provider.
Is Databricks Serverless cheaper than Classic clusters?
It depends on utilization. Serverless SQL DBUs cost about 3.2x more per DBU than Classic, but Serverless only bills while queries are actively running. If your warehouse serves queries less than about 30% of its otherwise-uptime, Serverless wins. For steady 24/7 dashboards, Classic + Photon is usually cheaper.
How much does Photon really save?
Photon roughly doubles the DBU rate but delivers 3-8x speedups on Photon-eligible workloads (SQL, Delta MERGE, Parquet scans). Net effect is typically 40-70% cheaper for SQL and ETL, but zero or negative for workloads dominated by Python UDFs or complex streaming logic.
Can I use spot instances safely on Databricks?
Yes. Keep the driver on-demand and put every worker on spot with fallback. A lost spot worker triggers Spark task retries, not cluster failure. Expect 60-90% savings on the compute side of the bill. Use mixed instance families in an instance pool to improve spot availability.
What is the difference between job clusters and all-purpose clusters in Databricks?
Job clusters are ephemeral: they start for a specific job, run it, and terminate. All-purpose clusters are shared, interactive, and long-running. Job clusters cost about 27% of the DBU rate of all-purpose clusters for the same SKU tier, so scheduled workloads should always run on job clusters.
How do I track Databricks costs by team or project?
Enforce cluster tags (cost_center, project, environment) via cluster policies so users can't create untagged clusters, then query system.billing.usage in Unity Catalog to aggregate DBU spend by tag. Combine that with system.billing.list_prices to get dollar figures per team.
The five levers that trim 40-70% off a DynamoDB bill: capacity modes, reserved capacity, Standard-IA table class, GSI audits, and TTL cleanup. With CLI and Terraform examples.
Amazon Redshift cost optimization in 2026 comes down to five levers: right compute model (RA3 vs Serverless), correct sizing, Reserved Instances timed to stable usage, capped concurrency scaling, and WLM plus materialized view tuning.
Which managed PostgreSQL is actually cheapest in 2026? Side-by-side pricing for RDS, Aurora, Cloud SQL, and Azure Flexible Server across four real workloads, with commitment discounts and workload-by-workload picks.
