Performance Tuning & FinOps

The methodology: measure before you touch anything

Every credible tuning answer starts the same way: you measure first. Random knob-turning is the number-one anti-pattern interviewers probe for. Your working order:

1. Measure. For notebooks and jobs, open the Spark UI: find the longest stage, then check shuffle read/write volumes, spill (memory and disk), and task-duration skew (compare median vs. max task time in the stage summary). For Databricks SQL, use the Query Profile - it shows time per operator, rows produced, files pruned vs. read, and whether the query hit a cache.
2. Fix data layout. File sizes, clustering, pruning, statistics. Layout fixes help every downstream query, not just the one you're staring at.
3. Fix code. Join strategy, eliminating UDFs, removing unnecessary wide transformations, pushing filters early.
4. Fix compute last. Resizing a cluster to outrun a layout problem is renting your way out of an engineering problem - it works until the data doubles.

Compute: sizing, autoscaling, spot, Photon, serverless

Cluster sizing and instance families

Match the family to the bottleneck: memory-optimized for wide aggregations, large joins, and anything that spills; compute-optimized for CPU-bound transforms and many small tasks; storage-optimized (instances with local NVMe) when you lean on the disk cache for repeated reads; general-purpose when you don't yet know. Two sizing heuristics worth quoting: prefer fewer, larger nodes for shuffle-heavy work (less network traffic between executors), and size total cluster memory so your largest shuffle stage doesn't spill - spill to disk is often the single biggest silent slowdown.

Autoscaling - and when it hurts

Autoscaling fits ragged, interactive, or multi-job workloads where demand varies. It actively hurts when: (1) the workload is a steady batch job - you pay scale-up latency every run for no benefit, so a fixed right-sized cluster is cheaper and more predictable; (2) the job is shuffle-heavy - removing a node can lose shuffle files and trigger recomputation; (3) it's a Structured Streaming job, where classic autoscaling reacts poorly to micro-batch load. For SQL warehouses, autoscaling adds whole clusters for concurrency rather than resizing one - that one is usually safe to leave on.

Spot instances

Spot/preemptible workers can cut compute cost 60-90 percent. Use them for retry-tolerant batch work: keep the driver (and ideally a small core of workers) on-demand, put the rest on spot with fallback to on-demand. Avoid spot for tight-SLA jobs and long-running streaming, where eviction mid-shuffle costs more in recomputation than it saves.

Photon: what it accelerates and when it pays

Photon is the C++ vectorized engine that replaces the JVM execution path for SQL and DataFrame operations: scans, filters, joins, aggregations, Delta writes and MERGE. It does not accelerate RDD code, Python UDFs, or most Scala UDFs - those fall back to the JVM row-at-a-time path. Photon carries a higher DBU rate, so the test is arithmetic: if it cuts runtime by more than the DBU multiplier increases cost (it often does on scan/join/aggregate-heavy SQL pipelines - 2-3x speedups are common), it is cheaper per job, not just faster. UDF-heavy or RDD-heavy code sees little speedup and pays the full premium. Check the Spark UI or Query Profile for how much of the plan actually ran "photonized" before deciding.

Serverless compute trade-offs

Serverless compute for notebooks, jobs, and pipelines is GA and is the strategic default Databricks is steering toward (serverless jobs and pipelines now default to a "Performance optimized" mode in the UI, with a cheaper standard mode available). What you gain: near-instant startup, no idle clusters burning DBUs, no sizing decisions, Databricks-managed Photon and scaling. What you give up: node-level control (no instance-family choice, no spot strategy, limited Spark conf), and a different cost shape - you pay a premium per DBU in exchange for paying only for execution. Rule of thumb as of mid-2026: serverless wins for spiky, short, or interactive workloads and for teams without tuning bandwidth; a well-tuned classic cluster on spot can still win for long, steady, predictable batch. Run the comparison with your own system-table billing data rather than asserting either way.

Data layout: the small-file problem and file-size management

Thousands of small files mean slow listings, bloated transaction-log state, and tasks that spend more time on overhead than on data. Classic causes: streaming or frequent small batch appends, over-partitioning, and high-cardinality partition keys. Target file sizes in the hundreds of MB to ~1 GB. The toolkit:

• OPTIMIZE - compacts small files into larger ones; the scheduled, explicit fix.
• Optimized writes and auto compaction - write-time features that produce fewer, larger files as data lands, reducing how often you need OPTIMIZE at all.
• Predictive optimization - Databricks runs OPTIMIZE, VACUUM, and ANALYZE automatically on Unity Catalog managed tables based on observed access patterns. It's enabled by default for accounts created on or after November 11, 2024 and has been rolling out broadly - so for UC managed tables, the modern answer is increasingly "let the platform handle maintenance, and verify via system tables that it's happening."

OPTIMIZE finance.gold.gl_actuals;          -- compact small files

-- New tables (current guidance): automatic liquid clustering
CREATE TABLE finance.gold.gl_actuals (...)
CLUSTER BY AUTO;

-- Existing table: adopt liquid clustering (drops partition/Z-ORDER strategy)
ALTER TABLE finance.silver.journal_lines CLUSTER BY (company_code, fiscal_period);

Partitioning vs. liquid clustering: the decision framework

As of mid-2026 the official recommendation is liquid clustering for all new Delta tables, preferably automatic via CLUSTER BY AUTO with key selection powered by predictive optimization. Hive-style partitioning plus Z-ORDER is legacy guidance - and the two are mutually exclusive: a table is either partitioned/Z-ORDERed or liquid-clustered, never both.

Caching: three layers, three different things

Joins and skew: the recap

The deep treatment lives in Module 01 - Apache Spark Internals & Troubleshooting; for this module you need the decision summary. Broadcast the small side of a join to eliminate the shuffle entirely (check the threshold; hint explicitly when statistics mislead the planner). Let AQE do its job - it coalesces post-shuffle partitions, converts sort-merge joins to broadcast at runtime, and splits skewed partitions via skewed-join handling. When AQE isn't enough - one customer or one ledger account dominating a key - salt the hot key manually. Diagnose skew in the Spark UI as a handful of straggler tasks running far longer than the stage median.

Writing patterns that keep Photon and the optimizer in play

• Never collect() large results to the driver - it's the canonical driver-OOM cause. Write results to a table; use display()/limit() to inspect.
• Prefer built-in functions over UDFs. Built-ins are visible to Catalyst and run in Photon; a Python UDF is an opaque black box that forces row-by-row serialization to a Python worker and disables Photon for that operator.
• If you need custom Python, use pandas UDFs - Arrow-based, vectorized batches instead of row-at-a-time, typically an order of magnitude faster than plain Python UDFs. Still slower than built-ins; treat as the fallback, not the default.

# Bad: opaque UDF - no Catalyst, no Photon
@udf("double")
def margin(rev, cost):
    return (rev - cost) / rev if rev else None

# Good: built-in expressions - optimizable, Photon-eligible
df = df.withColumn(
    "margin",
    F.when(F.col("rev") != 0, (F.col("rev") - F.col("cost")) / F.col("rev"))
)

Symptom → likely cause → fix

Cost observability: you can't optimize what you can't attribute

On the Data Intelligence Platform, cost data is just more Delta tables - which means you can do FinOps with the same SQL skills you use for finance data:

• System tables for billing: system.billing.usage records DBU consumption per workspace, SKU, and workload, with identity and tag metadata; join to system.billing.list_prices to express it in dollars. Related system schemas cover compute (node types, cluster timelines) and query history for DBSQL.
• Cluster policies: the enforcement layer - constrain instance families, max workers, auto-termination, spot usage, and require tags at creation time. A policy is an opinion about cost that users can't forget to follow.
• Tags: custom tags on clusters, jobs, and SQL warehouses propagate into billing data (and to cloud-provider billing for classic compute). No tags, no attribution, no accountability.
• Budgets and alerts: account-level budgets watch spend against tag- or workspace-scoped filters and notify on threshold breaches - your early-warning system, alongside alerts on the usage tables themselves.

-- Last 30 days of cost by team tag and SKU
SELECT u.custom_tags['team']                       AS team,
       u.sku_name,
       SUM(u.usage_quantity * p.pricing.default)   AS est_dollars
FROM system.billing.usage u
JOIN system.billing.list_prices p
  ON u.sku_name = p.sku_name
 AND u.usage_start_time >= p.price_start_time
 AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time)
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY est_dollars DESC;

The FinOps operating model: visibility → accountability → optimization

Frame cost work as an operating loop, not a one-off cleanup. Visibility: system tables, enforced tagging, and an AI/BI dashboard showing spend by team, job, and SKU - cost becomes a metric people see weekly. Accountability: attribute every dollar to an owner via tags, set budgets per team, and let cluster policies make the efficient path the default path. Optimization: only now do the technical work - right-sizing, autoscaling and spot policy, Photon and serverless evaluations, layout and compaction - prioritized by the dashboard, and measured against it afterward so wins are provable. Then the loop repeats, because workloads drift. Skipping straight to optimization without visibility is how teams "save" money on clusters nobody was using anyway while the real burn continues untouched.