DevOps on Databricks

Why this module carries senior-level weight

Anyone can demo a notebook. Senior candidates are judged on whether they can take a pipeline from a developer's branch to a governed production workspace with tests, approvals, rollback, and monitoring — and keep it running. Interviewers probe this area because it cannot be faked from tutorials: you either have run production hypercare or you have not. You have. Your job on this page is to map the Azure DevOps discipline you actually practiced at ADM onto Databricks-native tooling — bundles, Git folders, system tables — so you can speak both dialects.

Declarative Automation Bundles (formerly Databricks Asset Bundles)

Bundles are the Databricks-native infrastructure-as-code unit for workspace assets: a project folder containing source code plus YAML that declares the jobs, pipelines, and other resources the code needs, deployable to any target environment with one CLI command. Naming note for interviews: Databricks Asset Bundles were renamed Declarative Automation Bundles in March 2026. The rename is non-breaking — the CLI command is still databricks bundle and the YAML schema is unchanged — but using the current name (and knowing the old one) signals you are up to date. Most practitioners still say "DABs" or just "bundles."

Anatomy of databricks.yml

One root databricks.yml defines the bundle; resource definitions can be split into included files. The key blocks:

• bundle — the project name, used to namespace deployments.
• include — glob patterns pulling in additional YAML (e.g. resources/*.yml).
• variables — typed, defaultable values you can override per target or at the CLI (--var="catalog=prod"), so one definition serves every environment.
• resources — the assets the bundle manages: Lakeflow Jobs (the orchestrator formerly called Databricks Workflows), Lakeflow Spark Declarative Pipelines (the former DLT), model-serving endpoints, AI/BI dashboards, experiments, registered models, schemas, alerts, SQL warehouses, clusters, apps.
• targets — named environments (dev/staging/prod), each with its own workspace host, root path, run identity, and variable overrides. mode: development prefixes resource names with your username and pauses schedules so developers cannot collide; mode: production deploys clean names and enforces stricter validation, typically running as a service principal.

bundle:
  name: r2r_finance_pipelines

include:
  - resources/*.yml

variables:
  catalog:
    description: Target Unity Catalog
    default: fin_dev

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-dev.azuredatabricks.net
  staging:
    workspace:
      host: https://adb-stg.azuredatabricks.net
    variables:
      catalog: fin_staging
  prod:
    mode: production
    workspace:
      host: https://adb-prod.azuredatabricks.net
    variables:
      catalog: fin_prod
    run_as:
      service_principal_name: sp-r2r-deploy

The lifecycle is three commands: databricks bundle validate (schema and reference checks, runnable in CI without touching a workspace), databricks bundle deploy -t staging (sync code and create/update declared resources in that target), and databricks bundle run my_job -t staging (trigger and wait, ideal for smoke tests). Deploys are idempotent — the bundle is the source of truth and drift is overwritten.

Git folders and the source-control workflow

Git folders (the feature long called Repos) clone a Git repository into the workspace so notebooks and source files are edited on real branches. The reference workflow: each engineer works in their own Git folder on a feature branch; commits and pushes happen from the workspace UI or local IDE; a pull request triggers CI; merge to main triggers deployment. Two production rules matter:

• Nothing in production executes from a user's Git folder. Production code arrives via bundle deploy (or your CI artifact copy) as workspace files owned by a service principal. A service-principal-owned Git folder pinned to a release branch is the older pattern; bundles have largely superseded it.
• Keep logic in .py modules, not monolithic notebooks. Notebooks diff poorly and resist unit testing. A thin notebook (or job task) calling functions from a package gives you reviewable diffs and testable code.

CI/CD pipeline design

The canonical Databricks delivery pipeline has four gates, and it is the same shape whether you implement it in Azure DevOps or GitHub Actions:

1. Validate & test on the runner — lint (ruff), static checks, unit tests with a local SparkSession, databricks bundle validate. No workspace needed; fails fast and free.
2. Deploy to staging — bundle deploy -t staging under a service-principal identity (OAuth/OIDC, not PATs where avoidable).
3. Integration test in staging — bundle run the job against staging data on a small cluster or serverless; assert row counts, schema, and key data-quality expectations on the output.
4. Promote to prod — on merge to main or a tagged release, deploy to the prod target behind a manual approval gate (an "environment" approval in Azure DevOps or GitHub).

An Azure DevOps sketch — the tool you actually used — adapted to drive bundles:

# azure-pipelines.yml
trigger:
  branches: { include: [ main ] }

stages:
- stage: Test
  jobs:
  - job: unit
    steps:
    - script: pip install -r requirements-dev.txt
    - script: ruff check src/
    - script: pytest tests/unit --junitxml=results.xml
    - script: databricks bundle validate -t staging

- stage: Staging
  dependsOn: Test
  jobs:
  - deployment: deploy_staging
    environment: databricks-staging       # approvals/checks live here
    strategy:
      runOnce:
        deploy:
          steps:
          - script: databricks bundle deploy -t staging
          - script: databricks bundle run r2r_gl_load -t staging
          - script: pytest tests/integration  # asserts on staging output

- stage: Prod
  dependsOn: Staging
  jobs:
  - deployment: deploy_prod
    environment: databricks-prod          # manual approval gate
    strategy:
      runOnce:
        deploy:
          steps:
          - script: databricks bundle deploy -t prod

The GitHub Actions version is structurally identical: a pull_request workflow for stage 1, a push-to-main workflow using databricks/setup-cli for stages 2–4, with protected environments providing the approval gate and OIDC federation providing keyless auth to Azure.

Testing PySpark

Distinguish three layers — interviewers love asking where data-quality checks belong, and the answer is "a different layer than unit tests":

For unit tests, structure code so transformations are pure functions of DataFrames, then test with pyspark.testing.assertDataFrameEqual (built in since Spark 3.5 — no more hand-rolled collect-and-sort comparisons):

# src/transforms.py
def derive_variance(actuals_df, budget_df):
    return (actuals_df.join(budget_df, ["cost_center", "period"], "left")
            .withColumn("variance", col("actual_amt") - col("budget_amt")))

# tests/unit/test_transforms.py
from pyspark.testing import assertDataFrameEqual

def test_variance(spark):  # spark = local SparkSession fixture
    actuals = spark.createDataFrame([("CC10", "2026-05", 120.0)],
                                    "cost_center string, period string, actual_amt double")
    budget  = spark.createDataFrame([("CC10", "2026-05", 100.0)],
                                    "cost_center string, period string, budget_amt double")
    expected = spark.createDataFrame([("CC10", "2026-05", 120.0, 100.0, 20.0)],
        "cost_center string, period string, actual_amt double, budget_amt double, variance double")
    assertDataFrameEqual(derive_variance(actuals, budget), expected)

Databricks Connect is the alternative runtime for the same tests: the pytest fixture builds a DatabricksSession instead of a local one, and execution happens on remote serverless compute — useful when logic depends on Databricks-specific behavior, at the cost of speed and a workspace dependency in CI. Default to local Spark in CI; reserve Connect for interactive development and the rare DBR-dependent test. Data-quality checks, by contrast, run on every production execution because they guard against bad inputs, which no amount of unit testing can prevent.

Environment strategy: workspaces vs catalogs

Under Unity Catalog there are two viable isolation models, and a deliberate hybrid is the common enterprise answer:

Since UC metastores are per-region and shared across workspaces, the hybrid is natural: separate workspaces for prod vs non-prod, environment-named catalogs, and workspace-catalog bindings ensuring fin_prod is simply invisible outside the production workspace. Whatever the topology, parameterize the catalog in code (a bundle variable or job parameter) so promotion never requires an edit.

Secrets management

Never put credentials in notebooks, job parameters, or repo YAML. Databricks secret scopes hold key-value secrets accessed via dbutils.secrets.get(scope, key); values are redacted in notebook output. On Azure, prefer Azure Key Vault-backed scopes: the scope is a read-only mirror of a Key Vault, so rotation, access policies, and audit stay in Key Vault where the security team already lives — the pattern that matches your Azure estate. Two caveats worth volunteering: anyone with READ on a scope can print a secret if they try (redaction is cosmetic, ACLs are the control), and for service-to-service auth the stronger 2026 answer is to avoid long-lived secrets entirely — UC storage credentials with managed identities for data access, OAuth/OIDC federation for CI pipelines.

Terraform: account and workspace infrastructure

Bundles deploy into workspaces; the Databricks Terraform provider builds what bundles assume already exists — workspaces themselves, the UC metastore, catalogs and external locations, storage credentials, groups and service principals, cluster policies, network configuration. The clean division of labor: platform team owns Terraform (infrastructure, governance scaffolding, slow-changing), data teams own bundles (jobs and pipelines, fast-changing, per-project). Be plain about your level here: your Terraform is working knowledge — you can read provider configs and explain the division of responsibilities, but infrastructure provisioning was not your hands-on lane. Saying so, then articulating the bundle/Terraform boundary crisply, lands better than bluffing through a state-file question.

Observability in production

Production discipline is what you sell hardest, because you have lived it. The Databricks-native toolkit:

• Job monitoring — Lakeflow Jobs gives run history, durations, task-level retries, and per-run lineage in the Jobs & Pipelines UI; configure retries and timeout per task, not just per job.
• Alerting — job-level notifications (email, webhooks to Teams/Slack/PagerDuty) on failure and on duration thresholds — a job that usually takes 20 minutes still running at 90 is an incident before it fails. SQL alerts on top of gold tables catch silent data problems.
• System tables — UC-governed operational data under the system catalog: system.lakeflow.job_run_timeline for run outcomes and durations, system.billing.usage for cost per job, audit logs, query history. This is the basis for an SLA dashboard: trend run durations, flag jobs drifting toward their window, attribute spend to pipelines. It turns "we monitor jobs" into "we query our operations."
• SLAs — define them as data-availability commitments ("gold P&L tables current by 6:00 local on business days"), measure them from system tables, and alert on leading indicators, not just breaches.

Runbooks and hypercare

The least glamorous, most differentiating topic on this page. A runbook is the per-pipeline operational contract: what the pipeline does and its SLA, how failures present, triage steps in order (check the job run page → task error → upstream source availability → data-quality output), safe rerun procedure (which tasks are idempotent, what a backfill needs), escalation path with names, and known failure modes with fixes. The test of a good runbook is that the on-call who did not build the pipeline resolves the incident without paging the author. Hypercare is the deliberately elevated support window after go-live — daily run reviews, tightened alert thresholds, the build team on point rather than standard support, explicit exit criteria (for example, N consecutive clean cycles and no Sev-1s) before handover to steady-state operations. Treating hypercare as a planned phase with an exit gate, rather than "we'll watch it for a while," is a signal of operational maturity that most candidates cannot give — you can, from your ADM delivery work.