Lakeflow: Jobs & Declarative Pipelines

The Lakeflow umbrella - and the renames you must say out loud

At Data + AI Summit in June 2025, Databricks GA'd Lakeflow as the single brand for data engineering on the Data Intelligence Platform. Two of its three pillars are renames of products you already know, and interviewers consistently respect candidates who can use both names fluently - it signals you have real history with the platform and you keep current.

Two more naming notes worth a sentence in an interview: Lakeflow Designer is the no-code pipeline builder layered on top of declarative pipelines, and Declarative Automation Bundles is the March 2026 rename of Databricks Asset Bundles (the CLI command is still databricks bundle and the YAML is unchanged - a non-breaking rename). Also: classic billing SKUs still say "DLT" on the invoice even though the docs say SDP. Knowing that detail is exactly the kind of thing that separates "read a blog post" from "operates the platform".

Lakeflow Jobs in depth

Lakeflow Jobs is the control-flow layer: it doesn't know what a "good row" is, it knows what runs, in what order, on what compute, and what happens when something fails. If you've used ADF, Airflow, or Redwood, the mental model maps directly - a job is a DAG of tasks.

Tasks and dependencies

A job is one or more tasks with depends_on edges. Task types you should be able to list cold:

• Notebook - the workhorse; parameters arrive via widgets.
• Python script / Python wheel / JAR - for packaged, tested code rather than notebooks.
• SQL - run a saved query, a SQL file from a repo, refresh an AI/BI dashboard, or evaluate an alert on a SQL warehouse.
• Pipeline - trigger a Lakeflow Spark Declarative Pipeline update as one node in a bigger DAG (this is how the two halves of this page meet).
• dbt - run dbt projects against a SQL warehouse.
• Run Job - call another job as a child, enabling modular, composable orchestration.
• If/else condition and For each - native branching and fan-out/looping over a parameterized task.

Job and task parameters

Parameters exist at two levels. Job parameters are defined once and automatically pushed to every task; task parameters override or extend them per task. Dynamic value references like {{job.run_id}}, {{job.start_time.iso_date}}, or {{job.parameters.region}} are resolved at run time. For passing computed values between tasks, use task values: dbutils.jobs.taskValues.set(key="row_count", value=n) in an upstream task, then reference {{tasks.ingest.values.row_count}} downstream - for example, to feed an If/else task that skips the publish step when zero rows landed.

Conditional execution and repair runs

Beyond If/else tasks, every dependency edge carries a Run if condition: all succeeded (default), at least one succeeded, none failed, all done, at least one failed, all failed. That last pair is how you build native failure-notification or cleanup tasks without an external watcher. When a run does fail, a repair run re-executes only the failed and skipped tasks, reusing the same run ID, parameters, and successful task results - no re-running six hours of upstream work because the last task hit a transient API timeout.

Schedules and triggers

• Cron schedules - classic time-based, with a timezone and pause/resume.
• File arrival triggers - the job fires when new files land in a Unity Catalog volume or external location. This kills a whole class of "poll every 5 minutes" jobs.
• Continuous mode - the job restarts immediately on completion, for near-real-time loops without managing an always-on cluster yourself.
• Manual / API - Run Now from the UI, REST API, or an external scheduler (more on that below). Queueing and a max-concurrent-runs setting govern what happens when triggers overlap.

Compute: job clusters vs all-purpose vs serverless

Three options, one clear hierarchy as of mid-2026:

• Job clusters - created for the run, terminated after, billed at the cheaper jobs-compute SKU, isolated by design. Multiple tasks within one run can share a job cluster to avoid repeated spin-up.
• All-purpose clusters - interactive compute for humans. Attaching production jobs to them is a cost and reliability anti-pattern (see the warning below).
• Serverless jobs - GA and the strategic default: no cluster config at all, fast startup, and a UI default of "Performance optimized" mode (you can pick standard mode when cost matters more than latency).

Lakeflow Spark Declarative Pipelines in depth

Where Jobs is imperative control flow ("run this, then that"), declarative pipelines invert it: you declare the datasets and their quality contracts, and the framework derives the execution graph, manages checkpoints and incremental state, retries, and provisions its own compute. The former name - Delta Live Tables - still appears everywhere from old blog posts to billing SKUs, so use both names.

Streaming tables vs materialized views

Expectations: data quality as declared policy

An expectation is a named boolean constraint with one of three violation policies - and the three verbs are an interview staple:

• Warn (expect): keep the row, count the violation in pipeline metrics.
• Drop (expect_or_drop / ON VIOLATION DROP ROW): discard the row, count it.
• Fail (expect_or_fail / ON VIOLATION FAIL UPDATE): stop the update; the offending transaction does not commit.

Here is a small pipeline in the current Python API. Note the import: the historical import dlt still works, but new code uses from pyspark import pipelines as dp - the module rename that came with the Apache Spark Declarative Pipelines alignment. No migration is required for existing DLT code.

# Lakeflow Spark Declarative Pipelines (formerly: import dlt)
from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.table(comment="Raw journal entries landed as JSON.")
def journal_raw():
    return (
        spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/Volumes/finance/landing/journal/")
    )

@dp.table(comment="Validated journal entries.")
@dp.expect("amount_present", "amount IS NOT NULL")                 # warn: keep + count
@dp.expect_or_drop("valid_entity", "company_code IS NOT NULL")     # drop the row
@dp.expect_or_fail("no_future_posting", "posting_date <= current_date()")  # halt update
def journal_clean():
    return (
        spark.readStream.table("journal_raw")
        .withColumn("amount", F.col("amount").cast("decimal(18,2)"))
    )

AUTO CDC (formerly APPLY CHANGES INTO): CDC and SCD without hand-rolled MERGE

The declarative CDC API ingests a change feed and maintains a target as SCD Type 1 (overwrite, no history) or SCD Type 2 (history rows with __START_AT/__END_AT validity columns). Its quiet superpower is SEQUENCE BY: out-of-order change events are reconciled by the sequencing column, which a naive MERGE gets wrong. The current name is AUTO CDC (AUTO CDC INTO in SQL, dp.create_auto_cdc_flow() in Python); the legacy spelling APPLY CHANGES INTO / dlt.apply_changes() still appears in most material you'll study from - know both.

-- Target streaming table, then a CDC flow into it (SQL)
CREATE OR REFRESH STREAMING TABLE dim_supplier;

CREATE FLOW supplier_scd2 AS AUTO CDC INTO dim_supplier
FROM STREAM(supplier_changes_clean)
KEYS (supplier_id)
APPLY AS DELETE WHEN _op = 'D'
SEQUENCE BY change_ts
COLUMNS * EXCEPT (_op, change_ts)
STORED AS SCD TYPE 2;

-- Legacy equivalent began: APPLY CHANGES INTO dim_supplier FROM ...

Development vs production mode, and pipeline-managed compute

Pipelines own their compute - you don't attach them to a cluster; you configure (or let serverless handle) compute in the pipeline settings. The dev/prod toggle changes operational behavior, not logic: in development mode the cluster is reused across runs (fast iteration, no spin-up tax) and retries are limited so errors surface immediately; in production mode compute is fresh per update, terminates when done, and transient failures are retried automatically. Serverless pipelines are GA and remove the sizing question entirely - with the same "Performance optimized" default as serverless jobs.

Choosing: declarative pipelines vs Jobs + notebooks vs external orchestrator

These compose rather than compete: the cleanest 2026 answer is an external scheduler (if the enterprise mandates one) triggering a Lakeflow Job, which contains a pipeline task running the declarative transformation plus surrounding tasks for everything declarative pipelines don't do. Defined once and deployed across environments with Declarative Automation Bundles, that's the full reference answer.

Integrating with enterprise schedulers

Real enterprises rarely give Databricks the only seat at the orchestration table. The integration mechanics to know: trigger a job via the Jobs REST API (POST /api/2.2/jobs/run-now) with job parameters in the payload; poll runs/get for terminal state, or use webhooks/notifications for push-based status; let exactly one layer own retries; and pass a correlation ID (the scheduler's run key) in as a job parameter so you can trace a failure across systems. ADF additionally has first-class Databricks activities, and tools like Redwood RunMyJobs wrap the same API behind SAP-aware calendars and dependency chains.

Rapid recall

• Lakeflow = Connect (ingestion) + Spark Declarative Pipelines (ex-DLT) + Jobs (ex-Workflows); GA June 2025; UI says "Jobs & Pipelines".
• Jobs: task DAGs, two parameter levels + task values, Run-if edges, If/else and For-each tasks, repair runs, file-arrival/continuous triggers, serverless or job clusters - never all-purpose for production.
• Pipelines: streaming tables (incremental, append sources) vs materialized views (consistent recompute); expectations warn/drop/fail; AUTO CDC for SCD1/2 with SEQUENCE BY; dev mode reuses compute, prod mode is fresh + auto-retry.
• Code rename: import dlt → from pyspark import pipelines as dp; APPLY CHANGES INTO → AUTO CDC INTO; billing SKUs still say DLT.
• Layering answer: enterprise scheduler → Lakeflow Job → pipeline task; one retry owner per layer; deploy with Declarative Automation Bundles (ex-Asset Bundles, renamed March 2026).