Migration Playbooks

Why migrations are the RSA's bread and butter

RSAs are billable Professional Services consultants, and a large share of PS engagements are platform migrations: a customer has committed to the Databricks Data Intelligence Platform and needs someone embedded who can move a real estate - safely, on a timeline, with the business still running. In the interview loop, the project-delivery round and the panel presentation both reward people who can narrate a migration end to end: not just the target architecture, but the sequencing, the reconciliation, and the politics. This page gives you that narrative as a set of plays.

The engagement arc

Every migration, regardless of source platform, follows the same arc. Internalize it as a six-phase spine you can hang any war story on.

1. Discovery and assessment. Build an estate inventory: tables, jobs, scripts, reports, users, and the schedules that bind them. Profile workloads - which jobs burn the compute, which run daily vs. quarterly, which have failed silently for months. Map dependencies (job-to-job, table-to-report, system-to-system) because the dependency graph, not the table count, determines sequencing. Output a TCO model and business case: current platform cost, projected Databricks cost, migration effort, and the value unlocked (retired licenses, faster delivery, AI use cases the old stack cannot serve).
2. Wave planning. Pick a pilot that is real but contained - one business domain with a visible owner and a measurable outcome. Then group the remaining estate into waves by domain, not by technology layer, so each wave delivers something a business stakeholder can sign off. Plan the coexistence period explicitly: both platforms run in parallel, data flows are dual-fed or replicated, and there is a written rule for which platform is the system of record for what, at every point in time.
3. Execution. Convert code, replatform data, rebuild orchestration. Automate the repetitive 80 percent (DDL conversion, ingestion scaffolding) and spend human effort on the gnarly 20 percent (stored procedures, hand-tuned SQL, business logic nobody documented).
4. Validation. Parallel-run old and new, reconcile outputs automatically, and publish the evidence. This phase earns the cutover decision - see the reconciliation section below.
5. Cutover. Flip consumers per wave: repoint reports, switch schedules, freeze writes on the legacy side. Keep a rollback path defined and rehearsed for at least the first cycle.
6. Decommission. The phase customers skip and then pay for. Archive what compliance requires, kill the legacy schedules, reclaim the licenses, and book the savings - because the TCO case from phase 1 is only true once the old platform actually stops costing money.

Play 1: Hadoop to Databricks

The classic. On-prem Hadoop clusters are aging out - hardware leases expire, vendor support has consolidated, and the talent pool is shrinking. The play is a component-by-component remap plus a code uplift.

Spark code uplift is the deceptively easy part. Spark-on-YARN code generally runs on Databricks with minor changes, which tempts teams into lift-and-shift. Resist it: configurations tuned for static on-prem clusters (executor counts, shuffle partitions, memory fractions) are wrong on elastic cloud compute, and hardcoded HDFS paths, Hive Metastore calls, and RDD-era APIs all need cleanup. Treat the migration as the moment to modernize to DataFrame/SQL APIs and let the platform (Photon, adaptive query execution) do the optimization the old code did by hand.

Play 2: Legacy EDW to lakehouse

Teradata, Oracle, SQL Server data warehouses, and increasingly first-generation cloud DWs like Synapse Dedicated SQL Pools. These estates are smaller in raw volume than Hadoop but denser in logic: decades of stored procedures, dialect-specific SQL, and a semantic/reporting layer the business actually runs on.

Schema conversion is the easy layer - automated tools and conversion scripts handle DDL, type mapping, and constraint translation. Decide the target model deliberately: this is your chance to rationalize into a medallion structure (see Reference Architectures) rather than photocopy a 20-year-old star schema, including its mistakes.

Stored procedures are the hard layer. You have four honest strategies, and a real engagement uses all of them:

• Translate: mechanical conversion to Databricks SQL scripting or PySpark for procedures that are mostly set-based SQL.
• Refactor: procedures that are really orchestration in disguise (loops over dates, conditional branching, error handling) become Lakeflow Jobs task graphs with idempotent SQL/PySpark steps - the logic moves up into the orchestrator where it belongs.
• Redesign: row-by-row cursor logic gets rethought as set-based transformations; this is where you add the most value and the most risk, so it gets the heaviest reconciliation.
• Retire: discovery always finds procedures nobody can explain feeding reports nobody opens. Retiring them is a deliverable, not a shortcut - get sign-off.

Dialect translation is mostly tractable because Databricks SQL has absorbed many warehouse idioms. Teradata's QUALIFY, for instance, ports directly; BTEQ's procedural scaffolding does not:

-- Teradata BTEQ
.LOGON tdprod/svc_etl;
SEL acct_id, txn_dt, amt
FROM fin.gl_detail
QUALIFY ROW_NUMBER() OVER (PARTITION BY acct_id ORDER BY load_ts DESC) = 1;
.IF ERRORCODE <> 0 THEN .GOTO ERREXIT;

-- Databricks SQL: QUALIFY ports as-is
SELECT acct_id, txn_dt, amt
FROM fin.gl_detail
QUALIFY ROW_NUMBER() OVER (PARTITION BY acct_id ORDER BY load_ts DESC) = 1;
-- BT/ET transactions and .IF error branching do NOT port:
-- restructure as idempotent MERGE steps with Lakeflow Jobs
-- task dependencies and retry policies carrying the control flow.

The semantic and report layer decides whether the business calls the migration a success. Inventory every report, dashboard, cube, and extract reading the EDW; migrate the high-value ones to Power BI or AI/BI dashboards against Databricks SQL; and parallel-run the numbers, because a report that is fast but disagrees with last quarter's board pack is a failed migration in the eyes of the CFO.

Play 3: SSIS/SSRS-era Microsoft estates

This is your deep home turf - say so in interviews. The pattern: on-prem SQL Server databases, hundreds of SSIS packages doing ETL, SSRS for reporting, SQL Agent for scheduling. The Azure Databricks play: databases land in ADLS Gen2 as Delta via ADF or Lakeflow Connect ingestion; SSIS data-flow logic converts to PySpark or Databricks SQL (with ADF's SSIS Integration Runtime as a bridge for packages that must survive the transition unchanged); SSRS reports rebuild in Power BI or AI/BI dashboards; SQL Agent schedules become Lakeflow Jobs and ADF triggers. The trap is package count: 400 SSIS packages usually collapse into a few dozen parameterized, metadata-driven pipelines - which is exactly the metadata-driven ingestion pattern you built at Maersk and the Port Authority.

Snowflake: coexistence and migration honesty

Snowflake engagements are different in kind: the customer is not fleeing a dying platform, they are weighing two healthy ones. Your credibility - in the engagement and in the panel interview - depends on being honest about that.

Coexistence is increasingly practical: Unity Catalog's managed Iceberg tables and Delta Sharing mean Databricks-produced data can serve Snowflake consumers in open formats without copy pipelines, as of mid-2026. A phased "Databricks for engineering and AI, Snowflake keeps BI - for now" architecture is often the truthful recommendation, and it routinely converts to a fuller migration once the cost and capability evidence accumulates.

Validation and reconciliation discipline

Migrations are judged on one question: do the numbers match? Build the reconciliation harness in the pilot, run it every parallel-run cycle, and publish the results where stakeholders can see them. Layer the checks:

• Structural: table and column inventory, data types, nullability - per object, automated.
• Volumetric: row counts per table per load cycle, with trend tracking so drift is visible.
• Aggregate: sums, mins/maxes, and distinct counts on key measures by business dimension.
• Financial: balances tied to the penny against the legacy system and, where it exists, the general ledger. Tolerance is a business decision made explicit - in finance it is usually zero.
• Row-level: hash-based comparison on sampled or full keys for the tables where aggregates can mask offsetting errors.

from pyspark.sql import functions as F

KEYS = ["company_code", "account", "fiscal_period"]

legacy_agg = legacy.groupBy(KEYS).agg(F.sum("amount").alias("legacy_amt"))
lake_agg   = lake.groupBy(KEYS).agg(F.sum("amount").alias("lake_amt"))

breaks = (
    legacy_agg.join(lake_agg, KEYS, "full_outer")
    .withColumn("diff",
        F.coalesce("legacy_amt", F.lit(0)) - F.coalesce("lake_amt", F.lit(0)))
    .filter(F.abs(F.col("diff")) > 0.005)   # beyond half a cent = a break
)
# Persist every run to a recon Delta table; alert on breaks > 0.
# The published history IS the cutover evidence.

Two disciplines make the harness trustworthy: full outer joins, so rows existing on only one side surface as breaks instead of vanishing; and timing alignment, so you compare equivalent load cycles - half of all "reconciliation failures" in a coexistence period turn out to be one side reading data the other has not loaded yet.

The risk register staples

Every migration risk register contains the same recurring entries. Walk in with mitigations already drafted:

• Hidden consumers. The Access database in marketing, the cron job in a forgotten VM, the partner extract nobody owns - all reading the legacy platform via paths discovery missed. Mitigation: query-log mining, network-level connection auditing on legacy endpoints, and a read-only "brownout" period before hard cutover so survivors reveal themselves loudly but recoverably.
• SLA regressions. Some workloads will be slower or costlier at first - hand-tuned warehouse queries especially. Mitigation: benchmark the top-N workloads in the pilot, set expectations in writing, and budget tuning time per wave instead of discovering it in hypercare.
• Security model translation. Legacy grants, roles, row-level security, and Kerberos/AD integrations do not map one-to-one onto Unity Catalog. Design the target privilege model deliberately - groups, catalogs/schemas, row filters and column masks - and migrate to it; transliterating ten years of accreted grants reproduces ten years of accreted mistakes.
• People and process change. DBAs become platform engineers, ETL developers become PySpark engineers, the operations runbook changes shape. Mitigation: enablement as a planned workstream, embedded pairing during waves, and runbooks plus hypercare handover as deliverables - the platform is only migrated when the customer can run it without you.