Unity Catalog & Governance

Why Unity Catalog exists

Before Unity Catalog (UC), every Databricks workspace shipped with its own Hive metastore. That meant governance was fractured exactly along workspace boundaries: a table defined in the dev workspace did not exist in prod, permissions were managed per workspace with legacy table ACLs (or worse, with raw cloud storage IAM), there was no built-in lineage, and audit logs told you what clusters did, not who touched which column. Multiply that by five workspaces and three environments and you get the classic enterprise mess: nobody can answer "who can read salary data, and who actually did last Tuesday?"

UC fixes this by centralizing governance at the account level. One metastore per cloud region governs every workspace attached to it. Identities come from the account (synced from Entra ID / your IdP via SCIM), privileges are ANSI-style GRANTs stored once and enforced everywhere, and every access is logged and lineage-tracked automatically. On the Data Intelligence Platform, UC is no longer optional plumbing - it is the substrate for tables, files, models, functions, dashboards, Genie spaces, and serving endpoints alike.

The object model and the three-level namespace

UC organizes everything in a strict hierarchy. Get this picture solid - interviewers probe it casually ("where does a volume live?") to see if you have actually worked with it:

Metastore (one per region, account-level)
 └── Catalog                e.g. fin_prod
      └── Schema (database) e.g. r2r_gold
           ├── Tables       managed or external (Delta, and now managed Iceberg)
           ├── Views        standard + materialized; dynamic views for filtering
           ├── Volumes      governed file storage (managed or external)
           ├── Functions    SQL/Python UDFs - also usable as AI agent tools
           └── Models       MLflow registered models live here too

Every securable is addressed with the three-level namespace: catalog.schema.object. So fin_prod.r2r_gold.pl_actuals, not the two-level database.table of the Hive world. The legacy metastore still appears as a catalog literally named hive_metastore, which is the bridge you walk across during migration.

Sitting beside the data hierarchy are the objects that connect UC to cloud storage:

• Storage credential - wraps a cloud identity (on Azure, a managed identity via the Databricks access connector) that UC uses to reach ADLS/S3/GCS. Users never hold the storage keys.
• External location - a storage path plus a storage credential, itself a securable. Granting CREATE EXTERNAL TABLE on an external location is how you delegate "you may create tables under this container path" without handing out the credential.
• Connections - power Lakehouse Federation, so you can register and query external systems (SQL Server, Snowflake, PostgreSQL, etc.) as foreign catalogs under UC governance.

Catalogs are your coarsest isolation tool. A common enterprise pattern is catalog-per-environment-per-domain (fin_dev, fin_qa, fin_prod), with catalog-workspace bindings so the prod catalog is only even visible from the prod workspace - a natural fit for a finance domain like the R2R analytics estate Saikrishna leads.

The privilege model: GRANT, ownership, inheritance

UC privileges are ANSI-SQL style: GRANT / REVOKE on a securable, to a principal (user, group, or service principal - always prefer groups). Three rules carry most of the weight:

1. Inheritance flows downward. A grant on a catalog applies to every schema and object inside it; a grant on a schema covers its tables, views, volumes, and functions. Grant broad-read at the catalog level, restrict by exception below.
2. Access requires the full chain. To query a table you need SELECT on the table and USE SCHEMA on its schema and USE CATALOG on its catalog. Forgetting USE CATALOG is the number-one "why can't this group see anything?" ticket.
3. Owners are special. Every object has an owner (ideally a group, not a person) who can do anything to it, including granting. ALL PRIVILEGES exists but explicit grants read better in an audit.

GRANT USE CATALOG ON CATALOG fin_prod TO `grp-finance-analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA fin_prod.r2r_gold TO `grp-finance-analysts`;
REVOKE SELECT ON TABLE fin_prod.r2r_gold.pl_actuals FROM `grp-contractors`;
SHOW GRANTS ON SCHEMA fin_prod.r2r_gold;

Managed vs external - tables and volumes

Same Delta files either way; the difference is who controls the storage location and lifecycle.

Volumes follow the same split and exist to kill direct-path file access: instead of mounting storage or pasting abfss:// URIs into notebooks, you read and write files under /Volumes/catalog/schema/volume/... and UC enforces READ VOLUME / WRITE VOLUME. Landing zones for raw files, configs, and ML artifacts all belong in volumes now; DBFS mounts are legacy.

Row filters and column masks

Fine-grained access control attaches directly to the table, so it is enforced no matter which client queries it - SQL warehouse, notebook, dashboard, or a Genie space.

• A row filter is a SQL UDF returning BOOLEAN, bound to a table; rows where it returns false silently disappear for non-qualifying users.
• A column mask is a UDF bound to a column that rewrites the value at query time - return the real value for privileged groups, a redaction for everyone else.

-- Row filter: HR sees everything, business partners only their region
CREATE OR REPLACE FUNCTION hr_prod.gov.region_filter(region STRING)
RETURN is_account_group_member('grp-hr-admins')
    OR exists(
         SELECT 1 FROM hr_prod.gov.user_region_map m
         WHERE m.user_name = current_user() AND m.region = region
       );

ALTER TABLE hr_prod.gold.headcount_salaries
  SET ROW FILTER hr_prod.gov.region_filter ON (region);

-- Column mask: salary visible only to compensation admins
CREATE OR REPLACE FUNCTION hr_prod.gov.mask_salary(salary DECIMAL(12,2))
RETURN CASE WHEN is_account_group_member('grp-hr-comp-admins')
            THEN salary ELSE NULL END;

ALTER TABLE hr_prod.gold.headcount_salaries
  ALTER COLUMN salary SET MASK hr_prod.gov.mask_salary;

Older alternative: dynamic views with is_account_group_member() in the view body. They still work and remain useful when the logic must combine masking with joins or aggregation, but filters and masks are preferred because they protect the base table itself - no way to route around them by querying the table directly. Also know the newer ABAC direction: governed tags plus tag-driven policies, so one policy ("mask anything tagged pii.salary") covers many tables - in preview as of mid-2026, so hedge accordingly.

Lineage - and why auditors love it

UC captures lineage automatically from every query that runs on UC-enabled compute: table-level (this gold table is fed by these silver tables) and column-level (this KPI column derives from these source columns), across SQL, Python, Scala, and R, and extending to notebooks, jobs, and dashboards that touched the data. No agents to install, no third-party scanner to reconcile - it falls out of query execution. You can browse it in Catalog Explorer or query it programmatically via the lineage system tables.

Why auditors care: lineage turns "trust me, that P&L number comes from SAP" into a clickable, queryable evidence trail - source column to transformation to executive dashboard. Impact analysis works in reverse: before you change a silver table, lineage tells you every downstream consumer you are about to break.

System tables: observability and FinOps in SQL

The system catalog exposes platform telemetry as Delta tables you simply query - once an admin enables the schemas and grants access:

• system.access.audit - who did what: grants, logins, table reads, permission changes. Your audit evidence and anomaly-detection feed.
• system.billing.usage + system.billing.list_prices - DBU consumption by SKU, workspace, and tags. This is the FinOps backbone: join usage to custom tags and you get cost-per-team or cost-per-pipeline dashboards without exporting anything to a spreadsheet.
• system.access.table_lineage / column_lineage - lineage as data, for programmatic impact analysis.
• system.lakeflow.* - job and run telemetry for reliability reporting.

-- Top DBU consumers, last 30 days, by cost-center tag
SELECT u.custom_tags['cost_center'] AS cost_center,
       u.sku_name,
       SUM(u.usage_quantity) AS dbus
FROM system.billing.usage u
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY 1, 2
ORDER BY dbus DESC;

Delta Sharing

Delta Sharing is the open protocol for sharing live data without copying it. Two flavors to keep straight:

• Open sharing - the recipient is anyone with a Delta Sharing client (pandas, Spark, Power BI, Tableau); they authenticate with a token-bearing credential file. No Databricks required on their side. Read-only tables and volumes.
• Databricks-to-Databricks - both sides have UC; the share attaches as a catalog in the recipient's metastore. Richer: notebooks, views with masks intact, models, and governance stays attached. Cross-cloud and cross-region works.

The provider creates a SHARE (a named collection of objects), a RECIPIENT, and grants the share to the recipient - all auditable UC objects. This also underpins the Databricks Marketplace and Clean Rooms. For Saikrishna this is study knowledge, not a production claim - his sharing surface at ADM has been Power BI and Databricks Apps - but it is a reliable interview topic because it differentiates "copy the data over SFTP" thinking from lakehouse-native thinking.

Governing ML and AI assets

UC's quiet superpower is that the same model governs non-tabular assets:

• Models - MLflow models register into UC (catalog.schema.model) with versioning and aliases like @champion; EXECUTE governs invocation, and lineage links a model version to its training tables.
• Functions - SQL/Python UDFs in UC double as governed agent tools: a Genie space or Agent Bricks agent can only call functions its identity is allowed to execute.
• Vector Search indexes and serving endpoints - also UC-governed, which is why the ADM Policy RAG assistant's retrieval layer inherits the same access discipline as its tables. The newer Unity AI Gateway (Beta as of mid-2026) extends this control plane to LLM endpoints and agents.

Migrating from hive_metastore

High-level playbook an interviewer expects you to sketch:

1. Inventory the legacy metastore - tables, formats, mounts, who reads what (UCX, the Databricks Labs toolkit, automates assessment).
2. External tables: register the storage as external locations, then SYNC SCHEMA / SYNC TABLE to upsert UC entries pointing at the same files - metadata-only, repeatable, cheap.
3. Managed Hive tables: CREATE TABLE AS SELECT or deep clone into UC managed tables (data copy, but you land on predictive-optimization-eligible storage).
4. Repoint and govern: update jobs and views to three-level names, replace mounts with volumes, recreate ACLs as UC grants to groups, then lock down direct storage access so UC is the only door.
5. Decommission in waves - run dual-read for a sprint, watch audit/lineage system tables to confirm nothing still reads the old paths.

What's new as of mid-2026

• Metric views are GA - business metrics (measures, dimensions, joins) defined once in UC and queried consistently from SQL, AI/BI dashboards, and Genie; materialization of metric views is still Experimental. For someone who has built KPI frameworks for CFO-level consumers, this is the feature to name-drop: it moves metric definitions out of per-dashboard logic into governed objects.
• Managed Apache Iceberg tables are GA - full read/write through UC's Iceberg REST Catalog, alongside UniForm (Delta tables readable, read-only, by Iceberg clients). UC is positioning as the neutral catalog over both formats; Iceberg v3 brings deletion vectors, row tracking, and VARIANT.
• Unity AI Gateway (Beta) - the governance control plane extending UC-style policy to LLM endpoints, agents, and MCP servers; the legacy Mosaic AI Gateway features persist on serving endpoints. Naming is in flux - check current docs before quoting it in an interview.

Where to next: governance shapes how you model data in the first place - catalog and schema layout per layer is half of medallion design. Continue to Lakehouse Data Modeling, or revisit how the tables themselves work in Delta Lake Internals.