Databricks Migration & Data Governance
Case Study
From Local Scripts to a Governed Lakehouse: Migrating 500 GB/Month of Data Pipelines to Databricks
A national industry association processed over 500 GB of data per month — market research, membership data and public datasets — through pipelines that ran on analysts’ laptops. Nothing was versioned, nothing was reproducible, and the same metric could come out three different ways depending on who ran it.
Dateonic mapped every pipeline, defined a single source of truth per data domain, and rebuilt the platform on Azure Databricks with Terraform-managed infrastructure, dev/test/prod workspaces and governance enforced in Unity Catalog.
Pipelines now run as scheduled batch jobs for a few dollars per run, with end-to-end lineage behind every published figure — a trusted data platform for analytics and AI.
Turnover
5 Months
Technology:
Azure Databricks, Delta Lake, Unity Catalog
The challenge
- Pipelines ran locally. Transformations lived in notebooks on individual machines. If someone was away, the data was late.
- No single source of truth. Three scripts, three versions of the same number.
- No lineage. When a member questioned a figure in published research, reconstructing it took days.
- Undocumented access. Permissions had been granted ad hoc for years, with no complete picture of who could read what.
The goal was never just “move to the cloud” — it was to make every published number defensible.
Our approach
Discovery. We walked through each pipeline with the analysts who ran it: inputs, transformations, consumers, real-world frequency. This surfaced duplicated logic, pipelines running for years with no consumer, and the exact points where two pipelines disagreed. The output was a full pipeline inventory plus a single source of truth per domain, signed off by the business owner.
Governance first. Before any infrastructure code, we built a dependency matrix (dataset → upstream sources → downstream consumers) and an access matrix (business role → data domain → level of access), plus naming and catalog standards.
Target design. Each pipeline was mapped to a Databricks design: ingestion pattern, compute profile, schedule, output tables, owner. Most workloads are weekly research cycles, so we designed deliberately for batch execution on job clusters — the main driver of the cost result below.
The solution
Infrastructure as Code. Workspaces, clusters, jobs, catalogs, groups and permissions are all defined in Terraform. Nothing is created by clicking in a UI, so environments are reproducible and every change is reviewable.
Three-workspace model.
| Environment | Purpose | Access |
|---|---|---|
| Dev | Development and experimentation | Engineers and analysts, interactive |
| Test | Integration testing, release validation | Automated, via service principal |
| Prod | Scheduled production pipelines | Automated, via service principal |
Human access to test and prod is read-only by design — all writes are executed by service principals, so production data can’t be altered from an open notebook.
Governance in the platform, not in a document. Unity Catalog permissions map directly to the access matrix, entitlements are defined in Terraform (a role change is a code change with an approval trail), and table- and column-level lineage is captured automatically.
CI/CD with Azure Pipelines. Commit → validate → deploy to test → promote to prod. Pipeline code, infrastructure and job configs are versioned together, so a release is one reviewable unit.
Results
- All pipelines migrated to Databricks. Nothing runs on a laptop; every workload is scheduled, monitored and owned.
- A few dollars per pipeline run. Right-sized job clusters on weekly schedules mean compute is billed for minutes of execution, not idle infrastructure.
- End-to-end lineage. Any published figure can be traced from source system to report.
- One version of the truth. Conflicting metric definitions were eliminated in discovery and structurally prevented afterwards.
- Access defined in code. Reviewable, auditable and reproducible across environments.
- A foundation for AI. Governed, well-modelled data means AI use cases start from a trusted base, not a cleanup project.