Migrating from Hadoop to Databricks means moving data, processing logic, access policies and the systems that use them together. Start by copying HDFS data into cloud object storage.
Adapt Hive and Spark workloads, and redesign dependencies that need a different runtime or serving system. Validate each workload before switching production. Migration is complete when the new workloads are accepted and their legacy dependencies can be retired safely.
Why Migrate from Hadoop to Databricks?
Migration makes sense when a measured constraint, such as an approaching hardware refresh, missed processing windows or an unsupported dependency, justifies the conversion effort and future operating cost.
Establish the baseline first: infrastructure and support spend, engineering effort, failed runs, data freshness and the requirements the current platform cannot meet.
Separating object storage from processing capacity can change how you provision workloads. Databricks offers job orchestration, SQL analytics and governed data access, but those capabilities do not establish that every workload will become cheaper or faster.
Compare a representative implementation with the current operating requirement before committing the estate.
Keep or replace workloads selectively. An HBase application relying on random, real-time reads and writes needs a serving-system decision; migrating its analytical extracts does not migrate the application.
Apache HBase describes this access pattern explicitly. Residency restrictions, network constraints, existing contracts or an unfavorable business case may also justify coexistence or deferring part of the move.
Preparing for Your Migration
Build the inventory from job histories, table definitions, access records and conversations with workload owners. Record what actually runs and who consumes it; a filesystem listing alone cannot identify the migration scope.
DDL defines table structures, UDFs are custom functions, and SerDes control how Hive reads and writes stored data.
| Hadoop component | What to assess | Decision required |
|---|---|---|
| HDFS | Paths, size, file count, formats, partitions, encryption zones and active writers | Copy, rebuild from source, archive or retire; define a consistent input boundary |
| Hive and Impala | DDL, views, SQL, SerDes, UDFs, schema drift, ACID tables and JDBC/ODBC consumers | Choose target table format and SQL behavior; identify reads requiring export or rewrite |
| HBase | Row-key design, client APIs, latency, updates and analytical exports | Retain or select a serving system; migrate analytical data separately |
| YARN | Queues, resource limits, priorities and workload contention | Translate operating requirements into target compute, concurrency and scheduling controls |
| Spark and MapReduce | Versions, libraries, entry points, local files and filesystem assumptions | Port compatible code or redesign runtime-dependent logic |
| Oozie, cron and other schedulers | Dependencies, calendars, sensors, retries, backfills and external actions | Preserve scheduling behavior in the chosen orchestrator |
| Streaming and ingestion | Sources, offsets, retention, checkpoints, updates and deletes | Establish replay, synchronization and recovery methods |
| Governance and consumers | Kerberos/LDAP identities, Ranger/Sentry policies, storage access, reports and exports | Map identities, access intent, downstream contracts and acceptance owners |
For every workload, record its owner, upstream and downstream dependencies, refresh or latency requirement, historical coverage, conversion blockers and acceptance evidence. Include scripts and exports outside the Hadoop cluster: a scheduled spreadsheet extract can prevent retirement even after the main dashboards move.
Prioritize a first wave with a clear consumer, observable output and bounded dependencies. Include a representative compatibility challenge rather than choosing only trivial jobs.
Move critical or tightly coupled workloads once their unresolved dependencies and recovery requirements are understood. Retire unused workloads only with owner approval; absence from a short query-history window is insufficient evidence.
Illustrative prioritization: a daily Hive aggregation over immutable Parquet files may be an early candidate. A month-end pipeline using a custom SerDe needs a conversion pilot first. An HBase-backed operational application stays outside the analytical migration wave until its serving design is approved. These are planning examples, not Dateonic project results.
Which Migration Strategy Fits Each Workload?
Choose lift-and-shift, replatforming or refactoring per workload. The label describes the extent of change; it does not predict a fixed cost or delivery time.
| Strategy | Appropriate starting condition | Changes and limitations |
|---|---|---|
| Minimal-change port, often called lift-and-shift | Spark code and dependencies are compatible with the chosen Databricks runtime and compute | Change paths, identity, packaging and job submission. This does not transplant YARN, HBase or MapReduce unchanged. |
| Replatform | Business logic remains useful, but storage, table management or orchestration must change | Move to object storage, establish target tables and Unity Catalog permissions, and adapt SQL and jobs. Prove semantic equivalence. |
| Refactor | Legacy APIs, state handling or application behavior conflict with the target | Redesign processing or serving logic. Validate the new contract with consumers; more change does not automatically mean more value. |
| Retain, archive or retire | A workload has no approved migration case or still requires its source platform | Document the owner, retention obligations and continuing dependencies. Keep residual Hadoop costs in the business case. |
Dateonic’s enterprise warehouse migration playbook covers cross-platform workload disposition in more detail. For Hadoop, the immediate question is which filesystem, execution and access assumptions must change.
What Should the Target Architecture Look Like?
Design the destination around the selected cloud and consumer requirements. Object storage holds the migrated files and table data; Unity Catalog governs registered assets; processing and SQL serving use compute appropriate to their contracts. HDFS, Delta Lake and Unity Catalog serve different roles and are not interchangeable replacements.
Illustrative Azure target architecture, not a deployed customer design:
| Legacy responsibility | Target responsibility |
|---|---|
| HDFS historical datasets and arriving files | A controlled ADLS Gen2 landing area, with manifests and an agreed synchronization boundary |
| Hive transformations and Spark batch jobs | Validated Spark or SQL transformations producing Delta tables; orchestration through Lakeflow Jobs or an enterprise scheduler |
| Metastore definitions and access policies | Reviewed catalog/schema/table mappings, Unity Catalog grants and cloud storage permissions |
| Impala/Hive BI queries and operational applications | Databricks SQL for suitable analytical consumers; a separately approved serving system for application access patterns |
Keep landing data separate from curated tables so retries and reconciliation have a stable reference. In an AWS design, use S3; in a Google Cloud design, use Cloud Storage, with the corresponding cloud identity and network configuration. Reusing the logical pattern does not make cloud-specific configuration portable without changes.
Unity Catalog distinguishes managed and external assets. Choose ownership and lifecycle deliberately: an external table may suit controlled coexistence, while a managed table changes who manages its underlying storage lifecycle. Neither choice removes the need to identify all readers and writers.
Step-by-Step Guide to Migrate from Hadoop to Databricks
Run the migration in dependency-aware waves. Each phase below produces evidence needed by the next; the phases can overlap across workloads without bypassing their acceptance gates.
1. Establish the platform and transfer boundary
Approve target storage, identity, connectivity, encryption/key access and development-to-production deployment controls before copying production data. Confirm that transfer workers can reach the required HDFS and cloud endpoints, and limit their credentials to the migration scope.
Choose a historical snapshot or controlled export to establish the initial version of the data. Then define how changes made during the copy will reach the target. For files that never change after arrival, a file manifest, a list of files to transfer, and a final catch-up can be sufficient.
Tables that change need a consistent source export and a method that captures updates and deletes. Repeatedly copying their directories does not provide database change capture.
Deliverable: a tested access path, an approved snapshot/cutoff definition, transfer ownership and a recovery plan for interrupted copies.
2. Move HDFS data into cloud storage
Apache DistCp supports transfers to object stores, including S3, Azure ABFS and GCS, when the filesystem connector, dependencies and credentials are configured on the source workers. Schedule copy capacity alongside existing Hadoop jobs so the migration does not consume their required resources.
Cloud-specific alternatives have different prerequisites. Microsoft documents Azure Data Factory approaches for HDFS migration; AWS provides an on-premises HDFS-to-S3 DistCp pattern; Google Cloud documents agent-based HDFS transfers.
Select a method against the actual network, authentication, source load and restart requirements rather than data volume alone.
Do not assume HDFS behavior survives the copy. Apache’s DistCp guidance documents object-store limitations around permission preservation, atomic rename and checksum comparison. A successful copy or matching byte count does not establish table-level correctness or reproduce Ranger policies.
After landing, choose whether to retain supported source files temporarily or write target Delta tables. In-place Parquet-to-Delta conversion has specific requirements: avoid concurrent file writes, verify partition metadata and account for other tables sharing the directory.
ORC, Avro, text and custom SerDe-backed data require an appropriate read/export-and-rewrite path; changing a table declaration does not convert their files. Hive ACID readers merge base and delta files, so those layouts need transaction-aware reading or export.
Deliverable: transfer manifests, exception records, reconciliation of landed data and a documented table-format conversion method.
3. Rebuild metadata, identities and access policies
Export source DDL and map databases, tables, views, locations and ownership into the target namespace. Review partition locations, schema types, null and timestamp semantics, and supported formats. An on-premises Hive metastore is not a Unity Catalog metastore that can simply be restored on the destination.
Federation can assist some coexistence designs, but its boundary matters. The Databricks AWS Hive metastore federation documentation excludes on-premises HDFS data and remote Hive metastore access over Thrift.
Foreign catalogs backed by external Hive metastores are read-only. Verify the selected cloud’s requirements rather than treating federation as a bridge to any live Hadoop cluster.
Re-express Ranger/Sentry policy intent using target groups, grants and any required row/column controls. Map service identities separately from human users.
Test both permitted and denied access, including direct cloud-storage access that could bypass catalog controls. Databricks warns against giving non-Unity Catalog identities storage-level access to managed tables.
Deliverable: approved object and identity mappings, least-privilege access tests and a register of policies requiring redesign. Unity Catalog provides governance capabilities; configuring it does not by itself certify data quality or regulatory compliance.
4. Adapt SQL, processing and orchestration
For Spark applications, compare the source version with the Spark version in the chosen Databricks Runtime. Review Python/Scala/Java dependencies, UDFs, serialization, filesystem paths, local-disk use and submission parameters.
The Apache Spark SQL migration guide documents behavioral changes across versions; test the upgrade path that applies to your application.
For Hive and Impala SQL, test results as well as syntax. Concentrate on casts and overflow, decimal precision, dates and time zones, null handling, UDFs and storage-specific DDL. A query that compiles can still change a business measure.
Select compute after checking compatibility. For example, serverless notebooks and jobs do not support Spark RDD APIs. Porting an RDD-based application may require API changes or another compatible compute option. This limitation should be discovered during assessment, not during cutover.
MapReduce and Pig processing generally require logic conversion or redesign rather than resubmission to a Databricks YARN queue. Reusable libraries may reduce the work, but their compatibility still needs validation.
For Oozie and other schedulers, preserve calendars, data-availability conditions, retries, backfills and external tasks. Lakeflow Jobs provides task orchestration and dependencies; retaining Airflow or another enterprise scheduler can be appropriate where it coordinates systems outside Databricks.
Review the file layout as well as the code. Many small files can slow Delta reads; assess file sizes and layout against the target workload instead of copying legacy partitioning by default.
Photon accelerates supported operations, but unsupported operations fall back to Spark. Evaluate its actual runtime and total workload cost rather than assuming a fixed improvement.
Deliverable: versioned target code and job definitions, a compatibility exception list and equivalent scheduling behavior for the migration wave.
5. Establish incremental processing and recovery
Define how the target catches up after the initial historical load. Auto Loader ingests files arriving in supported cloud storage. It does not move HDFS data to the cloud or detect row-level deletes in a source database.
Specify how the file feed represents overwrites and corrections, and how the target handles duplicate business records.
For Kafka or other event sources, record the starting offsets, the positions from which events are read, and how long those events remain available for replay. Check how the destination behaves when a write is retried.
Streaming jobs that retain state need a separate recovery check. Checkpoint compatibility depends on query and source changes, so do not assume an old checkpoint can be copied to a redesigned query. If a new checkpoint is required, define replay and duplicate handling before starting it.
Deliverable: a documented synchronization watermark or offset boundary showing how far the target has caught up, tested retry/replay behavior and evidence that updates, deletes and late arrivals are handled as required.
Which Migration Tasks Can Tools Actually Automate?
Evaluate automation by source technology and task. Assessment support is not conversion support, and code conversion is not proof that a business output is correct.
| Tool or capability | Verified scope | Boundary to retain in the plan |
|---|---|---|
| DistCp | Distributed file copying with configured filesystem connectors | Does not migrate SQL, table semantics, identities or application behavior |
| Lakebridge | Assessment, conversion and reconciliation capabilities varying by source | The support matrix lists Hive Analyzer support, with no Hive converter or reconciler listed; Pig, Sqoop and Oozie analysis likewise does not establish automatic conversion |
| UCX | Assessment and migration workflows for upgrading an existing Databricks workspace’s Hive metastore assets toward Unity Catalog | Not an end-to-end on-premises Hadoop migration engine; storage, format and workflow requirements apply |
| Auto Loader | Incremental ingestion of files already in supported cloud storage | Not HDFS transport or a universal database CDC connector |
Check the Lakebridge support matrix against the intended version and source dialect before budgeting automation. Lakebridge and UCX are Databricks Labs projects provided without formal support SLAs. For unsupported tasks, retain an explicit manual conversion, review and testing allowance; do not estimate it away.
What Must Pass Before a Workload Can Cut Over?
Validate the source and target against the same logical input window. Capture the source cutoff, target code version, expected behavior and named acceptance owner before running comparisons.
- Data coverage: reconcile tables, partitions, counts, keys and business aggregates. Investigate missing, extra, changed and duplicate records. Matching counts alone are insufficient.
- Semantics: verify decimals, timestamps, nulls, schema contracts and downstream report measures. Normalize representations deliberately before using row hashes; different file formats need not have matching file checksums.
- Incremental behavior: exercise an update, delete, late arrival, duplicate, backfill and retry where relevant. Confirm that a failed run can recover without unexplained loss or duplication.
- Service requirements: measure completion windows, freshness, latency and concurrency using representative data and the intended compute configuration. Record startup conditions, repeat runs and total workload cost.
- Access and operations: prove positive and negative permission cases, alert routing, ownership and recovery procedures. Obtain consumer acceptance for the required business cycle.
Set tolerances by measure and business impact before inspecting the results. A known legacy defect may justify an approved target difference; an unexplained discrepancy remains unresolved.
Keep comparison outputs and exception decisions with each workload. Dateonic’s migration testing strategy provides the broader reconciliation and acceptance framework.
How Should Cutover, Rollback and Decommissioning Work?
Cutover is the point when production consumers switch to the target. Approve it only when blocking tests pass, owners accept any remaining exceptions, and the target meets the agreed data-freshness requirement. Operations must also be able to execute the recovery plan. A successful historical copy alone is insufficient.
Before switching, identify the authoritative writer, the system allowed to create or change records, and the final source cutoff. Document the catch-up process and every consumer connection that must change.
Stop legacy writes where required, reconcile data through that cutoff, and record the target as the system of record. If both platforms continue running, state which one owns writes and how their outputs are compared.
Rollback means returning production to the previous platform. It needs a data plan as well as a connection change. Record the trigger, decision owner and retained source state, then define how changes accepted after cutover will be replayed or reconciled.
If the legacy platform no longer receives those changes, switching consumers back can return stale results. Define when rollback stops being an immediate switch and starts requiring a data-recovery operation.
Decommission only after the agreed observation period and business cycles, consumer confirmation, audit/retention requirements and replay obligations are satisfied. Remove schedules and credentials deliberately; preserve required metadata, code and evidence.
Confirm that HBase, remaining YARN applications or shared services do not still require the cluster. Track residual risks in the migration risk register.
What Belongs in the Migration Cost and TCO Model?
Compare total cost of ownership (TCO) for the same workloads and operating period. Separate one-time discovery, conversion, transfer, testing and training from recurring operation. Include the cost of running both platforms during coexistence, and keep remaining Hadoop infrastructure in the model until it can be retired.
| Cost boundary | Include in the comparison |
|---|---|
| Current Hadoop operation | Hardware/support or cloud-managed service charges, storage, operations effort, failure recovery and remaining commitments |
| Migration and coexistence | Transfer/network charges, copy capacity, duplicated storage, parallel processing, conversion tools, engineering and acceptance work |
| Target operation | Databricks service usage, applicable cloud infrastructure, storage, networking, table maintenance, monitoring and support |
| Exit and residual estate | Archive/retention, equipment or contract exit, retained serving systems and workloads that cannot yet move |
Lakeflow Jobs pricing distinguishes classic and serverless compute. Classic pricing adds underlying cloud compute; serverless includes it in the service charge. Do not add that virtual-machine cost twice. Storage and connectivity may still add charges.
Use prices for the selected cloud, region, service and commercial terms. Microsoft sets Azure Databricks pricing.
Measure the cost of a complete workload cycle that meets its correctness and service requirements. A smaller hourly bill with more retries or a missed reporting deadline is not an equivalent result. Keep detailed post-migration tuning in the separate Databricks cost optimization guide.
Common Pitfalls When Migrating from Hadoop to Databricks and How to Avoid Them
Some failures sit between teams rather than inside the conversion code. Address these explicitly in the wave plan:
- Transfer work starves production: agree a Hadoop resource and network budget with operations, monitor source-job impact and pause the copy when it breaches that budget.
- Access becomes broader during coexistence: give transfer identities an expiry and owner, review temporary storage grants and revoke them when the migration path closes.
- Performance tuning hides a correctness defect: freeze an accepted functional baseline before changing runtime, compute or layout. Use workload performance measurements to evaluate subsequent changes.
- The retirement saving never arrives: assign a decommissioning owner and record the final dependencies, including non-analytical services. Keep their costs in the forecast until they are removed.
Start with an Assessment Your Team Can Act On
The first decision is whether a representative workload can meet its data, access, recovery and service requirements at an acceptable operating cost. Use that evidence to sequence the remaining estate and identify workloads that should stay, change or retire.
Dateonic’s published migration and data governance case study describes moving locally run analyst pipelines to Azure Databricks, with Terraform-managed environments and Unity Catalog controls. It illustrates platform and governance delivery, not a Hadoop-specific migration or evidence for Hadoop performance or savings claims.
If you need help assessing the Hadoop estate and turning it into an implementation plan, Dateonic’s Databricks Migration service covers assessment, target architecture, roadmap and migration delivery. Bring the workload inventory, source constraints and acceptance requirements to define a practical first wave.