A Databricks project should not enter implementation with key requirements scattered across sales notes, architecture assumptions and team backlogs. Discovery brings those inputs together so decision-makers can approve, revise or reject the proposed scope.
The work covers the intended business outcome, workloads, current data estate, constraints, ownership and acceptance criteria. It stops before detailed design and configuration.
This boundary also reflects the Databricks production-planning guidance, which treats requirements, network planning and identity-provider information as inputs to production design, then separates design from deployment and testing.
What is a Databricks discovery phase?
A Databricks discovery phase is a structured pre-project assessment that turns a proposed initiative into a scope that can be reviewed and estimated. It answers four questions:
- Which business outcome and production use cases justify the project?
- What must the implementation accommodate in the current data and technology estate?
- Which organizational, security, governance and operational constraints will shape the solution?
- What can be agreed now, and what still needs validation?
Evidence may come from workshops, documentation, sample data, architecture diagrams, an existing POC and stakeholder interviews. This phase is useful before a new implementation, POC productionization, a migration wave or a substantial new workload.
Discovery vs implementation vs Health Check
| Phase | Main question | Primary object of assessment | Typical outcome |
|---|---|---|---|
| Discovery | What should we build, why, under what constraints, and what must be known before scope approval? | Business needs, current estate, candidate workloads, requirements, ownership and dependencies | Agreed scope, decision framework, initial target state, risks, assumptions and estimate inputs |
| Implementation | How do we design, configure, build, test and productionize the agreed scope? | Databricks platform components, integrations, workloads and delivery process | Working production platform or workload, documentation and handover |
| Health Check / Audit | What should be improved in the Databricks environment we already operate? | Existing architecture, workloads, governance, security, performance, reliability and cost | Findings and a prioritized improvement roadmap |
The handoff needs to be clear. Discovery ends when the organization can approve or revise a defined scope. Databricks implementation begins with detailed design and delivery against that scope.
If the actual need is to diagnose a live platform, a Databricks Health Check or audit is the appropriate starting point.
Start with the business outcome, not a proposed feature list
Stakeholders often arrive with a solution already in mind: move the warehouse, implement a lakehouse, add streaming or create a GenAI application. The first task is to separate the desired outcome from the proposed technology.
For each use case, establish:
- the business problem and the decision or process it should improve;
- the owner accountable for the outcome;
- the people and systems that will consume the result;
- the workload that must reach production first;
- the evidence that will make the first release acceptable.
This discussion may expose a platform-fit issue. A use case might not need Databricks, might belong on an existing platform or might depend on a data foundation that does not yet exist.
Finding that out before approval is useful. Discovery is not there to justify a tool choice that has already been made.
Map the current data estate
The delivery team needs a usable picture of what exists before it can estimate what must change. A statement such as “data comes from ERP, CRM and e-commerce” rarely provides enough detail.
Build an inventory of the sources and data relevant to the proposed release. Depending on the project, that may cover:
- databases, SaaS applications, cloud storage and integration platforms;
- existing warehouses, lakes and transformation tools;
- datasets required by the first production use cases;
- approximate volume, growth and historical depth;
- batch, event and streaming ingestion patterns;
- current transformations and where they run;
- downstream reports, applications, APIs and models;
- known quality problems and duplicated logic;
- source owners, access routes and change windows.
The inventory need not document every table. Its detail should match the scope. For the first release, trace required data from origin to consumer and record gaps affecting feasibility, effort or acceptance.
Weak documentation is itself a finding. A source described as “available through an API” may still require vendor approval, authentication work, rate-limit testing or a different extraction method.
Until that is checked, it remains an assumption rather than a routine integration task.
Create a workload inventory
Source systems describe where data comes from. A workload inventory describes what the platform must do with it.
Candidate workloads may include data engineering, BI and SQL, streaming, machine learning, GenAI, data sharing and operational analytics. Each places different demands on latency, concurrency, data access, reliability and operating ownership.
| Attribute | Questions to answer during discovery |
|---|---|
| Purpose and consumers | Who uses the output, in which process and for which decision? |
| Inputs and dependencies | Which data, systems, teams or models must be available first? |
| Frequency and latency | Is the workload scheduled, continuous or user-triggered? How fresh must the output be? |
| Scale and concurrency | What volumes, growth patterns and simultaneous demand must the design accommodate? |
| Criticality | What is the business impact of delay, failure or incorrect output? |
| Quality and acceptance | How will the team validate completeness, correctness and fitness for use? |
| Ownership | Who develops, approves, supports and funds the workload after launch? |
Capture these characteristics before choosing cluster sizes or committing to a compute model. Those decisions belong to architecture and implementation planning.
The Databricks Well-Architected Framework is a useful cross-check. It covers operational excellence, security, reliability, performance, cost, governance and interoperability. Discovery needs to uncover the requirements that will shape these areas, not design each one.
Identify cloud and infrastructure constraints
Databricks will sit inside an existing enterprise environment. Cloud standards, landing zones, network ownership and identity controls can narrow the available design before an architect opens a diagram.
Suppose the data sits in Azure, users authenticate through Microsoft Entra ID, a central platform team owns Terraform modules, and production systems may communicate only over approved private routes.
Those facts affect the workspace and network design. They also identify teams whose approval or delivery work must appear in the plan.
Record the cloud footprint and regions, source and consumer locations, landing zones, required on-premises or cross-cloud connectivity, identity lifecycle, infrastructure-as-code standards, and restrictions involving firewalls, DNS, proxies or egress.
Also capture account ownership and the lead times of platform, network and security teams.
Current Databricks guidance shows that workspace choices can depend on custom networking, on-premises connectivity, organizational structure, geography, security and operational requirements.
The relevant references are the official guides to workspace strategy and network architecture.
A connectivity path with no owner or no prior validation belongs in the dependency register. It should not be buried in the estimate as standard configuration.
Clarify security, privacy and compliance requirements
“Must be GDPR compliant” is not a technical requirement an architect can implement on its own. Legal, privacy, risk and security teams need to translate applicable obligations and internal policies into controls and evidence expectations.
Relevant questions include:
- Which data is confidential, personal, regulated or otherwise sensitive?
- Which users, groups, applications and service identities need access?
- Must data or processing remain in a defined region?
- Are public network paths permitted?
- What are the encryption and key-management expectations?
- Which activity must be logged, and how long must evidence be retained?
- What separation is required between teams, clients, domains or environments?
- Which privileged-access, incident-response and security-monitoring rules apply?
- Are there restrictions on data sharing, model access or external services?
Databricks documents authentication, access control, private connectivity, encryption, secret management and compliance features as separate areas.
The security and compliance overview is a useful capability reference. It does not replace the organization’s own legal interpretation or security approval.
Establish data governance and ownership
Unity Catalog architecture should reflect how the organization owns and governs data. It cannot fix an ownership model that nobody has agreed.
Before catalogs, schemas or permissions are designed, clarify:
- which business domains are included in the first scope;
- who owns and stewards each important dataset or data product;
- who approves access and who administers it;
- existing classification, retention and deletion policies;
- lineage, discoverability, audit and quality expectations;
- how shared data crosses domain or organizational boundaries;
- whether governance operates centrally, locally or through a federated model.
These findings become inputs to the later design. Databricks’ Unity Catalog architecture guidance connects the governance operating model with ownership, access control and policy enforcement.
The current Unity Catalog overview also documents its role in access control, lineage, auditing, classification and data-quality monitoring. The platform structure should follow the agreed governance model, not stand in for it.
Assess the engineering and operating model
A production platform still needs owners after the implementation team leaves. The discovery work should show where responsibilities sit and whether the people expected to take them on are available.
| Area | What to establish |
|---|---|
| Skills | Internal experience with Databricks, Spark, SQL, cloud infrastructure and data engineering |
| Delivery practices | Repositories, review rules, CI/CD, infrastructure as code, testing and release management |
| Platform operations | Monitoring, incident management, support hours, escalation and on-call expectations |
| Ownership | Responsibility for the platform, workloads, governance, approvals and ongoing cost |
| Partner split | Work retained by the client, delivered externally or transferred during handover |
| Enablement | Documentation, training and support required after go-live |
This is narrower than a guide to the roles required for Databricks implementation. The concern here is whether every responsibility in the proposed scope has an owner.
Consider a project that assigns infrastructure deployment to the client’s platform team. If that team has not accepted the work or cannot join during the planned window, the schedule rests on an unconfirmed assumption. That needs resolving or recording before approval.
Define non-functional requirements in business terms
Requests for a platform that is “fast”, “scalable” or “highly available” are too vague to guide architecture or estimation. Translate them into expectations for each workload:
- How fresh must the data be for its consumers?
- How many concurrent users or requests are expected?
- Which processing deadlines must be met?
- How long can the business tolerate an outage?
- How much data loss would be acceptable after a major failure?
- Which events require an alert, and who responds?
- What growth must the first design accommodate?
- Which logs and evidence need to be retained?
- What support hours and escalation paths are required?
Do not assign one universal service level to every workload. A daily finance report, an inventory feed and an online model endpoint may need different latency, recovery and monitoring arrangements.
Databricks ties observability planning to workload criticality, service expectations and operational requirements.
Its high-availability and disaster-recovery guidance also treats recovery time and recovery point objectives as business inputs that influence replication and failover design.
Expose dependencies and migration constraints
Some of the work that controls the schedule will sit outside the Databricks team’s remit.
Common examples are access to upstream systems, changes owned by application teams, cloud and security approvals, vendor contracts, connector licensing, blackout periods and the availability of subject-matter experts.
Migration scopes can add parallel-run, reconciliation and legacy-retirement constraints. Workloads may need to move together because they share data or downstream consumers.
Others cannot move in the same window because the business cannot validate them at once.
Record each material dependency with four fields:
| Dependency | Owner | Required by | Current status or evidence |
|---|---|---|---|
| What must happen outside the delivery team’s control? | Who can make or approve it? | When does it block delivery? | Is it confirmed, assumed or unresolved? |
That small amount of structure is usually more useful than a long risk paragraph with no named owner.
Scope the first production outcome
Discovery turns a broad initiative into a bounded first release. “Build the enterprise lakehouse” or “move customer analytics to Databricks” cannot be estimated or accepted without more detail.
Before discovery
Move customer analytics to Databricks.
After discovery
The first production release will ingest Shopify, CRM and ERP data for marketing and BI reporting. Curated datasets will refresh every two hours and include five years of history. Real-time personalization is out of scope for this phase. CRM API access remains an unresolved dependency owned by the application team. Acceptance will be based on reconciliation of an agreed set of customer, order and campaign reports against the current warehouse.
The second version gives an architect and delivery lead something they can test, estimate and discuss with security and source-system owners. It also tells the sponsor what will not be delivered. The one-line requirement does none of that.
A usable first-release scope normally identifies included workloads, sources, consumers, integrations, security and governance requirements, acceptance criteria, client responsibilities and explicit exclusions. It should also name decisions deferred to later phases.
Where uncertainty cannot be resolved through documents or workshops, add a focused technical validation. Testing an uncertain connector or data format is different from hiding that uncertainty inside the implementation estimate.
Separate facts, assumptions, risks and open questions
No discovery phase removes every unknown. What matters is that the estimate does not present assumptions as confirmed requirements.
- Confirmed facts are supported by documentation, access, samples or an accountable owner.
- Assumptions are being used for planning but have not been verified.
- Open questions need an answer before a decision or build activity.
- Risks could affect scope, schedule, cost or acceptance.
- Dependencies require action or a decision outside the delivery team’s control.
This prevents a familiar estimating problem: a tentative statement made in a workshop is treated as fixed, then returns during delivery as a scope change.
Give each material assumption an owner, a validation method and a deadline. If it cannot be checked before approval, the estimate should explain how it affects confidence or contingency.
What should come out of a Databricks discovery phase?
The outputs should match the engagement. A focused workload may need a concise package; a multi-domain platform initiative will usually need more detail.
Depending on the scope, useful outputs include:
- agreed business objectives and first-release success criteria;
- a stakeholder, decision and ownership map;
- source-system, data and workload inventories;
- functional and non-functional requirements;
- security, privacy and governance requirements;
- an integration and dependency map;
- an initial target-state architecture and a record of unresolved design questions;
- implementation scope, exclusions and prioritized work packages;
- assumptions, risks, issues and dependencies;
- an acceptance approach, delivery outline and responsibility split;
- the basis and confidence level of the estimate.
These need not be separate documents. The practical test is whether sponsors and delivery teams agree on what will be built, why it matters, who owns each part and which uncertainties remain.
The decision gate: when is the scope ready for implementation?
Discovery is complete enough when decision-makers can approve the project without relying on important assumptions that exist only in someone’s notes or memory.
They should be able to identify the first production outcome, included data and workloads, material security and infrastructure constraints, named owners, external dependencies, testable acceptance criteria and explicit exclusions.
Remaining unknowns do not automatically block approval, but each needs an owner and a plan for validation. The estimate should state what it assumes and how confident the team is in it.
This is a decision gate, not a requirement to settle every configuration value or low-level design choice. Detailed architecture and delivery come next.
Once the scope passes that gate, teams can follow the step-by-step Databricks implementation process and use the Databricks implementation checklist to review execution coverage.
How Dateonic can support discovery and implementation
Dateonic can run discovery before taking responsibility for architecture and delivery, or review an existing scope before implementation begins. The engagement can stop with a documented scope and decision record or continue into the build.
If Databricks is already in production and the priority is to diagnose cost, performance, governance or reliability, a Health Check or audit is the better starting point.