Connecting LLMs to Private Knowledge Bases

Connecting a large language model to internal company data is not as simple as sending documents to an API. Enterprise knowledge changes constantly, access differs by user, and sensitive content must not appear in the wrong response.

 

The usual solution is Retrieval-Augmented Generation (RAG). Instead of training proprietary knowledge into the model, the application retrieves relevant passages at query time and adds only the authorized context to the prompt.

 

A production implementation therefore needs to solve four separate problems:

 

• ingest and structure private data;
• keep the search index synchronized with its source;
• enforce document-level authorization before retrieval;
• control where prompts and retrieved context are sent.

 

A Databricks-Native RAG Architecture

 

A practical Databricks flow looks like this:

 

1. Ingest documents from sources such as SharePoint, S3 or ADLS, SQL systems, and internal APIs.
2. Parse and chunk the content while preserving document IDs, source URLs, ownership, classification, tenant, and entitlement metadata.
3. Store the governed source in a Unity Catalog Volume or Delta table.
4. Synchronize the Delta table with a Databricks AI Search index.
5. Authenticate the caller in a trusted RAG service, resolve their entitlements, and inject the required search filters server-side.
6. Send only the authorized passages to an approved model endpoint through the configured serving and gateway path.
7. Trace retrieval and generation with MLflow, including citations, latency, effective filters, and answer-quality metrics.

 

 

The important design principle is that data governance, retrieval authorization, and network isolation are separate controls.

 

Unity Catalog can govern the table and search index as securable objects, but that does not automatically determine which individual rows a user may retrieve.

 

Databricks AI Search, Formerly Vector Search

 

The current product name is Databricks AI Search (formerly Databricks Vector Search). Older articles may also use the name Mosaic AI Vector Search.

 

Some REST paths and infrastructure fields still contain vector-search, but this is legacy API terminology rather than the current product name.

 

Databricks AI Search provides managed serverless search over Delta tables. It supports vector similarity, hybrid keyword and semantic retrieval, filters, reranking, and Delta Sync.

 

For standard endpoints, the source Delta table must have Change Data Feed enabled. Delta Sync can run continuously or on demand in triggered mode.

 

Standard endpoints process incremental changes, while storage-optimized endpoints support triggered synchronization and partially rebuild the index during a sync.

 

Delta Sync reduces custom indexing work and lowers the risk of the search index drifting behind the governed source. It does not remove the need to monitor synchronization failures, deletions, and changes to access metadata.

 

 

Unity Catalog Does Not Automatically Apply RLS to AI Search

 

This is the most important security clarification.

 

The current AI Search documentation states that row-level and column-level permissions are not supported inside the index. Row filters and column masks applied to the source table do not automatically propagate to AI Search.

 

Unity Catalog still controls access to the index itself. A user querying an index they do not own needs USE CATALOG, USE SCHEMA, and SELECT permissions. Endpoint ACLs separately control who can create, use, or manage the AI Search endpoint.

 

Document-level authorization must be implemented explicitly. A typical pattern is to:

 

• include fields such as tenant_id, department, classification, or document ACLs in the indexed metadata;
• authenticate the user in the application;
• resolve entitlements from a trusted identity or authorization service;
• construct the AI Search filter server-side;
• reject the request when required entitlement data is missing.

 

Do not accept a department, tenant ID, or classification directly from the user’s prompt or request body. For workloads requiring a stronger isolation boundary, use separate indexes, schemas, workspaces, or accounts instead of relying only on metadata filters.

 

Minimal Databricks AI Search Implementation

 

Assume a Unity Catalog table named main.rag.private_document_chunks with these columns:

 

• chunk_id — unique primary key;
• document_id — stable source-document identifier;
• chunk_text — text used to compute embeddings;
• department and classification — authorization metadata.

 

The following notebook example creates a standard endpoint and triggered Delta Sync index. Endpoint and index creation should normally be managed as one-time deployment operations.

 

%pip install databricks-ai-search
dbutils.library.restartPython()

from databricks.ai_search.client import AISearchClient

client = AISearchClient()

ENDPOINT = "private-kb-search"
SOURCE = "main.rag.private_document_chunks"
INDEX = "main.rag.private_document_chunks_idx"

client.create_endpoint(
    name=ENDPOINT,
    endpoint_type="STANDARD",
)

# Wait until the endpoint is online before creating the index.
index = client.create_delta_sync_index(
    endpoint_name=ENDPOINT,
    source_table_name=SOURCE,
    index_name=INDEX,
    pipeline_type="TRIGGERED",
    primary_key="chunk_id",
    embedding_source_column="chunk_text",
    embedding_model_endpoint_name="databricks-qwen3-embedding-0-6b",
    columns_to_sync=[
        "document_id",
        "department",
        "classification",
    ],
)

index.sync()

 

The query filter must come from trusted application logic, not from the end user’s request:

 

index = client.get_index(
    endpoint_name=ENDPOINT,
    index_name=INDEX,
)

# Populate this after authenticating the caller and resolving entitlements.
allowed_departments = ["finance"]

if not allowed_departments:
    raise PermissionError("No knowledge-base entitlement")

results = index.similarity_search(
    query_text="What is the policy for approving expenses?",
    columns=[
        "document_id",
        "chunk_text",
        "department",
        "classification",
    ],
    filters={
        "department": allowed_departments,
        "classification": ["internal"],
    },
    query_type="hybrid",
    num_results=5,
)

 

In production, use a service principal with OAuth, wait for provisioning and synchronization states, add retries and timeouts, preserve source URLs for citations, and log the effective authorization filter used for every retrieval.

 

What Private Networking Really Means

 

A private knowledge base does not automatically create a private network path. Databricks AI Search is a serverless service.

 

Databricks documents logical isolation, authentication and authorization, encryption at rest, and encryption in transit, but this is not the same as running the search service inside the customer’s VPC.

 

Private connectivity from Databricks serverless workloads to resources in a customer VPC requires explicit configuration for supported routes, such as a Network Connectivity Configuration with PrivateLink.

 

Availability and behavior depend on the cloud, region, service, and destination.

 

The model path must be assessed separately. If the application calls OpenAI, Anthropic, or another external provider, that provider receives the prompt and retrieved context under the configured contract and network path.

 

Routing the request through Unity AI Gateway adds governance, but it does not turn an external API call into an in-perimeter invocation.

 

Map every hop before making a claim that data “never leaves the private environment”: the source system, ingestion compute, Delta storage, AI Search, RAG service, model endpoint, logging destination, and evaluation tools.

 

Conclusion

 

Connecting an LLM to a private knowledge base requires more than a vector index and an API call.

 

A reliable architecture combines governed ingestion, synchronized retrieval, explicit document authorization, controlled model access, a reviewed network path, citations, tracing, and continuous evaluation.

 

Databricks provides an integrated foundation through Unity Catalog, Databricks AI Search, Delta Sync, Model Serving, Unity AI Gateway, and MLflow.

 

The platform reduces integration work, but access filters and network boundaries still need to be designed and tested deliberately.

 

Ready to productionize your AI architecture? Contact Dateonic’s Engineering Team to scope a production-ready enterprise RAG implementation on Databricks.