Building RAG systems for compliance: lessons from PolicyAI

Introduction

Retrieval-Augmented Generation (RAG) has become the default architecture for AI-powered document question-answering. The pattern is well-understood: chunk your documents, embed them, store them in a vector database, retrieve the most relevant chunks at query time, and pass them to an LLM with the user's question.

What the tutorials usually skip is what happens when your documents are sensitive, access-controlled, and used as audit evidence.

Generic RAG tutorials assume a single pool of documents, a single user, and no particular consequences for a wrong answer. Enterprise compliance is the opposite of all three: role-based document access, multiple user tiers with strict separation, and a regulatory environment where an incorrect or unattributed answer can mean a fine, a failed audit, or a legal liability.

Building PolicyAI at CoralShades surfaced a set of requirements that generic RAG implementations simply don't handle. This post covers three of the most important ones: role isolation via PostgreSQL Row-Level Security, source citations for audit compliance, and a multi-provider LLM strategy for resilience and data sovereignty.

Why compliance RAG is different

Start with the threat model. In a standard RAG system, the concern is retrieval quality: are you pulling the right chunks? In a compliance system, you have an additional concern: are you pulling from the right documents at all?

PolicyAI gives each user role a different subset of the policy library. For example, an Executive asking about remuneration policy should not see Board-level compensation documents. An Administrator handling HR procedures should not see Executive-level strategic plans.

This isn't a UI concern. It can't be solved by hiding a button or filtering a list. The access control must be enforced at the data layer, because if a sophisticated user can bypass the UI and query the vector store directly, role separation must still hold.

Three requirements that compliance adds on top of standard RAG:

1. Role isolation at the database level: not enforced in application code, which can have bugs and be bypassed
2. Mandatory source citations: every AI-generated answer must be traceable to a specific document, specific page, for audit purposes
3. Document lifecycle awareness: policies older than 18 months need to be flagged, because acting on an outdated policy carries compliance risk

Role isolation with PostgreSQL RLS

Supabase's Row-Level Security (RLS) provides the mechanism. RLS policies live at the database level and are enforced for every query, regardless of which application layer sends it. Even if a bug in the application code skips a filter, the database enforces the rule.

The schema design, in outline:

• documents table: stores document metadata and the role assignment
• document_chunks table: stores embedded chunks with a foreign key to the parent document
• user_roles table: stores each user's assigned role

The RLS policy on document_chunks enforces that users can only retrieve chunks from documents assigned to their role:

-- Enable RLS on document_chunks
ALTER TABLE document_chunks ENABLE ROW LEVEL SECURITY;

-- Policy: users can only read chunks from documents they have access to
CREATE POLICY "role_based_chunk_access" ON document_chunks
  FOR SELECT
  USING (
    EXISTS (
      SELECT 1
      FROM documents d
      JOIN user_roles ur ON ur.user_id = auth.uid()
      WHERE d.id = document_chunks.document_id
        AND d.assigned_role = ur.role
    )
  );

This policy means that when the RAG pipeline runs a similarity search against document_chunks, Supabase automatically filters the results to the current user's permitted documents. The application doesn't need to know the user's role, the database enforces it.

The critical consequence: the embedding similarity search only returns chunks the user is permitted to see. The LLM never receives context from out-of-scope documents. Role isolation holds at the retrieval layer, not just the presentation layer.

Source citations for audit compliance

Generic RAG implementations often include document titles in the response. Compliance needs more: page-level citations, traceable to the exact source, includable in an audit trail.

This requires storing citation metadata at the chunk level, not just the document level. During document ingestion, each chunk is stored with:

• Document ID and title
• Page number where the chunk originated
• Section heading (if extractable)
• The chunk's position within the document

# During document ingestion, store citation metadata with each chunk
async def ingest_document_chunk(
    text: str,
    document_id: str,
    page_number: int,
    section_heading: str | None,
    chunk_index: int,
    embedding: list[float],
) -> None:
    await supabase.table("document_chunks").insert({
        "document_id": document_id,
        "content": text,
        "embedding": embedding,
        "page_number": page_number,
        "section_heading": section_heading,
        "chunk_index": chunk_index,
        "created_at": datetime.utcnow().isoformat(),
    })

At query time, the retrieved chunks carry their citation metadata. The LLM prompt instructs the model to include citations in its response, and the application layer formats them for display:

# Query with citation-aware prompt
async def generate_cited_answer(
    question: str,
    retrieved_chunks: list[dict],
) -> dict:
    # Format context with inline citation markers
    context_with_citations = "\n\n".join([
        f"[Source: {chunk['document_title']}, Page {chunk['page_number']}]\n{chunk['content']}"
        for chunk in retrieved_chunks
    ])

    prompt = f"""Answer the following question using only the provided policy documents.
Include a citation for each claim in the format [Document Title, Page N].
If the answer cannot be found in the provided documents, say so explicitly.

Context:
{context_with_citations}

Question: {question}"""

    response = await llm_client.complete(prompt)

    return {
        "answer": response.text,
        "citations": [
            {
                "document_id": chunk["document_id"],
                "document_title": chunk["document_title"],
                "page_number": chunk["page_number"],
            }
            for chunk in retrieved_chunks
        ],
    }

The resulting answer includes structured citation data alongside the text. The frontend renders these as clickable references that open the source document at the relevant page. An auditor reviewing a compliance decision can trace exactly which policy document was used, at which page, to generate the answer.

Multi-provider LLM strategy

Running a compliance system on a single LLM provider introduces two categories of risk that regulated organisations cannot accept:

Availability risk. If OpenAI has an outage, your compliance system is down. For a system used to answer time-sensitive policy questions before meetings or regulatory submissions, that is not acceptable.

Data sovereignty risk. Some organisations, particularly government-adjacent or regulated financial entities, have constraints on which providers can process their data. A system that only works with OpenAI cannot be deployed in those environments.

PolicyAI handles this with a configurable, per-organisation LLM provider strategy. The sketch below shows the shape of it:

# Per-organisation LLM configuration
class OrganisationLLMConfig:
    primary_provider: str  # "openai" | "gemini" | "azure_openai"
    primary_model: str
    fallback_provider: str | None
    fallback_model: str | None
    data_residency_region: str | None  # e.g., "eu-west" for GDPR

async def get_llm_client(org_id: str) -> LLMClient:
    config = await load_org_config(org_id)

    try:
        return LLMClient(
            provider=config.primary_provider,
            model=config.primary_model,
        )
    except ProviderUnavailableError:
        if config.fallback_provider:
            return LLMClient(
                provider=config.fallback_provider,
                model=config.fallback_model,
            )
        raise

The fallback logic is transparent to the rest of the application. The RAG pipeline calls get_llm_client(org_id) and gets a client, it doesn't need to know whether that client is pointing at OpenAI or Gemini.

Per-organisation configuration means each customer can choose their preferred provider based on their own data sovereignty requirements. A government agency can point to an Azure OpenAI deployment in their approved region. A startup can use the default OpenAI configuration. Both use the same application code.

The multi-provider approach also enables cost optimisation: route bulk document summarisation tasks (which need volume but not the highest quality) to a cheaper model, while routing compliance Q&A (where answer quality matters most) to the highest-capability model.

Conclusion

RAG for compliance is not technically harder than RAG for general search. The chunking, embedding, and retrieval steps are identical. What changes are the operational requirements that surround the retrieval: who can see which documents, whether answers are traceable, and whether the system remains available when a specific provider has an incident.

The patterns in this post (PostgreSQL RLS at the retrieval layer, page-level citation metadata, configurable multi-provider LLM routing) address those operational requirements directly. They are not polishing; they are the minimum viable architecture for a system that will be used to support real compliance decisions.

Generic RAG is a proof of concept. Compliance RAG is a production system with a threat model.