Why RAG Will Make You Sad (In Law)
Three lossy steps between your question and the right answer—and why legal documents demand a fundamentally different approach.
Traditional RAG (Retrieval-Augmented Generation) treats legal documents like blog posts. They are not. A legal document is a precision instrument where structure, context, and exact language carry binding force. RAG introduces three compounding lossy steps—chunking, embedding, and similarity search—each of which discards information that is critical to legal interpretation. The result: confident, well-formatted, plausible answers that are substantively wrong. The alternative is inference-over-inference—using LLM reasoning at every stage instead of lossy mathematical transformations. It costs more. It's the price of getting it right.
The Three Lossy Steps Problem
Most legal AI tools rely on RAG: chunk the document, embed the chunks into vectors, then search those vectors for similarity to your query. Each step is a lossy compression. In most domains, the information loss is tolerable. In law, it's catastrophic.
Chunking
When you chunk a contract, a statute, or a brief, you make arbitrary cuts through a document that was drafted as an integrated whole. A definition in Section 1 governs a liability clause in Section 14. A "notwithstanding" in paragraph (b) modifies everything in paragraph (a). Chunking severs these relationships. In law, severing context doesn't just reduce quality—it can invert meaning.
Embedding
Embedding compresses semantic meaning into a fixed-dimensional vector. This is a mathematical projection—it necessarily discards information. Legal language is adversarially precise. The difference between "shall" and "may," between "and" and "or," between "reasonable efforts" and "best efforts"—these distinctions can represent millions of dollars in liability. An embedding treats them as roughly similar. They are not.
Similarity Search
Now you search for cosine similarity within those degraded representations of those severed chunks. You find what sounds like your query, not what legally governs your question. Similarity is computed within the chunk's embedding space, so you've lost the document's structural hierarchy entirely. You can't distinguish a holding from dicta, a rule from an exception, a defined term from its colloquial usage.
The Compounding Effect
These aren't three independent losses—they compound. You're taking a lossy representation of a lossy slice found through a lossy search. Each step degrades signal, and in law, the signal that gets degraded first is precisely the signal that matters most: precision, structure, and contextual dependency.
You can't build a precision tool on a foundation of approximation.
The Real Danger: Confident Wrong Answers
A RAG system won't tell you it missed the exception to the rule. It won't flag that the chunk it retrieved was from a dissent, not the majority opinion. It won't know that the definition it found was superseded by an amendment three sections later.
It will return a confident, well-formatted, plausible answer that is substantively wrong—the most dangerous kind of error for an attorney who might rely on it.
This isn't a theoretical concern. This is the scenario that gets lawyers sanctioned. The semantic false positive—a real answer that doesn't actually support your argument—passes every check except reading the source material. And RAG pipelines are architecturally designed to skip that step.
Where RAG Works—and Where It Doesn't
RAG works fine for "find me articles about X." It is dangerously inadequate for "what are my client's obligations under Section 4.2(b), subject to the exceptions in Section 7, as modified by Amendment 3?"
The difference is precision under complexity. The first query tolerates approximation. The second requires exact comprehension of cross-referencing relationships across the full document. Every step in the RAG pipeline is optimized for the first kind of query. Legal work is the second kind.
The Alternative: Inference Over Inference
The alternative is fundamentally different. Instead of converting documents into degraded mathematical representations, use LLM inference at every stage—the same kind of reasoning a lawyer applies when reading a document.
Understand the Document's Architecture
Let the LLM read and understand the document's architecture. Build a semantic map of how sections relate, what governs what, where definitions live, how exceptions modify rules. This is an inference task, not an embedding task.
Determine What Matters and Why
Given the user's actual question—their case, their filing, their issue—use LLM inference to determine which parts of the document matter and why. This isn't similarity search. It's legal reasoning about relevance.
Reason Over Actual Text
Feed the LLM the actual text within those intelligently selected windows, with structural context preserved. Let it reason over the real language with full awareness of how it fits into the document.
Yes, this means N LLM calls per document. It's more expensive. But you're doing what a lawyer actually does: reading the document with understanding, identifying what's relevant with judgment, and analyzing the text with precision.
The legal profession's entire value proposition is precision under complexity. Any AI pipeline that introduces lossy steps at the point of document comprehension is architecturally incompatible with that value proposition.
The Cost Argument
The immediate objection: inference-over-inference is more expensive per document. Multiple LLM calls instead of one embedding and one retrieval. This is true.
But consider what you're paying for. RAG optimizes for cost per query. Inference-over-inference optimizes for accuracy per question. In legal work, the cost of a wrong answer—sanctions, malpractice exposure, missed obligations—vastly exceeds the cost difference between a vector lookup and an LLM call.
The question isn't "how much does each query cost?" The question is "what does it cost when the answer is wrong?" For a blog search engine, a wrong result means a disappointed reader. For a legal analysis tool, a wrong result means a lawyer relying on authority that doesn't exist, obligations that were superseded, or exceptions that were severed by chunking. The cost difference is the price of getting it right.
What This Means for Legal AI Tools
When evaluating legal AI products, the architecture matters as much as the interface. Ask how the system processes your documents:
- Does it chunk your documents? If yes, cross-references between sections will be lost. A definition in Section 1 that governs Section 14 may never appear in the same context window.
- Does it use vector embeddings for retrieval? If yes, the precision distinctions that define legal language—"shall" vs. "may," "reasonable" vs. "best"—are being compressed into approximate similarity scores.
- Does it use similarity search to find relevant passages? If yes, it's finding what sounds like your question, not what answers your question. It cannot distinguish a holding from a rejected argument.
- Does it preserve document structure during analysis? If not, the architectural relationships that give legal text its meaning—cross-references, definitions, exceptions, amendments—are invisible to the system.
These aren't edge cases. They're the normal structure of legal documents. Any tool that discards them is optimized for a use case that isn't yours. (For a comprehensive framework for evaluating legal AI tools, see our AI vendor due diligence checklist.)
Conclusion
RAG is a powerful architecture for many applications. It is the wrong architecture for legal document analysis. The precision, structure, and cross-referencing relationships that define legal documents are exactly the information that RAG's three lossy steps discard.
The alternative—inference at every stage—is more expensive per query. But law isn't a domain where "approximately right" is acceptable. Approximately right, delivered with confidence, is the most dangerous possible output for an attorney.
The cost difference is the price of getting it right.
FAQ: RAG and Legal AI Architecture
Why is RAG bad for legal document analysis?
RAG introduces three compounding lossy steps—chunking, embedding, and similarity search—each of which discards information critical to legal interpretation. Chunking severs cross-references between sections. Embedding compresses adversarially precise language into approximate vectors. Similarity search finds what sounds like your query, not what legally governs your question.
What is the problem with chunking legal documents?
Legal documents are integrated instruments where a definition in Section 1 may govern a liability clause in Section 14, and a "notwithstanding" in one paragraph modifies everything in another. Chunking makes arbitrary cuts through these relationships. In law, severing context doesn't just reduce quality—it can invert meaning entirely.
What is the alternative to RAG for legal AI?
The alternative is inference-over-inference: using LLM reasoning at every stage instead of lossy mathematical transformations. Stage 1 uses inference to understand a document's architecture. Stage 2 uses inference to determine which parts matter for your question. Stage 3 uses inference to analyze the actual text with structural context preserved. This is more expensive per query but dramatically more accurate.
Why can't vector embeddings capture legal precision?
Embedding compresses semantic meaning into a fixed-dimensional vector—a mathematical projection that necessarily discards information. Legal language is adversarially precise: the difference between "shall" and "may," between "reasonable efforts" and "best efforts," can represent millions of dollars in liability. An embedding treats these as roughly similar. They are not.
AI Built on Inference, Not Approximation
inCamera analyzes your documents with full structural context—no chunking, no embeddings, no lossy shortcuts. Zero data retention. Direct document analysis.