GenAI Contract Review for a Top 5 Global Law Firm: 94% Accuracy Across 3 Million Documents in 16 Weeks

3,800 Fee Earners, 3 Million Documents, and a $240M Annual Contract Review Cost Base

The firm had 3,800 fee earners across 46 offices in 24 jurisdictions. Contract review and due diligence work represented approximately 31% of total fee-earner hours, with M&A due diligence and large commercial transaction support as the two highest-volume practice areas. The firm estimated this translated to an annual cost of approximately $240 million when fully loaded, and clients were increasingly challenging the hourly billing model for document review work that was visible, repetitive, and deliverable on a fixed-fee basis by competitor firms deploying GenAI tools.

The Managing Partner's Office had reviewed four commercially available legal AI products over the preceding 18 months. None had been deployed in production. Three had been rejected at the accuracy validation stage: extracted clause information contained errors at rates that partners considered commercially unacceptable for client-facing work. The fourth had been rejected at the information security review because the SaaS model required uploading client documents to the vendor's cloud infrastructure, which conflicted with the firm's client confidentiality obligations.

The firm needed a deployment that was private-by-design, attorney-validated for accuracy, and capable of handling the range of document types, jurisdictions, and matter types that a global M&A practice encounters. Off-the-shelf tools had failed on accuracy. Infrastructure constraints ruled out SaaS. The solution had to be built and operated within the firm's own environment.

Why Legal GenAI Accuracy Is Harder Than It Appears in Demos

The four commercially available tools the firm had evaluated all demonstrated impressive accuracy rates in vendor demonstrations. All four performed materially worse on the firm's actual work. Understanding why required a diagnostic analysis of the failure modes, which shaped the entire architecture of the solution we designed.

• Out-of-distribution document types: Vendor tools are typically trained and benchmarked on clean, structured commercial contract forms. The firm's M&A due diligence work involved assignment agreements, vendor contracts, IP licenses, employment agreements, and regulatory filings in 17 jurisdictions, many in non-standard formats, partially completed, or hand-annotated. Accuracy rates that vendors achieved on clean commercial leases and NDAs dropped by 25 to 40 percentage points on this document mix.
• Jurisdictional variation in standard clause language: A limitation of liability clause in a UK law agreement uses different standard language than the same clause under New York law, Delaware law, Singapore law, or German law. A model trained primarily on US documents applies US-centric interpretation heuristics to non-US agreements, producing systematic errors on cross-border matters.
• Hallucination in summarization tasks: All four vendor tools were observed generating plausible-sounding but inaccurate summaries of clause content on a non-trivial proportion of test documents. For due diligence work where clients rely on extracted clause summaries to make acquisition decisions, a 3% hallucination rate is not an acceptable baseline, regardless of what the vendor's benchmark accuracy figures show.
• Private infrastructure requirement: The firm's IT security policy required that all client documents remain within the firm's own infrastructure. Any architecture relying on external API calls for inference was non-compliant. The solution needed to run entirely on-premises on the firm's existing server infrastructure.

A Fine-Tuned On-Premises LLM with a Retrieval-Augmented Verification Layer

We designed a system built on three components working together: a fine-tuned large language model deployed on-premises, a retrieval-augmented generation architecture for clause extraction, and a verification layer designed specifically to suppress hallucination by requiring evidence grounding for every extracted claim.

Fine-Tuned LLM on Proprietary Legal Dataset. Rather than using a general-purpose LLM out of the box, we fine-tuned a 70-billion parameter base model on a dataset of 280,000 annotated legal documents assembled from the firm's historical matters (fully anonymized and with client consent obtained under the firm's matter closure process). The fine-tuning dataset was deliberately constructed to represent the full range of document types, jurisdictions, and practice areas that appeared in the firm's work, with annotations provided by 24 senior associates and partners serving as subject-matter experts. This produced a model with domain knowledge of the firm's specific document population rather than generic legal knowledge.

RAG Architecture with Clause-Level Indexing. For each document processed, the system built a clause-level vector index using the firm's matter management system as the retrieval store. Clause extraction queries were executed against this index before being passed to the LLM, ensuring that every extracted clause was grounded in a specific, locatable document passage with page and paragraph references. This architecture served two purposes: reducing hallucination by anchoring LLM outputs to retrieved evidence, and providing attorneys with a direct citation for every extracted item enabling fast review and verification.

Confidence-Scored Output with Mandatory Human Review Thresholds. Every extracted item was assigned a confidence score based on semantic similarity between the LLM output and the retrieved source passage. Items with confidence scores above 0.92 were presented as high-confidence extractions requiring review but not full re-reading. Items below 0.80 were flagged as low-confidence requiring full attorney review. Items between 0.80 and 0.92 were presented with highlighted source text for quick verification. This architecture meant that attorneys' review time was concentrated on genuinely uncertain items rather than uniformly distributed across all extracted clauses.

Jurisdiction-Specific Clause Libraries. We built clause interpretation libraries for 17 key jurisdictions covering standard formulations, market norms, and jurisdiction-specific risk flags for 84 standard clause types. These libraries were maintained by the firm's knowledge management team and served as both fine-tuning reference material and as context injected into the RAG pipeline for jurisdiction-specific queries.

On-Premises Infrastructure Deployment. The entire system was deployed on the firm's existing on-premises GPU cluster using a containerized architecture that isolated client matter data at the matter level. The legal operations and IT security teams validated the architecture against the firm's information security policy before deployment. No client data left the firm's own infrastructure at any point in the processing pipeline.

16 Weeks from Architecture Approval to Firm-Wide Production