Sharat Madanapalli

From Documents to Defensible Evidence: Rethinking RAG for Audits

Audits impose strict requirements for evidence, traceability and expert judgement. Each entity being audited supplies a new body of evidence, made up of manuals, records, tables, diagrams, scans and multilingual documents. Assessment against an audit standard follows defined guidelines and requires precise citations. A plausible answer has little value if an auditor cannot verify its source, understand why the system selected it or correct the result before making a compliance decision.

I will walk through the approaches we tried, what failed, what improved and how those lessons led to a workflow that balances AI assistance with expert judgement. The key shift was to separate evidence selection, compliance assessment and expert review, then measure each stage independently. From that work, I will share a practical method for designing document-based AI systems: define the outcome, measure the factors that matter, understand the task and document constraints, then choose the architecture. I will cover ingestion cost, latency, evidence quality, citation quality and where human review adds the most value. We tested the system against completed audits. It reached up to 94% accuracy on compliance assessments, and auditors approved more than 90% of the evidence mappings they reviewed. The system now runs in production for a firm that conducts more than 70 audits each year. The same method applies to other regulated document workflows where people must verify the evidence before acting.