Key takeaways
- Intelligent document processing (IDP) combines OCR, layout understanding, and ML extraction to turn unstructured documents into structured data.
- Modern systems handle invoices, receipts, contracts, forms, ID cards, medical records, and much more at near-human accuracy on well-defined formats.
- Vision-language models (GPT-4V, Claude 3, Gemini, LayoutLM) have unified what used to be separate OCR + NLP pipelines.
- Primary markets: finance (invoice processing, accounts payable), legal (contract review), insurance (claims), healthcare (medical records), HR (resume parsing).
- ROI is driven by speed and labour savings — invoice processing time drops from days to minutes; error rates drop with continuous learning.
Why document processing is hard
A PDF can be a clean text file, a scanned image, a mixed layout with tables and forms, a handwritten annotation, or all of the above. Documents arrive in hundreds of slightly-different templates from different vendors or jurisdictions. Fields drift over time, formats change, and the data extracted needs to match a downstream system’s schema.
Classical workflows solved this with rules plus manual intervention — regex templates for common formats, human reviewers for the rest. Modern AI collapses the workflow into a single pipeline that handles diversity through learning rather than explicit rules.
The pipeline
1. Input ingestion and classification
Documents arrive as PDFs, scanned images, emails with attachments, or photos taken on a phone. The first step is classifying what kind of document each is — invoice, purchase order, shipping manifest, medical record — to route to the appropriate extraction pipeline.
2. OCR and layout analysis
For non-native-digital documents, OCR extracts text from pixels. Modern OCR (Tesseract, AWS Textract, Google Cloud Document AI, Azure Document Intelligence, proprietary commercial engines) handles printed text, and increasingly handwriting, at high accuracy. Layout analysis identifies structural elements — text blocks, tables, key-value pairs, checkboxes, signatures, stamps.
3. Entity and field extraction
From the structured text + layout, extract the specific fields needed: invoice number, vendor name, line items, total amount, due date. Models like LayoutLMv3 that combine text, positional, and visual features have pushed this substantially.
4. Validation and business rules
Extracted fields are checked against business rules — totals match line-item sums, vendor is recognized, due date is realistic. Violations route to human review.
5. System integration
Extracted structured data flows into downstream systems — accounts payable, ERP, case management, data warehouses. This is often the longest integration step; pipelines that fail to land in the actual system of record do not deliver ROI.
What changed with vision-language models
Before 2023, document AI used specialized models for each step — an OCR engine, a layout model, a separate field extractor. Modern vision-language models (GPT-4V, Claude 3/4’s vision, Gemini, open-source alternatives like LLaVA and Qwen-VL) can ingest a full document image and produce structured output directly from a prompt.
For simple documents, you can send the image to a multimodal LLM with a prompt like “Extract invoice number, vendor, line items, and total as JSON” and get back high-quality results. Specialized models (LayoutLM, Donut, Nougat) still win on cost and latency for high-volume pipelines, but LLMs dominate for flexible or long-tail document types. See our large language models and computer vision primers for the underlying architectures.
Common use cases
Accounts payable automation
The highest-ROI use case by deployment count. Invoices arrive, get extracted, are matched against purchase orders and goods receipts, and routed for approval. Companies like AppZen, Stampli, Bill.com, Tipalti, and integrated ERP offerings (SAP, Oracle, Workday) all ship this.
Know Your Customer (KYC) and identity verification
Onboarding bank customers, verifying fintech users, validating insurance applicants — extract fields from passports, driver’s licenses, tax forms; compare photos; run authenticity checks. Jumio, Onfido, Persona, Alloy, and Sumsub operate in this space.
Contract review
Legal workflows use AI to extract key clauses, flag missing provisions, summarize obligations, and compare versions. Harvey, Evisort, LawGeex, and ContractPodAI target this use case.
Medical records
Extracting structured data from clinical notes, lab reports, imaging reports. Tools like Abridge, Suki, Nuance DAX, Oracle Health’s Clinical AI, and bespoke hospital systems. Particularly valuable for population health analysis, prior authorization, and clinical trial recruitment.
Insurance claims
Claim forms, supporting documents, photos of damage, medical bills — structured extraction feeds claim adjudication. Multiple insurers have deployed this to reduce claim processing time from weeks to days.
Resume parsing and HR
Applicant tracking systems parse resumes into structured candidate profiles. AI resume parsing has existed for 20 years but has dramatically improved with LLMs — handling non-standard formats, extracting skills and experience semantically rather than via keyword matching.
Accuracy and quality control
Well-deployed IDP achieves 90-99% field-level accuracy on common document types, with the specific number depending on document quality, diversity, and field type. Critical fields (amount, ID numbers) typically need higher thresholds than secondary ones.
Human review handles exceptions — fields with low model confidence, documents that don’t match expected templates, validation failures. Feeding human corrections back into model training (a practice called “human-in-the-loop” or active learning) steadily improves the system over time. For NLP fundamentals, see our natural language processing primer.
Things that still break
- Handwriting on complex forms. Modern OCR handles printed text well; handwriting accuracy varies widely by writer and script.
- Tables spanning multiple pages. Detecting continuation and merging tables correctly remains imperfect.
- Low-quality scans. Skew, noise, missing pages, dark scans — garbage in, garbage out.
- Non-English documents. Accuracy is meaningfully lower for less-resourced languages; specific-language OCR and extraction models help where volume justifies.
- Documents the model has never seen. A format that’s only 0.5% of volume may see dramatically lower accuracy than the main formats.
Economic impact
Document processing was one of the most labour-intensive office activities. IDP has meaningfully reduced the need for data-entry clerks and entry-level processing roles across AP, insurance, finance, and HR. ROI for deployments is often fast — a system paying back in under a year is common for high-volume workflows. The displacement is real but uneven across organizations; many have redeployed rather than eliminated roles.
How to start
For teams evaluating IDP: begin with a single document type and workflow where volume is high and format is reasonably stable (invoices are classic). Measure current processing time, error rate, and labour cost. Pilot with a managed service (AWS Textract, Google Document AI, Azure Document Intelligence, or specialized SaaS) before committing to a custom build. Expand scope only after you’ve demonstrated ROI and operational reliability on the first workflow.
Frequently asked questions
Is OCR solved?
For printed English text, essentially yes — commercial OCR engines achieve character-level accuracy above 99% on clean documents. Handwriting, low-resource languages, historical documents, complex layouts, and degraded scans still have significant error rates. “OCR is solved” is true for most enterprise use cases but leaves plenty of research and commercial opportunity in edge cases.
Should I use GPT-4V or a specialized IDP tool?
For small-volume, ad-hoc, or diverse document types — GPT-4V or Claude’s vision capability is often simpler and comparable in quality. For high-volume, latency-sensitive, cost-sensitive production pipelines — specialized IDP tools or dedicated models typically win on operating costs. Many teams use both: the specialized pipeline for bulk processing and LLMs for exceptions and unusual documents.
Can I extract data from scanned PDFs of contracts?
Yes, and it’s a major use case. Contract review tools combine OCR, layout analysis, clause classification, and LLM-based extraction to pull out key terms — parties, effective dates, renewal terms, liability caps, jurisdiction. Accuracy on standard commercial contracts is strong; bespoke or unusual contracts still benefit from human review. Legal teams increasingly treat AI contract extraction as first-pass work that lawyers verify rather than a replacement for lawyer attention.
