AI Document Processing: OCR, a Vision Model, or Both

I spent today adding Tesseract OCR to Heym's Converter node, which means I spent today reading a lot of documents that machines could not read. Somewhere around the fourth scanned invoice I remembered why this category has such a strange reputation: the tools mostly work, and people still describe them as broken. The gap between those two facts is almost always the same misunderstanding, and almost nobody writing about AI document processing bothers to name it.

Here it is. Your PDF parser is probably not broken. Your PDF is probably a stack of photos.

AI document processing is the practice of turning documents that were written for humans into structured data a system can use: recognizing the characters on the page, pulling named fields out of them, checking the result, and passing it somewhere useful. The interesting decisions in 2026 are not about whether to use AI. They are about which kind, at which step, and how you find out when it was wrong.

This guide covers both halves. An AI document processing pipeline is only four steps, and the hard part is choosing what runs at step two. First the decision: when a deterministic OCR engine beats sending the page to a multimodal model, and when it does not. Then the build: the exact node chain, with a template you can copy.

TL;DR: Check whether the PDF has a text layer before anything else. If it does not, recognize the pixels first. Deterministic OCR is cheap, repeatable, and fails visibly with garbled characters; a vision model is flexible, costs more per page, and fails invisibly with confident wrong values. Recent benchmarks disagree on which is more accurate, but agree that keeping OCR in the pipeline gives you better calibrated confidence, which is what makes routing to a human meaningful. Recognize with OCR, extract with a schema-constrained model, validate the result, and send the failures to a person.

What AI Document Processing Actually Means

Definition: AI document processing (also called intelligent document processing, or IDP) is the automated conversion of unstructured documents into structured, validated data, combining character recognition, field extraction, and verification into one pipeline.

The vocabulary in this space is muddled enough to be worth untangling, because vendors use three words for what are really three different jobs.

Most of the frustration people report comes from expecting one of these to do another one's job. A parser cannot recover text that was never stored as text. An extraction model cannot tell you it misread a digit, because from its perspective it did not misread anything.

The demand side of this is not subtle. Search for how to do this and the top results are almost entirely product pages. Search the discussions and you find people asking the same practical questions over and over, unanswered. One thread on r/automation about PDF extraction tools contains a question that has sat there without a real reply: a user describes a PDF built by combining photos into a single document, notes that the text cannot be selected because it was never OCR-scanned, and asks which software handles that and whether it has an API. That is the whole category in one comment. The question is basic, correct, and unserved.

Does Your Document Have a Text Layer?

This is the first question, and answering it wrong wastes more time than any other mistake in this pipeline.

A PDF is a container format, not a text format. It can hold real character data, in which case you can select a word with your cursor and copy it. It can also hold nothing but a rendered image of a page, in which case selecting does nothing, because there is no text to select. Scanners produce the second kind. So do phone cameras, fax gateways, and anyone who prints to PDF from a photo.

Three ways to tell, in order of speed:

1. Try to select a line. If your cursor highlights a rectangle instead of words, or highlights nothing at all, there is no text layer.
2. Search inside the document for a word you can plainly see. No match means the word exists as pixels only.
3. Run it through a parser and look at the output length. Zero characters from a visibly full page is diagnostic, not mysterious.

So when people ask how to extract data from a scanned PDF, the honest first answer is that you cannot, not yet. You have to recognize it first. If there is no text layer, no amount of parser tuning will help, and a lot of the "this tool does not work" reviews in this category come from people who never got past this point. You need recognition. That is what OCR is for, and it is the step the vendor landing pages tend to hide behind the word "AI".

If there is a text layer, you have a much cheaper problem. Extract the text directly and skip recognition entirely. Every page you send through OCR that did not need it costs you time you did not have to spend.

OCR, a Vision Model, or Both

Here is the decision the SERP does not make for you. Once you know the page needs recognition, you have two families of tool, and they are not interchangeable.

A classical OCR engine like Tesseract detects character shapes and maps them to characters. It runs locally, costs nothing per page beyond CPU time, and produces the same output for the same input every single time. A multimodal vision model looks at the page the way a person does, reading layout and context together, and can answer questions about what it sees rather than just transcribing it.

That last row deserves its own section, because it is the one that decides how you design everything downstream.

The Failure Modes Are Opposites

When Tesseract cannot read something, you get 1nvo1ce T0ta1: 4S8.0O. It is wrong, it is obviously wrong, and a regular expression that expects a number will reject it. The failure announces itself.

When a vision model cannot read something, you get {"invoiceTotal": 458.00}. Clean, well-formed, confidently typed, and possibly invented. The model's job is to produce plausible output, and a plausible number is exactly what it produces when the pixels are ambiguous. This has a name in the literature now. He and colleagues introduced KIE-HVQA in 2025, describing it as the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding, built on exactly the document types where this hurts most: identity cards and invoices.

Key principle: A pipeline that fails loudly is cheaper to operate than one that fails quietly, even when the loud one is wrong more often. You can write a check for garbled text. You cannot write a check for a confident lie.

What the Recent Benchmarks Actually Say

It would be convenient if the research settled this. It does not, and the way it fails to settle it is the most useful thing in this article.

In March 2026, Shen and colleagues published "OCR or Not? Rethinking Document Information Extraction in the MLLMs Era", a large-scale benchmark on business documents. Their conclusion is blunt: "OCR may not be necessary for powerful MLLMs, as image-only input can achieve comparable performance to OCR-enhanced approaches." On raw extraction accuracy, feeding the image straight to a strong model held its own.

On 3 August 2026, five months later, Roy and colleagues published ConfBench, the first calibration-specific benchmark for key information extraction, built by applying 20 controlled degradation pipelines to a document set to produce 1,346 variants and over 70,000 entity-level evaluations. Their framing of the problem is the part worth quoting: intelligent document processing "hinges on confidence scores trustworthy enough to route extractions between automation and human review." And their first finding is that "OCR+Image modality results in more accurate confidence estimates."

Notable fact: ConfBench had to manufacture failure in order to study it. The authors applied 20 controlled degradation pipelines to a clean document set precisely because existing document benchmarks are "dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment." The documents your pipeline actually struggles with are the ones the field has barely measured.

Read together, these are not contradictory. They are measuring different properties, and the distinction is the one nobody in this category makes:

• Accuracy asks whether the extracted value is right.
• Calibration asks whether the system's confidence in that value can be trusted.

A pipeline can be accurate and badly calibrated at the same time, which is the worst combination in production, because it is right often enough that you stop checking and wrong occasionally in ways you cannot predict. If you are building something where every document gets human eyes anyway, image-only is a defensible simplification. If you are building something where the whole point is that most documents go through untouched and the uncertain ones get flagged, then the thing you are optimizing is not accuracy. It is the quality of the flag. And that is the property the newer benchmark associates with keeping OCR in the loop.

So the honest answer to "OCR or a vision model" is usually both, in that order: recognize deterministically, then reason over the recognized text, and keep the image available for the cases that need a second look.

Building It, Node by Node

Enough theory. Here is the actual chain in Heym, our open-source, self-hosted AI workflow automation platform, using the Converter node I shipped today.

Step 1: Get the file into storage

File conversions read from Heym Drive rather than from a path or a URL, so the document has to be stored before anything can read it. There are three ordinary ways in:

• A File upload trigger, which mints a single-use upload URL when the workflow is invoked and runs the body synchronously once the file arrives.
• A Drive node with downloadUrl, which pulls a remote file into storage first.
• An Agent that generated the file itself, exposed as $reportAgent._generated_files[0].id.

Step 2: Recognize the pages

Add a Converter node, choose pdfToText for a PDF or imageToText for a single image, and point converterFileId at the upstream file:

{
  "type": "converter",
  "data": {
    "label": "readInvoice",
    "conversion": "pdfToText",
    "converterFileId": "$Upload.file.id",
    "ocrLanguage": "auto",
    "ocrDpi": 300,
    "ocrPageRange": "1-3"
  }
}

ocrLanguage: auto runs Tesseract's orientation-and-script detection first and then picks the best installed model for whatever script it found. Script models cover every language written in that script, which is why auto handles a Turkish invoice and an English one without being told which is which. Naming the language explicitly is still more accurate when you know it, and you can join several with a plus sign (eng+tur) for genuinely mixed documents, at the cost of speed and a little extra noise.

The node returns more than the text. $readInvoice.result is the full recognized text, $readInvoice.pages is an array of { page, text } for per-page handling, and $readInvoice.language tells you which model auto actually chose, which is the field you will want the first time a result looks strange.

If you would rather start from something already published, the Tesseract OCR PDF to text template is the minimal three-node version of this step: upload, recognize, return page-level results.

Step 3: Turn text into fields

Now the model, and this is where the schema matters. Feed the recognized text into an LLM or Agent node with jsonOutputEnabled on. A document processing AI agent is useful here when the field set varies between document types and you want the model to pick which schema applies; a plain LLM node is the better choice when every document is the same shape. Either way, give it a schema that names every field you expect. Without one you get prose that you then have to parse, which reintroduces exactly the fragility you were trying to remove.

The important design choice here is that the model receives text, not the image. Text tokens are cheaper than image tokens, the payload is smaller, and the expensive reasoning step runs once per document rather than once per page. That is the cost structure the "just send it all to a multimodal model" approach quietly gives up, and it compounds fast at volume, which is why it belongs in any serious agent cost conversation.

Step 4: Validate before you trust

Add a Condition node and check the things a model cannot get right by luck:

• Are all required fields present and non-empty?
• Do the line items sum to the stated total?
• Is the invoice date inside a plausible range rather than in 1970 or 2087?
• Does the vendor name match something you have seen before?

Arithmetic is the highest-value check in this whole pipeline, because a model that misread a digit will produce a total that does not reconcile, and reconciliation is a deterministic test. This is the same guardrail thinking that applies to any agent output, applied to the specific shape of document data.

Step 5: Send the failures to a person

Anything that fails validation goes to a human review gate, not back through the model. Retrying is tempting and usually wrong: a model that guessed badly once will typically guess the same way again, and you will have paid twice for the same error. A queue of flagged documents that a person clears in the morning is dramatically cheaper than a wrong number written into an accounting system.

{
  "heym": true,
  "nodes": [
    {
      "id": "doc_setup_note",
      "type": "sticky",
      "position": {
        "x": 360,
        "y": 540
      },
      "data": {
        "label": "SetupNote",
        "note": "### Before you run\n- Open ExtractFields and pick your own LLM credential.\n- ocrLanguage auto detects the script. Set a code like tur when you know the language.\n- Narrow ocrPageRange on long documents. Every selected page is rasterized and read.\n- The Condition node is the point of this workflow. Do not remove it."
      }
    },
    {
      "id": "doc_upload",
      "type": "fileUploadTrigger",
      "position": {
        "x": 60,
        "y": 240
      },
      "data": {
        "label": "Upload",
        "ttlMinutes": 60,
        "allowedTypes": "application/pdf"
      }
    },
    {
      "id": "doc_read",
      "type": "converter",
      "position": {
        "x": 380,
        "y": 240
      },
      "data": {
        "label": "ReadDoc",
        "conversion": "pdfToText",
        "converterFileId": "$Upload.file.id",
        "ocrLanguage": "auto",
        "ocrEncoding": "utf-8",
        "ocrDpi": 300,
        "ocrPageRange": "1-3"
      }
    },
    {
      "id": "doc_extract",
      "type": "llm",
      "position": {
        "x": 720,
        "y": 240
      },
      "data": {
        "label": "ExtractFields",
        "model": "gpt-5.5",
        "systemInstruction": "You extract invoice fields from recognized text. Return only JSON matching the schema. When a value is not clearly present in the text, return null for it rather than guessing. Never infer a number that is not written on the page.",
        "userMessage": "$ReadDoc.result",
        "jsonOutputEnabled": true,
        "jsonOutputSchema": "{\"type\":\"object\",\"properties\":{\"vendor\":{\"type\":[\"string\",\"null\"]},\"invoiceNumber\":{\"type\":[\"string\",\"null\"]},\"invoiceDate\":{\"type\":[\"string\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"lineItemsTotal\":{\"type\":[\"number\",\"null\"]},\"statedTotal\":{\"type\":[\"number\",\"null\"]}},\"required\":[\"vendor\",\"statedTotal\"]}"
      }
    },
    {
      "id": "doc_validate",
      "type": "condition",
      "position": {
        "x": 1060,
        "y": 240
      },
      "data": {
        "label": "LooksTrustworthy",
        "condition": "$ExtractFields.json.statedTotal !== null && $ExtractFields.json.vendor !== null && $ExtractFields.json.lineItemsTotal === $ExtractFields.json.statedTotal"
      }
    },
    {
      "id": "doc_store",
      "type": "output",
      "position": {
        "x": 1400,
        "y": 120
      },
      "data": {
        "label": "Accepted",
        "outputSchema": [
          {
            "key": "status",
            "value": "accepted"
          },
          {
            "key": "invoice",
            "value": "$ExtractFields.json"
          },
          {
            "key": "ocrLanguage",
            "value": "$ReadDoc.language"
          },
          {
            "key": "pages",
            "value": "$ReadDoc.page_count"
          }
        ]
      }
    },
    {
      "id": "doc_flag",
      "type": "output",
      "position": {
        "x": 1400,
        "y": 380
      },
      "data": {
        "label": "NeedsReview",
        "outputSchema": [
          {
            "key": "status",
            "value": "needs_review"
          },
          {
            "key": "reason",
            "value": "Totals did not reconcile or a required field was missing"
          },
          {
            "key": "extracted",
            "value": "$ExtractFields.json"
          },
          {
            "key": "recognizedText",
            "value": "$ReadDoc.result"
          },
          {
            "key": "sourceFile",
            "value": "$Upload.file.download_url"
          }
        ]
      }
    }
  ],
  "edges": [
    {
      "id": "doc_e1",
      "source": "doc_upload",
      "target": "doc_read"
    },
    {
      "id": "doc_e2",
      "source": "doc_read",
      "target": "doc_extract"
    },
    {
      "id": "doc_e3",
      "source": "doc_extract",
      "target": "doc_validate"
    },
    {
      "id": "doc_e4",
      "source": "doc_validate",
      "target": "doc_store",
      "sourceHandle": "true"
    },
    {
      "id": "doc_e5",
      "source": "doc_validate",
      "target": "doc_flag",
      "sourceHandle": "false"
    }
  ]
}

Read it against the decision above. Recognition is deterministic and free. The one model call receives text and is told explicitly to return null rather than guess, which converts an invisible failure into a visible one. The Condition node re-derives a fact from the extracted data instead of trusting it. The rejected branch keeps the recognized text and a link to the original file, so the person reviewing it can see both what the machine read and what was actually on the page.

What Only Shows Up Once You Run This

Documentation describes the happy path. These are the things that surprised me while building and testing the node, all of which are fixed platform behavior rather than settings you can tune.

Every selected PDF page is rasterized and re-read, even when it has a perfectly good text layer. This is a deliberate tradeoff and it cuts both ways. It means a scanned document and a digital one behave identically, which removes an entire class of "works on my file" bug. It also means you pay recognition cost on documents that did not need it. If you know your inputs are digital, extract the text layer instead and skip the Converter entirely. If your inputs are mixed and unpredictable, uniform behavior is worth the cost.

There is a hard ceiling of 50 pages per run and a 120 second timeout. OCR is CPU-bound and runs on the executor's worker threads, so these caps protect the whole workflow engine and not just this node. On long documents, narrow ocrPageRange to the pages that actually carry the fields you want. A 200-page contract where the money is on page 3 does not need pages 4 through 200 recognized.

DPI is the quality dial, and it is not linear. The default of 300 is right for almost everything, the maximum is 600, and the minimum is 72. Going above 300 helps with small print and fine detail and costs time on every page. Going below it will save time and start losing characters.

Unicode normalization is on by default and you should leave it on. A Turkish ş written as a plain s plus a combining cedilla is a different string from a single precomposed ş, and the two will not match in a comparison or a database lookup even though they render identically. NFC normalization collapses them, which is the difference between a vendor lookup that works and one that silently misses.

The output encoding exists for downstream systems, not for accuracy. Recognized text is always UTF-8 and keeps every character Tesseract produced. Setting ocrEncoding to something like cp1254 or iso-8859-9 guarantees the result fits a narrower charset that an old target system can store. Narrower options like latin-1 and ascii replace what they cannot represent with a question mark, so only reach for them when a target system forces your hand.

An image cannot become a document and a document cannot become an image. The separate fileConvert conversion rewrites a stored file in another format through pandoc for documents and Pillow for images, but it will not cross that boundary. To read text out of a picture, the operation you want is imageToText, not a conversion.

Where Documents Go After Extraction

OCR workflow automation earns its keep at the seams, not at the recognition step. Extraction is rarely the end of the job, and two destinations come up constantly, and they want different things from the same pipeline.

Structured storage. Fields that reconcile go into a database, a DataTable, or a spreadsheet. This is the accounting-style path, and the validation gate is doing the heavy lifting: what reaches storage should be data you would defend in an audit.

A knowledge base. Recognized text can also feed a RAG pipeline, which is a different job with a different tolerance for error. A slightly garbled word in a retrieval corpus degrades a search result; a slightly garbled digit in an invoice total is a financial error. Because the standards differ, so should the gates: a RAG ingest path can accept text that a ledger path would reject. Sending both down an identical pipeline is a common and expensive mistake.

Worth noting for anyone chaining these: the fileConvert conversion outputs a file, not text. It is easy to assume a conversion step will produce something a RAG node can ingest directly, and it will not. For a retrieval pipeline you want the OCR conversions, which return text, or an agent skill that reads the file's contents.

The Self-Hosting Argument

There is a reason regulated industries have been slow here, and it is not skepticism about accuracy.

For most automation, the sensitive thing is the credential. For document processing, the sensitive thing is the document. A scanned employment contract, a medical form, a signed agreement: these are precisely the artifacts that compliance regimes are written about, and the standard hosted model asks you to upload every one of them to a third party for processing.

What is Heym? Heym (heym.run, github.com/heymrun/heym) is a self-hosted, open-source AI workflow automation platform. It provides a visual canvas for building workflows with LLM and Agent nodes, multi-agent orchestration, built-in RAG, MCP server and client support, human-in-the-loop checkpoints, and a Converter node that runs Tesseract OCR and pandoc conversion locally. It runs on your own infrastructure with a single Docker Compose command and is published under MIT with Commons Clause.

Running OCR locally changes what leaves your network. In Heym, recognition shells out to tesseract and poppler's pdftoppm on your own server, with pandoc handling document conversion, and all three ship with every way of running the platform, so there is nothing to install and nothing to configure. The document never moves. The only step that can involve an outbound call is the model that turns text into fields, and that call carries recognized text rather than the original file, and can point at a private or local model endpoint instead of a public API.

That is a meaningfully smaller exposure than uploading the PDF itself, and for some document classes it is the difference between a project that can proceed and one that cannot. It also happens to be the answer to a question that keeps going unanswered in the discussions: yes, open source AI document processing is a real option, and no, it does not require sending your files anywhere.

When Not to Automate This

Three cases where the honest recommendation is to keep a person in the loop rather than build a better pipeline.

Low volume with high stakes. Twelve contracts a year does not justify a pipeline. Read them.

Handwriting that carries legal or financial meaning. Classical OCR is weak on handwriting because it was built for typeset characters. A vision model is much better and still not good enough to leave unattended when the value is a settlement amount. Extract it as a suggestion and have a person confirm it.

Documents whose layout is the meaning. Some forms encode information positionally in ways that flatten into nonsense as linear text. If a checkbox two columns to the right of a label changes what a field means, you need layout-aware handling or a person, not more OCR tuning.

For everything else, and it is a large everything else, the pattern in this article holds: recognize deterministically, extract against a schema, validate arithmetically, and route what fails to somebody who can look at the page. If you want to see the surrounding patterns, our AI workflow automation examples catalog places this alongside sixteen other builds, and the templates library has runnable starting points.

Frequently Asked Questions

Why can I not select the text in my PDF?

Because the PDF has no text layer. A PDF is a container, and it can hold either real character data or nothing but a picture of a page. Anything produced by a scanner, a phone camera, or a fax has pixels only, so a parser that reads the text layer finds an empty document and returns nothing. This is the single most common reason PDF extraction appears to fail for no reason. The fix is not a better parser, it is recognition: run OCR to turn those pixels into characters first, then parse.

Can AI extract data from handwritten or scanned PDFs?

Scanned print, yes, reliably. Handwriting is where classical OCR engines like Tesseract fall down, because they were trained on typeset characters and a cursive stroke does not resemble one. For handwriting, a multimodal vision model is the better tool, at a higher cost per page and with a different failure mode: instead of returning garbled characters it may return a confident, plausible, wrong value. If handwritten fields carry money or legal meaning, route them to a person rather than to either engine.

Is OCR still necessary now that vision models can read documents?

The evidence is genuinely split, and the split is instructive. A March 2026 benchmark by Shen et al. found that image-only input to a strong multimodal model can match OCR-enhanced setups on raw extraction accuracy. An August 2026 benchmark, ConfBench by Roy et al., found that the OCR plus image modality produces more accurate confidence estimates. Accuracy and trustworthy confidence are not the same property. If you never need to know how sure the system is, a vision model alone may be enough. If you need to route uncertain documents to a human, keeping OCR in the pipeline is what makes that routing decision meaningful.

What is the most cost-effective way to extract structured data from semi-structured PDFs?

Recognize once with a deterministic engine, then reason once with a model. OCR turns pixels into text at effectively zero marginal cost because it runs on your own CPU with no per-token billing. You then send that text, not the image, to a language model to pull fields out against a JSON schema. Text tokens are far cheaper than image tokens, the model sees a much smaller payload, and the expensive step runs once per document instead of once per page. The costly pattern is sending every page of every document to a multimodal model as an image.

Is there an open-source alternative to Parseur or Docparser?

Yes, if you are willing to assemble the pieces rather than buy a finished product. Heym is open source and self-hosted, and its Converter node runs Tesseract OCR on images and PDFs directly on your infrastructure, then hands the text to an Agent or LLM node for schema-constrained extraction. Docling and Unstract are the other commonly recommended open-source options. The tradeoff is real: you get no per-page fee and no document leaving your network, and in exchange you own the pipeline design that a hosted product would have made for you.

Do I have to send my documents to a third-party service?

Not with a self-hosted setup. In Heym, OCR shells out to tesseract and poppler running on your own server, so a scanned contract is recognized without leaving your infrastructure. The only step that can involve an external call is the language model that turns text into structured fields, and that step can point at a local or private model endpoint instead of a public API. That distinction matters for regulated documents, where the file itself is often the sensitive artifact.

Key Takeaways

• Check for a text layer before anything else. A PDF with no text layer is the most common cause of "the parser is broken", and it is a recognition problem, not a parsing problem.
• Classical OCR and vision models fail in opposite directions. Garbled characters can be caught by a validation rule; a confident wrong number cannot.
• The 2026 benchmarks disagree on accuracy and converge on calibration. Keeping OCR in the pipeline is what makes a confidence score worth routing on.
• Send text to the model, not images. It is cheaper, smaller, and runs once per document rather than once per page.
• Validate arithmetically, then escalate. Reconciliation catches misread digits that no amount of prompting will prevent, and a review queue beats a retry loop.
• The document is the sensitive artifact. Local recognition keeps the file on your infrastructure even when the extraction model is remote.

References

• Roy, P. et al. Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction, arXiv, 3 August 2026.
• Shen, J. et al. OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets, arXiv, 3 March 2026.
• He, Z. et al. Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models, arXiv, 2025.
• Tesseract documentation, tesseract-ocr project.
• Poppler and Pandoc, the PDF rasterization and document conversion tools Heym ships.
• r/automation, I Tried 6 PDF Extraction Tools discussion thread.