We tested six vendors before we built anything.
Every one of them got worse on the documents that actually arrive.
One invoice. Recorded scores only. Currency has no bounding box because it isn't printed on the page.
The problem
An extraction error isn't a typing error.
Every finance workflow starts by reading a document. Somebody opens an invoice and types the fields into an ERP. Vendor, dates, amounts, tax, terms.
That somebody had a name. In my first week I sat and watched her work. Forty-seven browser tabs, three spreadsheets, a legacy ERP, and a folder on her desktop called INVOICES_FINAL_FINAL_v3. She wasn't bad at her job. She was good at it. The software had never been built for the way she worked, so she'd built around it.
Get one wrong and it doesn't cost a retype. It propagates. A misread invoice number fails the downstream purchase order match. It surfaces three weeks later, in somebody else's reconciliation.
So the question was never how fast a machine reads. It was how often it's wrong, and whether it knows.
Everything below follows from that second clause.
The benchmark
Buying was the cheaper answer. So we ran the test the vendors had not.
Commercial document processing has existed for years, the vendors publish high accuracy numbers, and those numbers are real. They are measured on clean documents.
What arrives in an AP inbox is a phone photograph at an angle. A fax of a scan. Handwriting. Vietnamese, Bahasa, Chinese. Excellent on a clean PDF and poor on a photograph isn't excellent. It's untested.
What decided it
Not the highest number. The smallest drop.
Every platform we tested got worse on handwriting than on its own clean documents. One dropped 27 points against itself. We dropped five.
Our worst category beat every platform's best. We scored 87.8% on handwriting. No commercial platform reached higher than 87.3% on anything, including their clean digital PDFs.
How it's measured
Neither number means anything alone.
The benchmark's two headline figures aren't the same kind of thing. Naming them precisely is the difference between reporting a result and understanding one.
| Benchmark term | What it counts | In ML terms |
|---|---|---|
| Accuracy, 92.5% | correct ÷ applicable | recall |
| Coverage, 97.6% | attempted ÷ applicable | attempt rate |
| 94.8%, derived | correct ÷ attempted | precision |
The gap between the first two is 5.1 points, the smallest of any platform tested. Almost everything we attempt to extract turns out correct.
They pull against each other. You reach 100% precision by only answering when certain, and coverage collapses onto the AP team. You reach 100% recall by always guessing, and precision collapses into bad data in the ERP.
The confidence gate is the mechanism that trades one for the other, deliberately.When the system is unsure it declines to commit, converting a would-be precision failure into a coverage failure. A blank and a person, which is the cheaper of the two.
So the headline isn't 92.5%. It's that the 7.5% we got wrong, we flagged.
Two evals, and both are correction rate. Offline against the golden set, as a regression gate on every model change. Online against live traffic. The same metric on both sides. A regression in testing is directly comparable to a drift in production.
And every quality metric needs a counter-metric, or it gets gamed.
Escape rate is the one we don't have. Everything measured today is work the system knows it did. Nothing measures what it got wrong without noticing. Until that exists, confidence that the gate holds is an assertion, not a number.
Calibration is the other gap. Stated confidence plotted against observed correctness, per field. We have never drawn that curve. The currency finding below is what its absence looks like from the outside.
The architecture
Four of the eight stages run no model at all.
The organising idea isn't the order. It's the cost. Every stage is built to avoid paying for the next one. The expensive model is fenced into a single place that almost never fires.
Junk filter
Logos, signatures and tracking pixels die before anything is paid for.
Digital or image
Both try to pull the text out. The ratio they recover decides it. A rule, not a classifier.
Document type
Three tiers. Each pays only for what the tier before could not settle.
Extraction
Every field returns a value, a bounding box, and a confidence score.
Verification
Asked is this right. Never what is the right value. Payload is the bounding box, never the page.
Repair
Only where the check failed. Single shot, no retry loop.
Arithmetic
Line totals against header totals. A bigger model can't tell you which number is the lie.
Review gate
Configurable per tenant and per workflow, as is the field schema itself.
Scroll the pipeline sideways →
Three decisions I would defend
The most expensive model reads every field, and answers in one word.
Input tokens cost less than output tokens. So the judge is asked is this right, not what the right value is. Checking everything becomes affordable because the answer is one token. Repair stays expensive, and fires only where checking failed. Most systems trade coverage against cost. Constraining the output removes the trade.
Arithmetic failures never reach a model. They go to a person.
An inconsistency is localisable but not resolvable by re-reading. If the line items sum to 40 and the header says 44, a higher-fidelity read tells you nothing. It can't say which of the two is the lie. That isn't escalating hard cases to a bigger model. It's knowing which hard cases a bigger model can't help with.
Review is a product surface, not an error path.
A reviewer sees one uncertain field, highlighted where it sits on the page. They correct it. That correction fixes the document, measures the system, and produces a training label. One action, three jobs. Which is why we could ship before accuracy was good enough.
What the reviewer sees
The bounding box is the whole product.
Extraction returns three things per field. The value, the bounding box it came from, and a confidence score per character.
The confidence score decides whether a person ever sees it. And the bounding box is why a reviewer sees one uncertain field in context. Not a whole document to re-read.
Nothing here depends on the model being right. It depends on the model knowing when it isn't.
The finding I didn't expect
Our most accurate field is the one we send to a person most often.
Currency scores 98.4%, the highest of the fifteen. A reviewer sees it more than any other field. Both are true.
The currency is usually not written on the invoice at all. It's worked out from where the invoice was billed. An invoice sent to the Philippines is in pesos, and nobody writes Philippines on an invoice. So the model is deducing, not reading. It's correctly less sure about a deduction than about something it can see.
That's right as behaviour and wrong as a threshold. Not an accuracy problem. A calibration problem.
The hardest problem
It was never extraction.
One of these was an invoice. Nothing downstream will ever know.
It was deciding what a document is, and what it belongs to. An invoice, a purchase order, a tax document and a delivery note all carry the same shape. A vendor, a number, a date, amounts. In Indonesia the tax document is itself an invoice, legally distinct, with its own number format, and a required relationship to the commercial one. A single email arrives with five attachments and no indication of which belongs to which.
And the failure mode is the worst kind: silent. Every other error here is loud. A low confidence score, an arithmetic mismatch, a model refusing to answer. A misclassified invoice produces no error at all. It's processed successfully as something else, so nothing downstream ever learns an invoice arrived.
Nobody chases a document they don't know exists. It surfaces weeks later as a vendor asking where their money is.
"Irrelevant" is a terminal state. So it should require the most confidence, not the least.
What it still can't do
The part that makes the rest believable.
Ground truth was compiled incrementally, so not every invoice carries a value for all fifteen fields. Scoring runs only over the fields that were annotated.
A blank doesn't mean the field was absent from the document. It may mean nobody recorded it yet. So the aggregate over 13,308 scored combinations is solid, and the thinnest per-field results aren't. Publishing confidence intervals per field would cost nothing and would say which differences are real.
A field never found has no bounding box, so nothing checks it and nothing flags it. We catch a wrong answer. We don't catch a missing one.
92.5% measures reading, on documents already known to be invoices. Of everything entering, what share comes out completely correct is necessarily lower, and it isn't measured.
One ratio, costing almost nothing, deciding which reader wins a disagreement. No measurement would show it going wrong. The piece nobody measured is the one that looked too simple to need it.
What comes next
Four things, and none of them is a bigger model.
The two measurement gaps above, escape rate and calibration, are the first two items on the list. These are the four build items behind them.
Line items across documents
Read the line items on the purchase order and match them against the invoice. Documents are tied together today by one identifier. Line-item agreement is a second, independent signal over the same relationship. Unlike a reference number, it can't be transcribed correctly onto the wrong document.
It turns a lookup into a check. A purchase order that matches on number but not on contents is a different failure. Not the same as one that doesn't match at all. Today only the second is visible.
Validate the tax identifier rather than read it harder
Tax identifiers are structured and checkable, unlike most fields. VAT and GST numbers have country-specific formats, and many carry a checksum digit. So a guess that fails the checksum is known wrong before a person ever sees it.
And the buyer already holds the supplier's tax ID in the onboarding record. The right output is often matches vendor master, not a fresh read. Six tax logics should mean six validators chosen once the country is known, not one generic extractor.
Embedding-based classification
Document type is decided today by a prompt. Moving it to embeddings makes the decision measurable and cheap to re-run over history, which matters because classification is the stage whose failures are silent.
It also gives the terminal label a threshold of its own. Irrelevant ends a document's life, so it should require the most confidence, not the least.
A smaller model trained on our own documents
Not for the cost. The documents are Thai, Bahasa, Vietnamese and Chinese mixed with English, carrying tax fields most Western-trained models have never seen.
A general model here isn't merely expensive. It's trained on the wrong distribution. The direction is the same one every time. Build the narrow version first. It teaches you what the general version has to handle.
