We tested 14 AI models on rate confirmation data extraction, using real documents from US freight brokers: 3 scanned and 2 digital, from 3 broker templates. Two inexpensive models per document type came out clean after prompt fixes, at a fraction of gpt-4o's cost. No model was reliable on its own, and scanned and digital documents needed different models.
The test ran for Zonik AI, a logistics product we co-founded, in September 2026. We ran every comparison in TestAImodels, our own tool for comparing AI models side by side. Zonik's team shared the full results, including what went wrong, because that's the useful part for anyone automating freight documents.
What we tested
A rate confirmation is the document a freight broker sends a carrier for a load: pickup and delivery stops, dates, time windows, rate, weight and reference numbers. Zonik reads each one and turns it into structured load data.
Zonik reads rate confirmations two ways, and we tested both separately.
| Scanned path | Digital path | |
|---|---|---|
| Documents | Scanned or photographed PDFs with no usable text layer | Digital PDFs with a real text layer |
| How the model reads it | From the image (vision) | From the extracted text |
| Model in production today | gpt-4o | gpt-4o-mini |
| Documents tested | 3 real rate confirmations, plus a re-run of document 1 after the prompt fixes | 2 real rate confirmations |
| Models tested | 10 | 10 |
The documents came from 3 different broker templates. Every model got the same prompt. Costs are measured token use on these documents multiplied by each provider's list price on September 16, 2026, shown per 1 million documents.
This is a small, real-world test: a handful of documents, one prompt, one document type. It isn't a ranking of AI models, and the same models can behave differently on other documents.
Results: scanned documents
"Not run" means the model wasn't tested on that document.
| Model | Doc 1 | Doc 2 | Doc 3 | Doc 1 after fixes | Cost per 1M documents |
|---|---|---|---|---|---|
| Gemini 3.1 Flash Lite Preview | Left the delivery time empty | Clean | Clean | Clean | $1,937-$3,169 |
| Grok 4 Fast | Wrong delivery time | Not run | Clean | Clean | $1,370-$1,966 |
| gpt-4o (current production) | Clean | Not run | Not run | Clean | $12,450-$18,208 |
| GPT-4.1 | Not run | Not run | Not run | Clean | $14,838 |
| Claude Sonnet 5 | Clean | Incomplete output | Not run | Not run | $26,166-$33,342 |
| Claude Haiku 4.5 | Put the broker in the carrier field | Not run | Not run | Not run | $14,463 |
| GPT-5.4 nano | Corrupted an address | Not run | Not run | Not run | $2,800 |
| GPT-5.4 mini | Wrong appointment flag | Not run | Not run | Not run | $8,969 |
| Claude Opus 5 (Fast) | Clean | Not run | Not run | Not run | $183,860 |
| Grok 4.5 | Wrong delivery time | Not run | Clean | Not run | $15,228-$21,086 |
After the prompt fixes, Gemini 3.1 Flash Lite Preview and Grok 4 Fast were clean on every document we ran them on. Their cost was 83-89% lower than gpt-4o's, comparing the low and high ends of each price range.
Results: digital documents
| Model | Doc 1 | Doc 2 | Cost per 1M documents |
|---|---|---|---|
| Grok 4 Fast | Clean | Clean | $729-$1,103 |
| DeepSeek V3.2 | Clean | Clean | $1,061-$1,499 |
| Grok 3 Mini | Clean | Clean | $1,013-$1,477 |
| GPT-5.4 mini | Carrier ID read as the rate confirmation number | Clean | $2,061-$5,869 |
| gpt-4o | Carrier ID read as the rate confirmation number | Clean | $10,300-$16,720 |
| Mistral Small 4 | Zip code on the wrong stop | Reference line read as a time window | $905-$1,311 |
| Gemini 3.1 Flash Lite Preview | Carrier ID read as the rate confirmation number | Reference line read as a time window | $1,677-$2,817 |
| GPT-5.4 nano | Dropped the commodity | Dropped a time and window | $1,001-$1,730 |
| Claude Haiku 4.5 | Zip codes shifted on every stop | Reference line read as a time window | $5,313-$9,008 |
| Qwen3.6 Flash | Slow and costly | Time window error, slow | $7,660-$8,333 |
Three models were clean on both documents, and all three were among the cheapest in the test. Production uses gpt-4o-mini on this path, which we haven't tested yet, so these figures don't show savings against production here.
The 7 errors we found
Every one of these was reproducible, and all 7 fixes are now live in Zonik's prompt, which also improves the models Zonik uses in production today.
On scanned documents:
- The wrong delivery time. When a stop had no appointment but a From/To window, models picked the time the window opens instead of the deadline.
- A duplicated reference number. The rate confirmation number was filled with a copy of the load number, even though the prompt said not to.
- A repeated time window. When the From and To times were the same, models filled both fields with that time instead of leaving them empty.
- Legal text in every stop. Full legal and accessorial boilerplate was copied into the notes of every stop instead of a short summary.
On digital documents:
- The carrier's ID read as the rate confirmation number. A carrier's MCID (Motor Carrier ID), printed near the load number, was taken as the rate confirmation number by 3 of the 10 models, including gpt-4o.
- Zip codes on the wrong stop. Claude Haiku 4.5 attached each stop's zip code to a different stop's address, on every stop. The cause was the PDF's text layout: a zip code can sit far from the rest of its stop on the page.
- A reference line read as a time window. A system reference line such as "OH 07:00-16:30" was read as the stop's appointment window by 3 of the 10 models, overriding the correct time on the stop's own date line.
None of these errors came with a warning. Each one was a clean, well-formatted value that was wrong.
What the test taught us
The expensive models didn't win. The two best models on each path were among the cheapest we tested. Claude Opus 5 (Fast) was clean on the one document it ran, at about 60 to 130 times the cost of the winners.
Scanned and digital documents need different models. Gemini 3.1 Flash Lite Preview was the strongest model on scanned documents and failed both digital documents, with two different errors. One model for both paths would have been the wrong choice.
The prompt mattered. After the fixes, every model we re-ran on document 1 was clean, including gpt-4o, the model already in production.
No single model knows when it's wrong. Every error above looked like a correct answer. A second, different model disagreeing is what can surface them, which is the point of our earlier post on what 98% accuracy really means.
What Zonik built next
Instead of switching to one cheaper model, Zonik now runs two. Since September 18, 2026, Zonik AI, a logistics product we co-founded, runs two models on the rate confirmations its first users upload: Gemini 3.1 Flash Lite Preview and Grok 4 Fast on scanned documents, and Grok 4 Fast and DeepSeek V3.2 on digital ones. The dispatcher reviews and confirms before the load is saved.
The check compares the two answers field by field:
| Fields | How they are compared |
|---|---|
| Dates, times, rate, weight, miles | Must match exactly |
| Broker name | Compared after normalizing it |
| Notes, labels | Not compared |
A field where the two models disagree is flagged for the dispatcher.
As a projection, running the two scanned-document winners together would cost roughly $3,300-$5,100 per 1 million scanned documents: their measured costs from the table above added together ($1,937-$3,169 plus $1,370-$1,966, at list prices on September 16, 2026). That is about 72% less than one gpt-4o call ($12,450-$18,208). It is a projection from the test, not a measured cost of the live check.
It's the same approach we describe in how we build with AI: the AI prepares, and a person confirms.
The limits of this test
- Small sample. 3 scanned and 2 digital documents from 3 broker templates.
- One prompt, one document type. Results for invoices, customs forms or other freight documents could differ.
- Not every model ran on every document. The tables show exactly which runs happened.
- Prices change. Costs use list prices on September 16, 2026.
- Model versions change. These are the versions available in September 2026.
We'd rather publish a small test with its limits than a big number without them.
FAQ
What is the best AI model for extracting data from freight documents?
In our test on US broker rate confirmations, Gemini 3.1 Flash Lite Preview and Grok 4 Fast were clean on scanned documents after prompt fixes, and Grok 4 Fast and DeepSeek V3.2 were clean on digital ones. The sample was small, so test on your own documents.
Are more expensive AI models better at document extraction?
Not in this test. The winning models were among the cheapest, and Claude Opus 5 (Fast) cost about 60 to 130 times more than the winners for a clean result on one document.
Should I use the same AI model for scanned and digital documents?
Our results say no. The strongest model on scanned documents failed both digital documents.
Why run two AI models on the same document?
A single model gives no sign when it's wrong. Two different models disagreeing on a field is a reason for a person to check it.
How much does AI document extraction cost?
In this test, the winning models cost between about $700 and $3,200 per 1 million documents at September 2026 list prices. As a projection, running the two scanned-document winners together would cost roughly $3,300-$5,100 per million, adding their measured costs at September 16, 2026 list prices. Since September 18, 2026, Zonik AI, a logistics product we co-founded, runs two models on the rate confirmations its first users upload, and the dispatcher reviews and confirms before the load is saved.
What is a rate confirmation?
The document a freight broker sends a carrier to confirm a load: stops, dates, time windows, rate, weight and reference numbers.
If you're automating freight documents, the model matters less than the checks around it. Want to see how this would work on your documents? Book a free audit call.
