ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction

ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction, Ferguson et al., arXiv’26

Here’s a tough problem: can you turn a huge amount of unstructured prose into a structured representation of the data inside? This is the problem facing anyone who’s trying to run LLMs against large documents.

It’s a common problem (legal AI startups; anyone working with contracts), it’s a hard problem, and in the ExtractBench paper Ferguson et al. go a long way towards demonstrating why.

Failure modes

Let’s talk about ways an LLM might fail to parse unstructured data into structured data. There’s some standard failures:

• fails to parse something which exists
• hallucinates something which does not exist
• parses something which exists but which then fails structured data validation
• required value in the structured data not present

…then there’s some failures which are specific to large inputs:

• filesize limits
• output token limits
• schema breadth
• nested schema

…and there’s failures specific to references:

• text contains a reference to something which we could not find
• text contains reference which we found but could not determine how to use

The problem only gets harder from there. It’s not covered in the ExtractBench paper, but we can take this problem further: what happens when you extend this problem beyond a single document? If filesize limits were already an issue on even a single document, how can your system handle an arbitrary number of documents? If you need to provide a search-based approach to resolve references, how can you make that search semantic?

Findings

ExtractBench doesn’t try to tackle all of the above - it makes no mention of either reference resolution or multi-document searching. Instead it focuses specifically on single document extraction, and makes one very compelling point: even on this simplified case, frontier models fail.

Evaluations of frontier models on ExtractBench reveal sharp degradation as schema breadth grows, reaching 0% valid output on a 369-field financial reporting schema across all tested frontier models. This exposes a failure mode hidden by benchmarks limited to small schemas: frontier models become unreliable when required to generate long, deeply structured JSON. Valid JSON also does not imply correct extraction – on one domain, models achieve 90% valid output but only a 12.5% pass rate.

Usefully, they provide findings as to why these frontier models fail. GPT failed on output volume:

Output volume drives GPT failures. Gold JSON token count – a proxy for required output complexity – reveals a provider-specific pattern. GPT models succeed on credit agreements (0.9k gold tokens) but struggle on all other domains, which require 3k–25k tokens. GPT-5 achieves 100% validity on credit agreements yet only 0–14% elsewhere, suggesting that output volume is a binding constraint for this model family.

Claude failed on page count:

Page count constrains Claude models. Claude’s 100-page PDF ingestion limit blocks all credit agreement documents (97–218 pages, averaging 137). This eliminates Claude from the domain where other models perform best, illustrating how provider-specific API constraints interact with document properties to shape benchmark outcomes.

Array expansion tripped everyone up:

Array complexity amplifies difficulty beyond schema size. Research papers have only 16 schema keys – comparable to credit agreements (13) – yet achieve far lower validity (39% vs. 67%). The difference lies in array complexity: research paper gold annotations average 309.5 array items per document (driven by citation lists of 100+ items), yielding 25,366 gold tokens despite a compact schema. Credit agreements average only 15.7 array items and 883 gold tokens. This demonstrates that total output volume – reflecting both schema fields and array expansion – is a better predictor of difficulty than schema key count alone.

And turning on structured output was certainly no panacea:

We repeated the full 210-extraction experiment using each provider’s structured output API. The results show that structured outputs reduce both validity and accuracy relative to prompt-based extraction: overall validity dropped from 51% (107/210) to 37% (77/210), and the best-model pass rate fell from 6.9% to 5.5%.

Overall, no model covered itself in glory:

This is genuinely useful! Knowing the dominant failure modes is a great first step for anyone tasked with implementing such a system - here are some specific challenges you will need to overcome.

Old Man Yells at Claudes

OK, so the authors outline specific failure modes that make structuring unstructured data hard. Let’s take a slightly closer look though - I have some quibbles.

1. Treating limits as failures

At the time of writing, Claude had a 100-page limit on PDFs. The authors chose to report this as Claude scoring a 0% on all the credit agreement documents, which were over this limit. This is arguably defensible, but not practically useful; any system facing this limit in practice would presumably seek to address this limit via a mechanism such as chunking or segmenting the input. It would be far more useful to demonstrate the failures from a naive segmentation rather than simply treating a page-limit as complete failure.

Unsurprisingly, these limits are already outdated as well - we’re only 5 months post-publication, and the Anthropic models already support 600 page requests. I would be very curious to see updated results from running 600-page Fable 5 against this benchmark - I have a feeling we may not be that far from saturation.

2. Failing to cover genuine failure modes

We discussed above some specific hard problems in this space. How do we tackle reference resolution? How do we handle information dispersed across multiple documents? These are questions that anybody in this space will be immediately confronted with, yet they aren’t even mentioned. At least mention them in a ‘future work’ section!

3. ‘Tolerance’ for numeric differences

How do we evaluate correctness of structured data ingestion? The authors make the argument that evaluating correctness should be tackled differently based on the type of field:

Correctness is field-dependent: IDs need exact match, quantities need tolerance, names need semantic comparison, and arrays demand careful handling of ordering and per-item matching.

That seems…mostly correct. IDs definitely do need exact match, and names do need semantic comparison. But quantities receive tolerance? I admit it - I’m skeptical. An accountant wouldn’t dare claim ‘surely somewhat close is good enough’, and I wouldn’t accept that from an LLM either. Numbers are either correct or incorrect, and close doesn’t cut it.

When you put these things together, you’re left with a bit of an unflattering picture. The paper comes out of Contextual AI, and my expectation is that it was written to justify a product, the goal being to generate a picture something like this:

…this, but for ExtractBench instead of RAG-QA. (source)

In that light, numeric tolerance and failing to discuss genuine failure modes aren’t benchmark oversights - they’re papering over genuine coverage gaps in the product. It really isn’t fair for me to speculate as to motive, but what I can say is this: if you’re implementing these systems in production, you WILL run into these problems, and you WILL need to come up with a mechanism to solve them.

Good luck~