Gemini 2.5 Pro vs Claude Sonnet for Reading Long Contracts: What the Benchmarks Actually Say

Reading a long contract with a general-purpose AI model is not one task but several: locating clauses across dozens of pages, extracting defined terms and obligations into structured fields, and doing so without inventing language that isn't in the document. The published evidence on Gemini 2.5 Pro and Claude Sonnet speaks mainly to the middle task — structured extraction — and it is thinner than a buyer might hope.

Two sources are relevant here, and it is worth being precise about what each one measured. An academic evaluation of structured contract extraction scored several frontier models, including Gemini 2.5 Pro and Claude Sonnet 4.6, on precision, recall and F1 across six field categories ^¹. Separately, Unstructured, a document-processing vendor, published an undated-in-excerpt account of its SCORE-Bench test covering 224 real-world enterprise documents, including legal contracts, run with a simple extraction prompt and no additional configuration ^². Neither source is a head-to-head review of long-contract reading in the way a practitioner would run it, and neither should be read as one.

The criteria that decide it

For long contracts, four criteria do most of the work. Extraction accuracy on structured fields is the first, and the most measurable. Precision versus recall is the second: a model that misses a clause and a model that invents one fail differently, and which failure hurts more depends on whether a human reviews the output. Document robustness is the third — whether performance holds on scanned pages, skewed text and complex layouts rather than clean PDFs. And context handling is the fourth, since long contracts are exactly the case where a model must keep track of defined terms introduced a hundred pages earlier.

Gemini 2.5 Pro

In the contract-extraction study, Gemini 2.5 Pro recorded a macro-averaged F1 of 0.769, with precision at 0.704 and recall at 0.878. That recall figure is the highest of the two models discussed here and among the highest in the comparison, which fits a model that casts a wide net: it is relatively likely to surface a clause, at the cost of more unsupported extractions. The study's authors note that the leading models were closely clustered, so this is a difference of degree rather than category.

Gemini 2.5 Pro also appears in the Unstructured SCORE-Bench test alongside Claude Sonnet 4, GPT-5.2 and others, run with a simple extraction prompt and no extra configuration. The excerpt does not report per-model scores, so no ranking between the two models can be drawn from that test on the evidence available; the vendor's broader argument is that document parsing remains harder than frontier-model benchmarks imply.

Claude Sonnet 4.6

Claude Sonnet 4.6 scored a macro-averaged F1 of 0.766 in the same contract-extraction study, with precision of 0.717 and recall of 0.839. Read against Gemini 2.5 Pro, the profile is inverted: slightly higher precision, meaning fewer unsupported extractions, and lower recall, meaning more clauses left unfound. The aggregate F1 difference between the two is 0.003 — small enough that the study's own framing of a tightly clustered leaderboard applies directly.

The Unstructured test used Claude Sonnet 4 rather than 4.6, so it does not directly speak to the newer version. That is a real gap in the evidence, not a detail to gloss over: anyone comparing the current Sonnet release against Gemini 2.5 Pro on messy enterprise documents is extrapolating from a test of an earlier model.

Side by side

All figures in the first three rows come from the contract-extraction study. The final row reflects the Unstructured test and its limits.

Which reader each one suits

The honest answer is that the choice turns less on the leaderboard than on workflow. A reader who uses the model as a first pass — dumping a long contract into structured fields and then having a human check every entry — is better served by the higher-recall option, because missed clauses are the expensive failure and false positives get caught downstream. On the published numbers, that points to Gemini 2.5 Pro.

A reader who wants the extraction to be trustworthy without a line-by-line re-read, or who is feeding output into an automated system where an invented clause is worse than a missing one, should weight precision more heavily. Claude Sonnet 4.6's precision edge is modest but consistent with that preference.

Neither conclusion is strong enough to declare a universal winner, and the document-parsing vendor's warning is the right caution: strong benchmark scores do not guarantee clean results on real scanned invoices, nested tables and multi-column legal layouts. If your contracts are clean digital text, the extraction numbers are the best guide available. If they are scans and exports from a dozen different systems, test both on your own documents before committing — and treat any claim of a clear winner as unconfirmed.

For readers weighing broader assistant choices on document work, our earlier comparison of Claude vs ChatGPT vs Gemini for summarizing work documents makes a similar point about criteria over rankings, and Claude Projects for recurring research work covers how to structure repeatable review workflows once a model is chosen.

Sources

1. ¹ A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract ExtractionCCS: Computing methodologies Natural language generationCCS: Computing methodologies Informa arxiv.org
2. ² Frontier Models Are Strong But Document Parsing Is Harder | Unstructured web-preview.unstructured.io