I spoke to a data scientist today who had set up a sophisticated agentic system for users to semantically query a large, complex database. Three weeks in, he could show me a live demo - it appeared to be working.
When I asked what evals he had done to ensure the responses were accurate to the data, he said "oh yes, we definitely need to do that".
It's a common pattern to build something based on the intuition of "how it should work", and leave the testing for later - but our assumptions about how LLMs "should" behave are shallow, based on news articles, benchmark scores, maybe 'gut feel' from having used them a lot across a range of tasks.
What matters when building software, though, is how well a model does for this task.
I've got a project going here (to be released later) that uses an LLM step to triage some items - some should be selected for the next pipeline stage and some dropped. I hand-labelled 3 days' worth of inputs (246 items, run 3x to measure variance in results) to see how the LLMs were doing.
| provider | recall | regression rate | avg s/call | $/run | $/9-run pool |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 69.6% (n=303) | 14.0% (n=435) | 49 | $0.05 | $0.45 |
| DeepSeek-V4-Pro | 71.0% (n=303) | 17.9% (n=435) | 260 | $0.01 | $0.09 |
| Sonnet 4.6 | 65.3% (n=303) | 14.5% (n=435) | 205 | $0.25 | $2.24 |
| Opus 4.7 | 64.0% (n=303) | 14.3% (n=435) | 158 | $0.47 | $4.22 |
On paper, Opus 4.7 is the most capable model in this list, and possibly in the world today (you can debate this with GPT-5.5, but I haven't seen anyone try that argument with Gemini Flash 2.5!). Opus leads Gemini Flash by 13.4pp on GPQA Diamond and 33.6pp on SWE-bench Verified. Sonnet leads by 25.6pp on SWE-bench and 20pp on AIME 2025.
| benchmark | Gemini 2.5 Flash | Sonnet 4.6 | Opus 4.7 |
|---|---|---|---|
| GPQA Diamond (PhD-level science MCQ, pass@1) | 80.8% | 74.1% | 94.2% |
| SWE-bench Verified (agentic SWE, pass@1) | 54.0% | 79.6% | 87.6% |
| AIME 2025 (olympiad math, pass@1) | 75.6% | 95.6% | not published |
But for this task, Flash is 3x faster, and 9x cheaper (the accuracy looks better too - but if you work out the stats here, none of them are different to each other with statistical significance at this sample size).
While changing a model is trivial, often the outcome of an eval might have a bigger impact on your architectural choices - so it's important to do them early and often in your project!
Sources: Anthropic Opus 4.7 announcement, Sonnet 4.6 announcement; Google Gemini 2.5 Flash model card (Dec 2025), DeepMind Gemini Flash page. All Anthropic figures are with adaptive thinking on (the only headline configuration Anthropic now publishes); Gemini figures are thinking-on, single attempt, no majority voting (per Google's model card methodology). AIME 2025 was not reported by Anthropic for Opus 4.7 — they substituted Humanity's Last Exam (46.9% no tools / 54.7% with tools) as the headline reasoning benchmark this cycle. As a footnote: Google has since shipped Gemini 3 Flash (90.4% GPQA Diamond); we ran 2.5 Flash because that's what's in our production pipeline today.