MaxxBench questions DeepSWE score validity
The gist
MaxxBench revised 89 DeepSWE coding tasks and found model scores spread far wider than on the originals.
The gap suggests high scores on a familiar benchmark may not carry over.
How it unfolded
- Oct 20, 2026The MaxxBench site is published, describing a benchmark that alters the instructions or starting code of 89 DeepSWE tasks while keeping the main engineering problem.
- Nov 4, 2026Cognition introduces SWE-2 Flash, reporting a DeepSWE 1.1 score of 60.1 for SWE-2 Flash and 77.0 for DeepSeek V4.1 Flash.
- Nov 6, 2026On the DeepSWE 1.1 Agents & Tool Use leaderboard, GPT-6 Astra (74.1) leads, followed by Gemini 3.8 Flash (73.8) and Claude Opus 5 (73.6), with 44 models ranked.
Key terms
- benchmark
- A standard test or set of tests used to compare the performance of computer hardware or software.
Source: Wikipedia