MaxxBench – "Nice DeepSWE score. But is it true?"

Particularly interesting cases here are of GLM 5.3 and Claude Opus that seem to rely on assumptions despite clearly stated requirements. > Let me just do float64; the risk of a test with >2^53 numbers is minimal. > such cases are unlikely … Truncation toward zero is fine. Apart from this, we also a scenario where the…

This is a great example of how to test how well a benchmark score generalises. The first order approximation is not complicated: just vary a parameter and see how much the score changes. Which parameter to vary depends on the result of the test, but the MaxxBench route of varying the task spec seems like a good…

Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a…