The Daily Byte
Breaking
Welcome to The Daily Byte — real stories from across Pakistan.
Saturday, 21 November 2026 · Pakistan

The Daily Byte

Stories from Pakistan that deserve to be heard
Minor World

Gemini 4 Carbon Tops MathArena at 15 Times the Cost

By The Daily Byte ·16 Nov 2026 ·3 min read
The Google headquarters building in Mountain View with the multicoloured Google logo on its glass facade

Google's Gemini 4 Carbon has taken first place on MathArena, the independent benchmark that tests AI models on new mathematics problems, with an expected performance of 85.4%. It is the first time in a while that a model other than one from OpenAI has led the ranking. Anthropic's Claude Fable 5.2 is second at 82.8%, and OpenAI's GPT-6.1 Sol, the previous leader, is third at 80.6%. MathArena added results for both new models on 13 November, days after they were released: Carbon reached general availability on 10 November and Fable 5.2 was released on 9 November.

The headline comes with a large caveat on cost. At the reasoning settings used in testing, solving one problem once costs about $15.70 with Carbon, against $1.07 with GPT-6.1 Sol, roughly 15 times as much. Fable 5.2 is more expensive still at $40.20 per problem, about 40 times the cost of Sol. The best openly licensed model on the board, Kimi K3.1 Pro, scored 54.3%, far behind the three leaders.

The lead is narrower than it looks

The three leaders' uncertainty ranges overlap. MathArena reports 95% confidence intervals of plus or minus 3.0 points for Carbon, plus or minus 5.3 points for Fable 5.2 and plus or minus 2.7 points for Sol, so the order at the top is not statistically decisive. The 2.6-point gap between Carbon and Fable sits inside those ranges.

There is also a limit to what was actually measured. Because of the cost of running Carbon, MathArena ran it only on the ArXivLean evaluation, and its scores on the other evaluations are statistically estimated from its observed performance, a method the benchmark discloses. Expected performance is the average of normalised scores across all questions in the competitions currently in use, with each question weighted equally, so Carbon's headline number mixes measured results with modelled estimates.

Carbon's behaviour in testing was striking in another way. Its most common failure mode was exceeding the time limit: it almost never gave up on a task, and in one run it launched a script that ran for more than eleven hours without checking its progress. That persistence may help explain the score, and it is a practical warning for anyone running the model on real work, where time and cost matter as much as accuracy.

Google itself has not announced the ranking. MathArena's operator, ETH Zurich researcher Jasper Dekoninck, announced the change on X on 13 November, and Google chief executive Sundar Pichai and Google DeepMind chief Demis Hassabis reposted the announcement without adding a comment of their own.

The lesson for buyers is the one the benchmark's own numbers point to: rank is not cost-efficiency. A model that tops a leaderboard at 15 times the price per problem is the right choice only for work where the extra accuracy is worth the extra spend, and where the model's habit of working past time limits will not cause problems of its own.

Source: The Decoder