New math benchmark reveals AI models confidently solve problems that have no solution

A team of 64 mathematicians constructed a new AI benchmark called SOOHAK, comprising 439 handwritten mathematical tasks, of which 99 were intentionally designed to be unsolvable. This test aims to evaluate AI models not only in solving problems but also in recognizing whether the problem itself is valid. Currently, Google's Gemini 3 Pro leads in research-level problems but achieves only a 30% accuracy rate. More notably, in identifying unsolvable tasks, no model surpasses 50% accuracy. The research found that increasing computational resources enhances models' problem-solving abilities but does not improve their capability to identify unsolvable problems. The purpose of the SOOHAK benchmark is to explicitly quantify the significant gap that exists in current AI systems between sporadic flashes of brilliance and comprehensive mastery of research skills.