Epoch AI Launches FrontierMath AI Benchmark to Take a look at Capabilities of AI Fashions

Epoch AI, a California-based analysis institute launched a brand new synthetic intelligence (AI) benchmark final week. Dubbed FrontierMath, the brand new AI benchmark assessments giant language fashions (LLMs) on their functionality of reseasoning and mathematical problem-solving. The AI agency claims that present math benchmarks aren’t very helpful as a consequence of components like knowledge contamination and AI fashions scoring very excessive scores on them. Epoch AI claims that even the main LLMs have scored lower than two p.c on the brand new benchmark.

Epoch AI Launches FrontierMath Benchmark

In a submit on X (previously often called Twitter), the AI agency defined that it collaborated with greater than 60 mathematicians to create tons of of origins and unpublished math issues. Epoch AI claims that these questions would take even mathematicians hours to resolve. The rationale behind creating the brand new benchmark was cited as the constraints with present benchmarks equivalent to GSM8K and MATH, the place AI fashions typically rating a excessive level.

The corporate claimed that the excessive scores achieved by LLMs are largely as a consequence of knowledge contamination. This implies the questions one way or the other had been already fed into the AI fashions, leading to them simply fixing the questions.

FrontierMath solves the issue by together with new issues which might be distinctive and haven’t been printed wherever, mitigating the dangers related to knowledge contamination. Additional, the benchmark contains a variety of questions together with computationally intensive issues in quantity principle, actual evaluation, and algebraic geometry, in addition to matters equivalent to Zermelo–Fraenkel set principle. The AI agency says all of the questions are “guess proof”, which means they can’t be solved by chance with out sturdy reasoning.

Epoch AI highlighted that to measure AI’s aptitude, benchmarks needs to be created on inventive problem-solving the place the AI has to take care of reasoning over a number of steps. Notably, many business veterans imagine that the present benchmarks aren’t ample to accurately measure how superior an AI mannequin is.

Responding to the brand new benchmark in a submit, Noam Brown, an OpenAI researcher who was behind the corporate’s o1 mannequin welcomed the brand new benchmark and stated, “I really like seeing a brand new eval with such low move charges for frontier fashions.”