As synthetic intelligence methods started scoring extraordinarily excessive on lengthy used educational benchmarks, researchers observed a rising situation. The checks that when challenged machines had been now not troublesome sufficient. Well-known evaluations such because the Huge Multitask Language Understanding (MMLU) examination, which had beforehand been seen as demanding, now fail to correctly measure the capabilities of right this moment’s superior AI fashions.
To resolve this drawback, a worldwide group of practically 1,000 researchers, together with a professor from Texas A&M College, developed a brand new sort of check. Their aim was to construct an examination that’s broad, troublesome, and grounded in professional human information in ways in which present AI methods nonetheless battle to deal with.
The result’s “Humanity’s Final Examination” (HLE), a 2,500 query evaluation masking arithmetic, humanities, pure sciences, historic languages, and a variety of extremely specialised educational fields. Particulars of the undertaking seem in a paper revealed in Natureand extra details about the examination is out there at lastexam.ai.
Among the many many contributors is Dr. Tung Nguyen, educational affiliate professor within the Division of Pc Science and Engineering at Texas A&M. Nguyen helped write and refine lots of the examination questions.
“When AI methods begin performing extraordinarily effectively on human benchmarks, it is tempting to assume they’re approaching human-level understanding,” Nguyen mentioned. “However HLE reminds us that intelligence is not nearly sample recognition — it is about depth, context and specialised experience.”
The aim of the examination was to not trick or defeat human check takers. As a substitute, the aim was to rigorously establish areas the place AI methods nonetheless fall quick.
A World Effort to Measure AI’s Limits
Specialists from all over the world wrote and reviewed the questions included in Humanity’s Final Examination. Every drawback was rigorously designed so it has one clear, verifiable reply. The questions had been additionally crafted to stop fast options by easy web searches.
The matters come from superior educational challenges. Some duties contain translating historic Palmyrene inscriptions, whereas others require figuring out tiny anatomical constructions in birds or analyzing detailed options of Biblical Hebrew pronunciation.
Researchers examined each query in opposition to main AI methods. If any mannequin was in a position to reply a query accurately, that query was faraway from the ultimate examination. This course of ensured the check remained simply past what present AI methods can reliably resolve.
Early testing confirmed that the technique labored. Even highly effective AI fashions struggled with the examination. GPT-4o achieved a rating of two.7 %, whereas Claude 3.5 Sonnet reached 4.1 %. OpenAI’s o1 mannequin carried out considerably higher with 8 %. Essentially the most succesful methods thus far, together with Gemini 3.1 Professional and Claude Opus 4.6, have reached accuracy ranges between about 40 % and 50 %.
Why New AI Benchmarks Are Wanted
Nguyen defined that the difficulty of AI surpassing older checks is greater than a technical concern. He contributed 73 of the two,500 publicly out there questions in HLE, the second highest quantity amongst contributors, and wrote probably the most questions associated to arithmetic and laptop science.
“With out correct evaluation instruments, policymakers, builders and customers threat misinterpreting what AI methods can truly do,” he mentioned. “Benchmarks present the inspiration for measuring progress and figuring out dangers.”
In line with the analysis staff, excessive scores on checks initially designed for people don’t essentially point out real intelligence. These benchmarks primarily measure how effectively AI can full particular duties created for human learners, quite than capturing deeper understanding.
Not a Menace, however a Instrument
Regardless of the dramatic title, Humanity’s Final Examination is just not meant to recommend that people have gotten out of date. As a substitute, it highlights the big quantity of data and experience that also stays uniquely human.
“This is not a race in opposition to AI,” Nguyen mentioned. “It is a methodology for understanding the place these methods are sturdy and the place they battle. That understanding helps us construct safer, extra dependable applied sciences. And, importantly, it reminds us why human experience nonetheless issues.”
Constructing a Lengthy Time period AI Benchmark
Humanity’s Final Examination is designed to function a sturdy and clear benchmark for future AI methods. To help that aim, the researchers have launched some questions publicly whereas maintaining the bulk hidden in order that AI fashions can not merely memorize the solutions.
“For now, Humanity’s Final Examination stands as one of many clearest assessments of the hole between AI and human intelligence,” Nguyen mentioned, “and regardless of fast technological advances, it stays extensive.”
A Huge Worldwide Analysis Effort
Nguyen emphasised that the size of the undertaking demonstrates the worth of collaboration throughout disciplines and nations.
“What made this undertaking extraordinary was the size,” he mentioned. “Specialists from practically each self-discipline contributed. It wasn’t simply laptop scientists; it was historians, physicists, linguists, medical researchers. That range is strictly what exposes the gaps in right this moment’s AI methods — maybe paradoxically, it is people working collectively.”
