“Nothing Could Solve It”: Brutal New AGI Test Exposes Alarming Limits as Top AI Models Fail Spectacularly

The Arc Prize Foundation has recently introduced a groundbreaking test, the ARC-AGI-2, designed to evaluate the general intelligence of top-tier AI models. Developed by François Chollet, a notable figure in AI research, this test presents a formidable challenge to existing AI systems. Despite the prowess of models like OpenAI’s o1-pro, DeepSeek’s R1, and GPT-4.5, their performance on this test is surprisingly low. The introduction of ARC-AGI-2 marks a significant advancement in assessing AI capabilities, aiming to push boundaries and redefine what we consider intelligent behavior in machines.

The ARC-AGI-2 Test: A New Benchmark in AI

ARC-AGI-2 represents a significant shift in how we measure AI intelligence. Unlike previous benchmarks, this test focuses on an AI’s ability to identify visual patterns from a series of colored squares and generate a correct answer grid. This approach challenges AI models to adapt to unfamiliar problems, moving beyond traditional data training methods. The Arc Prize Foundation’s intention is to create a more accurate reflection of an AI’s capability to learn and innovate independently.

Current AI models have struggled with ARC-AGI-2. The leaderboard shows top reasoning models like OpenAI’s o1-pro and DeepSeek’s R1 scoring a mere 1% to 1.3%. Even powerful non-reasoning models such as GPT-4.5 and Claude 3.7 Sonnet hover around the 1% mark. These results starkly contrast with the human baseline, where participants averaged 60% accuracy. This gap underscores the challenges AI faces in stepping beyond programmed responses to genuine problem-solving.

Enhancements Over ARC-AGI-1

The introduction of ARC-AGI-2 addresses several shortcomings identified in its predecessor, ARC-AGI-1. The previous version allowed AI models to use brute force methods, leveraging immense computing power to solve problems, which did not accurately reflect intelligence. François Chollet and his team have now introduced the concept of efficiency into ARC-AGI-2. This new metric assesses not only the ability to solve tasks but also the cost of achieving those solutions.

This change challenges AI developers to focus on creating models that can interpret patterns dynamically rather than relying on memorized data. The emphasis on efficiency is crucial, as it aligns AI capabilities more closely with human cognitive processes. Greg Kamradt, co-founder of the Arc Prize Foundation, highlights that intelligence encompasses both problem-solving and the efficiency of applying such skills, broadening the criteria for evaluating AI models.

AI Models Face New Challenges

The ARC-AGI-2 test has exposed the limitations of existing AI models. OpenAI’s advanced reasoning model, o3, which excelled in ARC-AGI-1, manages only 4% accuracy on the new test despite using $200 worth of computing power per task. This stark performance drop indicates that previous success was heavily reliant on computational force rather than genuine intelligence.

The challenge of ARC-AGI-2 is not just in achieving high scores but doing so with minimal resources. The new metric forces developers to rethink their strategies, moving towards more human-like problem-solving approaches. The test’s introduction coincides with industry calls for more robust benchmarks to measure AI’s creative and adaptive capabilities, pushing the frontier of artificial general intelligence.

The Path Forward: Innovation and Competition

With ARC-AGI-2 setting a new standard, the Arc Prize Foundation has launched the Arc Prize 2025 contest. This competition dares developers to achieve 85% accuracy on the test while limiting spending to $0.42 per task. Such stringent criteria aim to foster innovation, encouraging the development of more efficient and intelligent AI models.

This initiative aligns with the broader tech industry’s pursuit of advanced benchmarks to gauge AI progress. As leaders like Hugging Face’s Thomas Wolf note, the need for diverse and comprehensive tests is vital. The Arc Prize Foundation’s efforts reflect a commitment to refining AI evaluation, ensuring that future models are not only powerful but also truly intelligent and resource-efficient.

As ARC-AGI-2 reshapes the landscape of AI testing, it prompts critical questions about the future of artificial intelligence. Can these new benchmarks inspire the next generation of AI models to bridge the gap between computational prowess and authentic intelligence? How will developers innovate to meet the challenges posed by efficiency-focused metrics? The journey towards creating truly intelligent systems continues, with ARC-AGI-2 marking a pivotal step forward.

Did you like it? 4.4/5 (25)