Arc-AGI-2 and ARC Prize 2025

Benchmark structure and test-set security

• ARC-AGI-2 uses four sets: public train, public eval, semi-private eval, and private eval.
• Semi-private eval is shared with partners under data agreements but acknowledged as not fully secure; organizers accept the leak risk and say it’s cheaper to regenerate than to perfectly secure.
• Private eval is only on Kaggle in an offline environment; no public model testing is done on it.
• For scoring proprietary models (e.g., o3), only the semi-private set was used under no-retention agreements and dedicated hardware that was to be wiped afterward.
• Several commenters are skeptical that large labs can be trusted not to log or reuse data; concerns include misleading investors, users, and the public.

What ARC-AGI-2 is measuring (and what it isn’t)

• The benchmark aims at test-time reasoning and “fluid intelligence” on novel, visual grid tasks built from minimal “core knowledge” rather than language or world knowledge.
• Organizers state a philosophical criterion: when we can no longer invent human-easy / AI-hard quantifiable tasks, we effectively have AGI; ARC-AGI-2 is presented as evidence we’re not there.
• Critics argue this doesn’t map to everyday capabilities like cooking or driving and that embodiment and motor control are separate but practically important.
• Others see ARC more as proof that AGI has not been reached than as an eventual AGI certification test.

Human difficulty and calibration

• Every ARC-AGI-2 task was solved by at least two human testers (out of small per-task samples) in ≤2 attempts; this is intended as a fairness check, not a population-level solve rate.
• Some users find the puzzles enjoyable but far from “easy,” often needing more than two tries, and liken them to IQ-style or “aha” puzzles.
• There’s interest in formal psychometrics (e.g., what IQ level would clear most tasks quickly), but this remains unclear.

Compute, “brute force,” and novel ideas

• A major thread debates whether o3’s success on ARC-AGI-1 reflects brute-force test-time compute or genuine algorithmic progress (e.g., RL + search over chain-of-thought).
• Some argue similar search-style ideas existed for years; what’s new is their scaled application to LLMs. Others say o3’s run was so expensive it’s not a practical “solution.”
• ARC Prize now explicitly incorporates efficiency: Kaggle entries must stay within a tight compute budget (e.g., <$10k for 120 tasks), aiming for human-adjacent costs.
• Commenters note that compute budgets are a moving hardware target and often negligible in high-value domains, but also accept that unbounded compute makes “intelligence” metrics less meaningful.

Impact on general AI research

• A concern is that the prize might incentivize narrow, ARC-specific hacks rather than general intelligence.
• Organizers respond with a “paper prize” track rewarding conceptual contributions; last year saw dozens of papers, with some methods (e.g., test-time fine-tuning schemes) presented as more broadly relevant.
• Supporters see ARC as emphasizing sample-efficient learning of novel tasks, contrasting with current LLM practice of massive pretraining on static data.

Design choices, modality, and future directions

• ARC avoids natural language to minimize prior knowledge and focus on visual-spatial abstraction; organizers say tasks could be tokenized but would then involve linguistic priors.
• Some worry about circular reasoning: designing tasks to “require fluid intelligence” and then inferring fluid intelligence from performance. Others compare this to historical language benchmarks and the Turing test, arguing that benchmarks often overclaim what they measure.
• There’s mention of ARC-3 remaining 2D but becoming temporal and interactive, raising concerns that interactivity and heavy attention demands could filter out many humans.
• Related ideas appear: desire for similar out-of-domain benchmarks in computer vision, interest from cognitive/neurological perspectives on why these puzzles feel intuitive to humans, and discussion of whether “general intelligence” is even well-defined.

User experience and misc. feedback

• Several people found ARC-AGI-2 more fun than ARC-AGI-1 and used the puzzles socially (e.g., with family), while also noting that the web editor is clunky and could use drag-to-paint, brush sizing, and better tools.
• The built-in “select” tool for counting/copying is appreciated once discovered.
• There are nitpicks about typos (“pubic” vs “public”) and interest in seeing the hardest puzzles.
• One external reasoning system is claimed (via a shared screenshot) to solve at least one “hard” puzzle, but no systematic evaluation is discussed.