[2/2] François Chollet: How We Get To AGI (AI column)

Finishing. story About the performance of Francois, we need to talk about his framework ARC-* and the laboratory Ndea

6. Proposed benchmarks

• ARC-1. Released in 2019 yearly test 1000 unique tasks that require mobile intelligence rather than learned knowledge. All tasks are built on basic knowledge (objectivity, elementary physics, geometry)This is what any four-year-old child has. For five years, despite 50,000- a multiple increase in the scale of pretrained models, performance on ARC-1 grew up 0percentage 10%. This convincingly proved that mobile intelligence does not arise from simply scaling pre-training.
• ARC-2. Released in March 2025 ARC-year2 It focuses on compositional reasoning. Unlike ARC--1where many problems can be solved intuitively, ARC-2 It requires thoughtful thinking, but remains doable for people. Testing 400 People have shown that groups of 10 man could achieve 100Percent accuracy. However, the basic models show 0% accuracy, static reasoning systems 1-2%, and even advanced systems with TTA are significantly inferior to the human level. - ARC-3. Scheduled for release at the beginning 2026 ARC-year3 This is a major departure from the entry-exit format. Instead, it evaluates agency—the ability to explore, learn interactively, and autonomously set goals. The AI will be placed in a whole new environment where it doesn't know what the controls are doing, what the goal is, and what the rules of the game are. Efficiency will be the central criterion – models must solve problems with the same level of efficiency as humans. 7. Future direction: Ndea Laboratory At the end of the talk, François unveiled his new research lab, Ndea, which focuses on building AI capable of independent invention and discovery. Ndea’s approach is to create a programmable meta-learner who, when confronted with a new task, synthesizes a program adapted to that task. This program will integrate deep learning sub-modules for tasks such as 1 (perception) and algorithmic modules for type problems 2 (reasoning). The system will use a global library of reusable abstraction building blocks, which is constantly evolving in the process of learning new tasks. The first milestone will be the solution of the ARC-AGI system, which initially knows nothing about this benchmark.

Scholle’s vision looks impressive, I really enjoyed the performance – I learned a lot of new + added a lot of new material to the to read list.

#AI #ML #Software #Architecture #Processes