Former Google engineer and influential AI researcher François Chollet is co-founder of a nonprofit that goals to assist develop benchmarks that test AI for “human” intelligence.
The nonprofit ARC Prize Foundation is led by Greg Kamradt, a former technical director at Salesforce and founding father of AI product studio Leverage. Kamradt will probably be president and board member.
Fundraising for the ARC Prize Foundation will begin later in January.
“(We) are evolving… right into a true nonprofit foundation that acts as a useful north star for artificial general intelligence,” Chollet said wrote in a post on the nonprofit's website. (Artificial general intelligence is a vague term, however it is mostly understood to mean AI that may perform most tasks that humans can do.) “(We) seek to stimulate progress by bridging (the gap) in basic human intelligence Encourage skills.”
The ARC Prize Foundation will expand ARC-AGI, a test developed by Chollet to judge whether an AI system can efficiently acquire latest skills outside of the info on which it was trained. It consists of puzzle-like tasks through which an AI has to generate the right “answer” grid from a set of various coloured squares. The problems should force an AI to adapt to latest problems it hasn't seen before.
Chollet introduced ARC-AGI, short for “Abstract and Reasoning Corpus for Artificial General Intelligence,” in 2019. Many AI systems can pass Mathematical Olympiad exams and find possible solutions to doctoral-level problems. But until this 12 months, probably the most powerful AI could only solve slightly below a 3rd of the tasks in ARC-AGI.
“Unlike most frontier AI benchmarks, we don’t try and measure AI risk with superhuman exam questions,” Chollet wrote within the post. “Future versions of the ARC-AGI benchmark will deal with narrowing the (human capabilities) gap toward zero.”
Last June, Chollet and Zapier co-founder Mike Knoop launched one Competition construct an AI able to outperforming ARC-AGI. OpenAI's unpublished o3 model was the primary to attain a qualification rating – but only with a unprecedented amount of computing power.
Chollet has made it clear that ARC-AGI has flaws – many models have been capable of brute force their approach to high scores – and that he doesn't imagine o3 has human-level intelligence.
“(Early) data points suggest that the upcoming benchmark (successor to the ARC-AGI) will still pose a big challenge to o3, potentially reducing its rating to under 30% even at high computing power (while an intelligent human at that will have the option). to attain over 95% without training), Chollet said in an announcement last December. “You will know artificial general intelligence is here when the duty of making tasks which are easy for normal humans but difficult for AI becomes simply unattainable.”
button says that it plans to launch a second-generation ARC-AGI benchmark “in the primary quarter” alongside a brand new competition. The nonprofit may also begin designing the third edition of ARC-AGI.
It stays to be seen how the ARC Prize Foundation will cope with this criticism Chollet was faced with the overstatement of ARC-AGI as a benchmark for achieving AGI. The actual definition of AGI is currently hotly debated; an OpenAI worker recently claims that AGI has “already” been achieved if one defines AGI as AI that’s “higher than most humans at most tasks.”
Interestingly, OpenAI CEO Sam Altman said in December that the corporate intends to work with the ARC-AGI team to develop future benchmarks. Chollet didn’t provide any updates on potential partnerships in today's announcement.
In a series of Posts At
