https://open.substack.com/pub/dwarkesh

“Okay, today, I have the pleasure to speak with Francois Chollet, who is an AI researcher at Google and creator of Keras. And he’s launching a prize in collaboration with Mike Knoop, the co-founder of Zapier, who we’ll also be talking to in a second, a million dollar prize to solve the ARC benchmark that he created. So first question, what is the ARC benchmark?

And why do you even need this prize? Why won’t the biggest alum we have in a year be able to just saturate it?

Sure. So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from most LM benchmarks out there is that it’s designed to be resistant to memorization.

So if you look at the way LMs work, they’re basically this big interpolative memory. And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them. And by contrast, ARC does not require a lot of knowledge at all.

It’s designed to only require what’s known as core knowledge, which is basic knowledge about things like elementary physics, objectness, counting, that sort of thing. The[…]”

From Dwarkesh Podcast: Francois Chollet, Mike Knoop – LLMs won’t lead to AGI – $1,000,000 Prize to find true solution, Jun 11, 2024
https://podcasts.apple.com/ca/podcast/francois-chollet-mike-knoop-llms-wont-lead-to-agi-%241/id1516093381?i=1000658672649
This material may be protected by copyright.