3 ms·
From Chollet's perspective, the LLM hype started well before, with at least GPT-2 half a year before his paper, and he spent plenty of time mocking GPT-2 on Twi
by gwern 2y ago
From Chollet's perspective, the LLM hype started well before, with at least GPT-2 half a year before his paper, and he spent plenty of time mocking GPT-2 on Twitter before he came up with ARC as a rebuttal.
- modeless 2y agoIt's a very convincing rebuttal considering that GPT-3 and GPT-4 came out after ARC but made no significant progress on it. He seemingly had the single most accurate and verifiable prediction of anyone in the world (in 2019) about exactly what type of tasks scaled LLMs would be bad at.
- gwern 2y agoWell, that's true inasmuch as every other prediction did far worse. Saying ARC did the best is passing a low bar when your competition is people like Gary Marcus or HNers saying 'yeah but no scaled-up GPT-3 could ever write a whole program'... But since ARC was from the start clearly a vision task - most of these transforms or rules make no sense without a visual geometric prior - it wasn't that convincing, and we see plenty of progress with LLMs.