5 ms·
I guess I get to brag now. ARC AGI has no real defences against Big Data, memorisation-based approaches like LLMs. I told you so: https://news.ycombinator.com/
by YeGoblynQueenne 2y ago
I guess I get to brag now. ARC AGI has no real defences against Big Data, memorisation-based approaches like LLMs. I told you so:
https://news.ycombinator.com/item?id=42344336 https://news.ycombinator.com/item?id=42344336
And that answers my question about fchollet's assurances that LLMs without TTT (Test Time Training) can't beat ARC AGI:
[me] I haven't had the chance to read the papers carefully. Have they done ablation studies? For instance, is the following a guess or is it an empirical result?
[fchollet] >> For instance, if you drop the TTT component you will see that these large models trained on millions of synthetic ARC-AGI tasks drop to <10% accuracy.
- Vecr 2y agoHow are the Bongard Problems going?
- YeGoblynQueenne 2y agoThey're chilling it out together with Nethack in the Club for AI Benchmarks yet to be Beaten. Interestingly, Bongard problems do not have a private test set, unlike ARC-AGI. Can that be because they don't need it? Is it possible that Bongard Problems are a true test of (visual) reasoning that requires intelligence to be solved? Ooooh! Frisson of excitement! But I guess it's just that nobody remembers them and so nobody has seriously tried to solve them with Big Data stuff.