3 ms·
It's a useless meaningless benchmark though, it just got a catchy name, as in, if the models solve this it means they have "AGI", which is clearly rubbish. Arc
by saberience 8mo ago
It's a useless meaningless benchmark though, it just got a catchy name, as in, if the models solve this it means they have "AGI", which is clearly rubbish.
Arc-AGI score isn't correlated with anything useful.
- jabedude 8mo agohow would we actually objectively measure a model to see if it is AGI if not with benchmarks like arc-AGI?
- WarmWash 8mo agoGive it a prompt like >can u make the progm for helps that with what in need for shpping good cheap products that will display them on screen and have me let the best one to get so that i can quickly hav it at home And get back an automatic coupon code app like the user actually wanted.
- Legend2440 8mo agoIt's correlated with the ability to solve logic puzzles. It's also interesting because it's very very hard for base LLMs, even if you try to "cheat" by training on millions of ARC-like problems. Reasoning LLMs show genuine improvement on this type of problem.
- HDThoreaun 8mo agoARC-AGI 2 is an IQ test. IQ tests have been shown over and over to have predictive power in humans. People who score well on them tend to be more successful
- fsh 8mo agoIQ tests only work if the participants haven't trained for them. If they do similar tests a few times in a row, scores increase a lot. Current LLMs are hyper-optimized for the particular types of puzzles contained in popular "benchmarks".