4 ms·
If ARC-AGI were a good benchmark for "AGI", then MindsAI should effectively be blowing away current frontier models by order of magnitude. I don't know what Min
by krackers 2y ago
If ARC-AGI were a good benchmark for "AGI", then MindsAI should effectively be blowing away current frontier models by order of magnitude. I don't know what MindsAI is, but the post implies they're basically fine-tuning or using a very specific strategy for ARC-AGI that isn't really generalizable to other tasks.
I think it's a nice benchmark of a certain type of spatial/visual intelligence, but if you have a model or technique specifically fine-tuned for ARC-AGI then it's no longer A"G"I
- alphabetting 2y agoI clarified in a another post I mean for benchmarking standalone models, not ones fine-tuned for solving ARC
- nightski 2y agoI mean, there are a lot of tasks that frontier models excel at which many humans wouldn't be able to complete.
- drdeca 2y agoPerhaps a benchmark could be a good approximate upper bound for something without being a good approximate lower bound for that thing?
- deleted 2y ago[deleted]