4 ms·

Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100

• kqr 4y ago
  Wow, this is a much more interesting answer than I expected. Thank you!
  • famouswaffles 4y ago
    It is. Although I'd be careful with the conclusions. The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here. For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196
• freilanzer 4y ago
  This seems more like an issue with the tokenizers, no? This doesn't seem more difficult than other problems it solves.
  • famouswaffles 4y ago
    You're right that LLMs can solve abstract analogical reasoning problems https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2212.09196 Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the equivalent of an artificial visual cortex. We don't have that yet