Some very hand-wavey (and late) thoughts from an outsider: The current batch of LLMs can be uncharitably summarized as "just predict the next token". They're p

Some very hand-wavey (and late) thoughts from an outsider: The current batch of LLMs can be uncharitably summarized as "just predict the next token". They're pretty good at that. If they were perfect at it, they'd enable AGI - but it doesn't look like they're going to get there. It seems like the wrong approach. Among other issues, finite context windows seem like a big limitation (even though they're being expanded), and recursive summarization is an interesting kludge. The ARC-AGI tasks seem more about pattern matching, in the abstract sense (but also literally). Humans are good at pattern matching, and we seem to use pattern matching test performance as a proxy for measuring human intelligence (like in "IQ" tests). I'm going to side-step the question of "what is intelligence, really?" by defining it as being good at solving ARC-AGI tasks. I don't know what the solution is, but I have some idea of what it might look like - a machine with high-order pattern-matching capabilities. "high-order" as in being able to operate on multiple granularities/abstraction-levels at once (there are parallels here to recursive summarization in LLMs). So what is the difference between "pattern matching" and "token prediction"? They're closely related, and you could use one to do the other. But the real difference is that in pattern matching there are specific patterns that you're matching against. If you're lucky you can even name the pattern/trope, but it might be something more abstract and nameless. These patterns can be taught explicitly, or inferred from the environment (i.e. "training data"). On the other hand, "token prediction" (as implemented today) is more of a probabilistic soup of variables. You can ask an LLM why it gave a particular answer and it will hallucinate something plausible for you, but the real answer is just "the weights said so". But a hypothetical pattern matching machine could tell you which pattern(s) it was matching against, and why. So to summarize (hah), I think a good solution will involve high-order meta-pattern matching capabilities (natively, not emulated or kludged via an LLM-shaped interface). I have no idea how to get there!