Spot on. We struggle to define what we don't understand. And with AGI, that would include understanding ourselves. And when people start dragging in philosophy,

Spot on. We struggle to define what we don't understand. And with AGI, that would include understanding ourselves. And when people start dragging in philosophy, religion, etc. you kind of know it's one of those things that definitely don't have much consensus. That doesn't mean it's all nonsense. I think objectively we are getting quite a bit of emergent definitions that emerge from established fact and technology that kind of narrow this down. LLMs seem part of the solution on a path to some form of artificial intelligence that can keep up with us and that we might struggle to keep up with. But LLMs are not the whole solution. Though what they can't do keeps shifting. From glorified autocomplete to solving mathematical problems that were previously not solved. In the space of less than 3 years since the launch of chat GPT in November 2022. If the math wasn't solved before, there has to be a bit more to it than a glorified autocomplete. Of course there are also counter points to that. But it at least challenges the notion that LLMs can't come up with new stuff. Even the notion that they can is deeply upsetting to some people because it challenges their world views. This debate is as much about that as it is about coming up with workable definitions. LLMs are obviously lacking in a lot of ways. Perhaps the most obvious thing is that after their training is completed, they stop learning and adapting. They have no memory beyond their prompts and what they learned during training. A chat gpt conversation is just a large prompt with the entire history of the conversation and some clever engineering to load that into GPU memory relatively quickly. There are a lot of party tricks that involve elaborate system prompts and other workarounds. The ability to remember and learn seems pretty fundamental to AGI, whatever that is. It's also a thing that doesn't sound like it's unsolvable. Some kind of indexed DB external to the AI for RAG kind of works but it does not solve the learning problem. And it shifts the problem to how to query and filter the data, which might not be optimal. And it's more similar to us using Google to find something than it is to us remembering information. Also, it's not like we're particularly good at cramming large amounts of information down or learning. Learning is a slow process. And we kind of get set in our ways as we age. Meaning we're reluctant to learn more stuff and less capable of doing so. That suggests that even modest improvements might have dramatic results. LLMs + some short term memory and ability to learn might end up being a lot more useful. I like duck typing in programming and that's also my mental model for AGIs. The Turing test is kind of obsolete. But if it quacks like duck and walks like a duck, it probably is a duck. Turing was onto something. Once we have a hard time telling apart the average person in society from an AI over days/weeks/years of interaction, we'll have AGIs. I think that's doable. But I'm not an ML engineer. A lot of those seem to believe it's doable too though. The rest of society in the form of self appointed arm chair professors, religious leaders, eminent philosophers, and (too put this politely) lesser qualified individuals with strong opinions makes a lot of confused noises. But it does not generate a lot of definitions that have broad consensus.