Your observation that the LLM is probing through humans is astute. Ilya talked at great legnth at the nVidia fireside chat that the primary intelligence goal n

Your observation that the LLM is probing through humans is astute. Ilya talked at great legnth at the nVidia fireside chat that the primary intelligence goal now is building out the world model through multiple observations. My own research team has found that we can fill in world models at micro levels that are missing from the LLM with human input and given enough memory you GPT4 has reached sufficient reasoning capabilities to add that it its world model without re-training. This to me, is the breakaway point. You simply need enough human probes indexing the smaller unpublished details of the world plus efficient memory lookup and storage and you have AGI. However, my own personal theory is that even if you achieve that, we will find limitations in giving a highest order (superposition) answer to big questions based on that detailed of a world model. The LLM will struggle to interact with humans in a meaningful way because its superposition will be a full order higher than any other humans individual perspective. This will result in answers that might be AGI, but humans won't recognize as AGI -- because they simply won't understand the super-order perspective they are computed from.