4 ms·

I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / f

by rmunn 1y ago

I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / fact model / whatever terminology you prefer, and is therefore not achievable by models that simply chain words together without having any kind of understanding baked into the model. In other words, LLMs cannot lead to AGI.

theptip 1y ago
Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)
- Xss3 1y ago
  Ah, so you dont know anything about how they work. Thanks for clarification.
- meheleventyone 1y ago
  Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
  - theptip 1y ago
    The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?

by rmunn 1y ago

theptip 1y ago

Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)

Xss3 1y ago
Ah, so you dont know anything about how they work. Thanks for clarification.
meheleventyone 1y ago
Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
- theptip 1y ago
  The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?