4 ms·

> Where are they going to get more data? This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fi

> Where are they going to get more data? This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fine. We seem to be making progress with some combination of synthetic training datasets on coding/math tasks, textbooks authored by paid experts, and new tokens (plus preference signals) generated by users of the LLM systems. It wouldn’t surprise me if coding/math turned out to have a dense-enough loss-landscape to produce enough synthetic data to get to AGI - though I wouldn’t bet on this as a highly likely outcome. I have been wanting to read/do some more rigorous analysis here though. This sort of analysis would count as the kind of rigorous prediction that I’m asking for above. E2A: initial exploration on this: https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3c7 https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3... Gives some relevant papers such as https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI

rmunn 1y ago
I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / fact model / whatever terminology you prefer, and is therefore not achievable by models that simply chain words together without having any kind of understanding baked into the model. In other words, LLMs cannot lead to AGI.
- theptip 1y ago
  Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)
  - Xss3 1y ago
    Ah, so you dont know anything about how they work. Thanks for clarification.
  - meheleventyone 1y ago
    Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
    - theptip 1y ago
      The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?

rmunn 1y ago

I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / fact model / whatever terminology you prefer, and is therefore not achievable by models that simply chain words together without having any kind of understanding baked into the model. In other words, LLMs cannot lead to AGI.

theptip 1y ago
Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)
- Xss3 1y ago
  Ah, so you dont know anything about how they work. Thanks for clarification.
- meheleventyone 1y ago
  Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
  - theptip 1y ago
    The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?