4 ms·

NOTE IN ADVANCE: I'm generalizing, naturally, because talking about specifics would require an essay and I'm trying to write a comment. Why predict that the gr

by rmunn 1y ago

NOTE IN ADVANCE: I'm generalizing, naturally, because talking about specifics would require an essay and I'm trying to write a comment. Why predict that the growth rate is going to slow now? Simple. Because current models have already been trained on pretty much the entire meaningful part of the Internet. Where are they going to get more data? The exponential growth part of the curve was largely based on being able to fit more and more training data into the models. Now that all the meaningful training data has been fed in, further growth will come from one of two things: generating training data from one LLM to feed into another one (dangerous, highly likely to lead to "down the rabbit hole forever" hallucinations, and weeding those out is a LOT of work and will therefore contribute to slower growth), or else finding better ways to tweak the models to make better use of the available training data (which will produce growth, but much slower than what "Hey, we can slurp up the entire Internet now!" was producing in terms of rate of growth). And yes, there is more training data available because the Internet is not static: the Internet of 2025 has more meaningful, human-generated content than the Internet of 2024. But it also has a lot more AI-generated content, which will lead into the rabbit-hole problem where one AI's hallucinations get baked into the next one's training, so the extra data that can be harvested from the 2025 Internet is almost certainly going to produce slower growth in meaningful results (as opposed to hallucinated results).

theptip 1y ago
> Where are they going to get more data? This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fine. We seem to be making progress with some combination of synthetic training datasets on coding/math tasks, textbooks authored by paid experts, and new tokens (plus preference signals) generated by users of the LLM systems. It wouldn’t surprise me if coding/math turned out to have a dense-enough loss-landscape to produce enough synthetic data to get to AGI - though I wouldn’t bet on this as a highly likely outcome. I have been wanting to read/do some more rigorous analysis here though. This sort of analysis would count as the kind of rigorous prediction that I’m asking for above. E2A: initial exploration on this: https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3c7 https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3... Gives some relevant papers such as https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI
- rmunn 1y ago
  I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / fact model / whatever terminology you prefer, and is therefore not achievable by models that simply chain words together without having any kind of understanding baked into the model. In other words, LLMs cannot lead to AGI.
  - theptip 1y ago
    Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)
    - Xss3 1y ago
      Ah, so you dont know anything about how they work. Thanks for clarification.
    - meheleventyone 1y ago
      Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
      - theptip 1y ago
        The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?
ottah 1y ago
Alternative argument, there is no need for more training data, just better algorithms. Throwing more tokens at the problem doesn't solve the fact that training llms using supervised learning is a poor way to integrate knowledge. We have however seen promising results coming out of reinforcement learning and self play. Which means that anthropic and openais' bet on scale is likely a dead end, but we may yet see capability improvements coming from other labs, without the need for greater data collection.
- rmunn 1y ago
  Better algorithms is one of the things I meant by "better ways to tweak the models to make better use of the available training data". But that produces slower growth than the jaw-droppingly rapid growth you can get by slurping pretty much the whole Internet. That produced the sharp part of the S curve, but that part is behind us now, which is why I assert we're approaching the slower-growth part at the top of the curve.
sdenton4 1y ago
Curiously, humans don't seem to require reading the entire internet in order to perform at human level on a wide variety of tasks... Nature suggests that there's a lot of headroom in algorithms for learning on existing sources. Indeed, we had models trained on the whole internet a couple years ago, now, yet model quality has continued to improve. Meanwhile, on the hardware side, transistor counts in GPUs are in the tens of billions and still increasing steadily.
- meheleventyone 1y ago
  This is a time horizon thing though. Over the course of future human history AI development might look exponential but that doesn’t mean there won’t be significant plateaus. We don’t even fully understand how the human brain works so whilst the fact it does exist strongly suggests it’s replicable (and humans do it naturally) that doesn’t make it practical in any time horizon that matters to us now. Nor does there seem to be fast movement in that direction since everyone is largely working on the same underlying architecture that isn’t similar to the brain.

by rmunn 1y ago

theptip 1y ago

> Where are they going to get more data? This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fine. We seem to be making progress with some combination of synthetic training datasets on coding/math tasks, textbooks authored by paid experts, and new tokens (plus preference signals) generated by users of the LLM systems. It wouldn’t surprise me if coding/math turned out to have a dense-enough loss-landscape to produce enough synthetic data to get to AGI - though I wouldn’t bet on this as a highly likely outcome. I have been wanting to read/do some more rigorous analysis here though. This sort of analysis would count as the kind of rigorous prediction that I’m asking for above. E2A: initial exploration on this: https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3c7 https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3... Gives some relevant papers such as https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI

rmunn 1y ago
I am extremely confident that AGI, if it is achievable at all (which is a different argument and one I'm not getting into right now), requires a world model / fact model / whatever terminology you prefer, and is therefore not achievable by models that simply chain words together without having any kind of understanding baked into the model. In other words, LLMs cannot lead to AGI.
- theptip 1y ago
  Agreed, it surely does require a world-model. I disagree that generic LLMs plus CoT/reasoning/tool calling (ie the current stack) cannot in principle implement a world model. I believe LLMs are doing some sort of world modeling and likely are mostly lacking a medium-/long-term memory system in which to store it. (I wouldn’t be surprised if one or two more architectural overhauls end up occurring before AGI, I also wouldn’t be surprised if these occurred seamlessly with our current trajectory of progress)
  - Xss3 1y ago
    Ah, so you dont know anything about how they work. Thanks for clarification.
  - meheleventyone 1y ago
    Isn’t the memory the pre-trained weights that let it do anything at all? Or do you mean they should be capable of refining them in real-time (learning).
    - theptip 1y ago
      The human brain has many systems that adapt on multiple time-frames which could loosely be called “memory”. But here I’m specifically interested in real-time updates to medium/long term memory, and the episodic/consciously accessible systems that are used in human reasoning/intelligence. Eg if I’m working on a big task I can think through previous solutions I learned, remember the salient/surprising lessons, recall recent conversations that may indirectly affect requirements, etc. The brain is clearly doing an associative compression and indexing operation atop the raw memory traces. I feel the current LLM “memory” implementations are very weak compared to what the human brain does. I suppose there is a sense in which you could say the weights “remember” the training data, but it’s read-only and I think this lack of real-time updating is a crucial gap. To expand on my hunch about scaffolding - it may be that you can construct an MCP module that can let the LLM retrieve or ruminate on associative memories in such a way as to allow the LLM to not make the same mistake twice and be steerable on a longer timeframe. I think the best argument against my hunch is that human brains have systems which update the synaptic weights themselves over a timeframe of days-to-months, and so if neural plasticity is the optimal solution here then we may not be able to efficiently solve the problem with “application layer” memory plugins. But again, there is a lot of solution-space to explore; maybe some LoRA-like algorithm can allow an LLM instance to efficiently update its own weights at test-time, and persist those deltas for efficient inference, thus implementing the required neural plasticity algorithms?

ottah 1y ago

Alternative argument, there is no need for more training data, just better algorithms. Throwing more tokens at the problem doesn't solve the fact that training llms using supervised learning is a poor way to integrate knowledge. We have however seen promising results coming out of reinforcement learning and self play. Which means that anthropic and openais' bet on scale is likely a dead end, but we may yet see capability improvements coming from other labs, without the need for greater data collection.

rmunn 1y ago
Better algorithms is one of the things I meant by "better ways to tweak the models to make better use of the available training data". But that produces slower growth than the jaw-droppingly rapid growth you can get by slurping pretty much the whole Internet. That produced the sharp part of the S curve, but that part is behind us now, which is why I assert we're approaching the slower-growth part at the top of the curve.

sdenton4 1y ago

Curiously, humans don't seem to require reading the entire internet in order to perform at human level on a wide variety of tasks... Nature suggests that there's a lot of headroom in algorithms for learning on existing sources. Indeed, we had models trained on the whole internet a couple years ago, now, yet model quality has continued to improve. Meanwhile, on the hardware side, transistor counts in GPUs are in the tens of billions and still increasing steadily.

meheleventyone 1y ago
This is a time horizon thing though. Over the course of future human history AI development might look exponential but that doesn’t mean there won’t be significant plateaus. We don’t even fully understand how the human brain works so whilst the fact it does exist strongly suggests it’s replicable (and humans do it naturally) that doesn’t make it practical in any time horizon that matters to us now. Nor does there seem to be fast movement in that direction since everyone is largely working on the same underlying architecture that isn’t similar to the brain.