9 ms·

I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion: We're reaching scaling limits with transformers. The number of parameters

by cs702 1y ago

I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion: We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the order of D². OpenAI and Google were the first to show these transformer "scaling laws." We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. As the OP puts it, if we want to increase the number of GPUs by 2x, we must also increase the number of parameters and training tokens by 1.41x, but... we've already run out of training tokens. We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams).

FloorEgg 1y ago
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
- jvanderbot 1y ago
  Play in the real world generates a data point every few minutes. Seems a bit slow?
  - FloorEgg 1y ago
    What are you basing that statement on? What exactly are you considering a "data point"? Are you assuming one model = one agent instance? I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.
  - pizzly 1y ago
    Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.
    - mannykannot 1y ago
      That's true, but still, a single individual or small group living isolated in the real world will, over a lifetime, learn only a tiny fraction of what we can learn from the written knowledge accumulated over millennia. Having said that, I tend to agree that having AI interact with the world may be key: for one thing, I'm not sure whether there is any sense in which LLMs understand that most of the information content of language is about an external world.
- tliltocatl 1y ago
  That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.
  - FloorEgg 1y ago
    Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate). Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.
    - airstrike 1y ago
      > Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. I believe you but I would love to know where this number came from just so I can read more about it
      - FloorEgg 1y ago
        It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³ # If there are any bioinformaticians around please come eviscerate or confirm this calc # Then compared it to TSMC 2nm research macro of (38.1 Mbit/mm^2) normalized to cell scale: 0.00019 Mbit/μm³ Living Cells: 1–10 Mbit/μm³ Current best chips: 0.00019 Mbit/μm³ https://research.tsmc.com/page/memory/4.html https://research.tsmc.com/page/memory/4.html
antirez 1y ago
This is true for the pre-training step. What if advancements in the reinforcement learning steps performed later may benefit from more compute and more models parameters? If right now the RL steps only help with sampling, that is, they only optimize the output of a given possible reply instead of the other (there are papers pointing at this: that if you generate many replies with just the common sampling methods, and you can verify correctness of the reply, then you discover that what RL helps with is selecting what was already potentially within the model output) this would be futile. But maybe advancements in the RL will do to LLMs what AlphaZero-alike models did with Chess/Go.
- cs702 1y ago
  It's possible. We're talking about pretraining meaningfully larger models past the point at which they plateau, only to see if they can improve beyond that plateau with RL. Call it option (3). No one knows if it would work, and it would be very expensive, so only the largest players can try it, but why the heck not?
charleshn 1y ago
> We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. > We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams). Of course we can, this is a non issue. See e.g. AlphaZero [0] that's 8 years old at this point, and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1]. [0] https://en.m.wikipedia.org/wiki/AlphaZero https://en.m.wikipedia.org/wiki/AlphaZero [1] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948
- jeremyjh 1y ago
  AlphaZero trained itself through chess games that it played with itself. Chess positions have something very close to an objective truth about the evaluation, the rules are clear and bounded. Winning is measurable. How do you achieve this for a language model? Yes, distillation is a thing but that is more about compression and filtering. Distillation does not produce new data in the same way that chess games produce new positions.
  - voxic11 1y ago
    Synthetic data is already widely used to do training in the programming and mathematics domains where automated verification is possible. Here is an example of an open source verified reasoning synthetic dataset https://www.primeintellect.ai/blog/synthetic-1 https://www.primeintellect.ai/blog/synthetic-1
    - jeremyjh 1y ago
      Are they actually producing new data though? This is the sort of thing I called "compression and filtering" because it seems to be new information content is not being produced, but LLMs are used to distill the information we already have. We need more raw information.
      - voxic11 1y ago
        Yes this is new synthetic data which did not exist before. I encourage you to read the link.
dosnem 1y ago
This seems so simple but I’m totally not understanding it.. If C = D^2, and you double compute, then 2C ==> 2D^2. How do you and the original author get 1.41D from 2D^2?
- yberreby 1y ago
  If C ~ D^2, then D ~ sqrt(C). In other words, the required amount of data scales with the square root of the compute. The square root of 2 ~= 1.414. If you double the compute, you need roughly 1.414 times more data.
  - dosnem 1y ago
    Thanks for clarification!

by cs702 1y ago

FloorEgg 1y ago

What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).

jvanderbot 1y ago
Play in the real world generates a data point every few minutes. Seems a bit slow?
- FloorEgg 1y ago
  What are you basing that statement on? What exactly are you considering a "data point"? Are you assuming one model = one agent instance? I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.
- pizzly 1y ago
  Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.
  - mannykannot 1y ago
    That's true, but still, a single individual or small group living isolated in the real world will, over a lifetime, learn only a tiny fraction of what we can learn from the written knowledge accumulated over millennia. Having said that, I tend to agree that having AI interact with the world may be key: for one thing, I'm not sure whether there is any sense in which LLMs understand that most of the information content of language is about an external world.
tliltocatl 1y ago
That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.
- FloorEgg 1y ago
  Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate). Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.
  - airstrike 1y ago
    > Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. I believe you but I would love to know where this number came from just so I can read more about it
    - FloorEgg 1y ago
      It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³ # If there are any bioinformaticians around please come eviscerate or confirm this calc # Then compared it to TSMC 2nm research macro of (38.1 Mbit/mm^2) normalized to cell scale: 0.00019 Mbit/μm³ Living Cells: 1–10 Mbit/μm³ Current best chips: 0.00019 Mbit/μm³ https://research.tsmc.com/page/memory/4.html https://research.tsmc.com/page/memory/4.html

antirez 1y ago

This is true for the pre-training step. What if advancements in the reinforcement learning steps performed later may benefit from more compute and more models parameters? If right now the RL steps only help with sampling, that is, they only optimize the output of a given possible reply instead of the other (there are papers pointing at this: that if you generate many replies with just the common sampling methods, and you can verify correctness of the reply, then you discover that what RL helps with is selecting what was already potentially within the model output) this would be futile. But maybe advancements in the RL will do to LLMs what AlphaZero-alike models did with Chess/Go.

cs702 1y ago
It's possible. We're talking about pretraining meaningfully larger models past the point at which they plateau, only to see if they can improve beyond that plateau with RL. Call it option (3). No one knows if it would work, and it would be very expensive, so only the largest players can try it, but why the heck not?

charleshn 1y ago

> We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. > We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams). Of course we can, this is a non issue. See e.g. AlphaZero [0] that's 8 years old at this point, and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1]. [0] https://en.m.wikipedia.org/wiki/AlphaZero https://en.m.wikipedia.org/wiki/AlphaZero [1] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948

jeremyjh 1y ago
AlphaZero trained itself through chess games that it played with itself. Chess positions have something very close to an objective truth about the evaluation, the rules are clear and bounded. Winning is measurable. How do you achieve this for a language model? Yes, distillation is a thing but that is more about compression and filtering. Distillation does not produce new data in the same way that chess games produce new positions.
- voxic11 1y ago
  Synthetic data is already widely used to do training in the programming and mathematics domains where automated verification is possible. Here is an example of an open source verified reasoning synthetic dataset https://www.primeintellect.ai/blog/synthetic-1 https://www.primeintellect.ai/blog/synthetic-1
  - jeremyjh 1y ago
    Are they actually producing new data though? This is the sort of thing I called "compression and filtering" because it seems to be new information content is not being produced, but LLMs are used to distill the information we already have. We need more raw information.
    - voxic11 1y ago
      Yes this is new synthetic data which did not exist before. I encourage you to read the link.

dosnem 1y ago

This seems so simple but I’m totally not understanding it.. If C = D^2, and you double compute, then 2C ==> 2D^2. How do you and the original author get 1.41D from 2D^2?

yberreby 1y ago
If C ~ D^2, then D ~ sqrt(C). In other words, the required amount of data scales with the square root of the compute. The square root of 2 ~= 1.414. If you double the compute, you need roughly 1.414 times more data.
- dosnem 1y ago
  Thanks for clarification!