4 ms·

The model's arent there yet. I am confused how people cannot extrapolate into the future and understand that the models will improve

lxgr 2y ago
And I'm surprised how many people think they can confidently tell whether they're looking at an s-curve or an exponential function based on very limited data points. I don't even doubt that superintelligence is a very real possibility! But it might or might not happen, and if it does, it might or might not be based on deep learning. As a counterexample: The maximum speed of travel for the average person for millenia used to be as fast as they could run, then it was as fast as the fastest horse can run, and then within a century it has accelerated to almost the speed of sound – at which it has plateaued. Looking purely at the decades of acceleration, you might have very well concluded from the data that we'd be making significant headway towards getting within double-digit percentages of the speed of light at this point.
- danielmarkbruce 2y ago
  Maybe don't think of it as curves or functions. Just go through an LLM and think about all the things that could be improved. It's a long list once you get into the details. By and large...sota models are: a) trained on crappy data, including questionable RLHF feedback. b) trained with questionable embedding layers. c) trained with questionable loss functions d) trained with questionable optimizers e) trained at questionable precision (somewhat related to d) f) are very big which stops fast iteration around all the above. It's kinda like semiconductors. You don't have to think of it as a curve - just ask people who are really close to them and they'll have a laundry list of stupid stuff which is currently done and will likely be improved upon over time.
  - lxgr 2y ago
    I don't doubt that there's tons of work ahead to even just integrate current-day capabilities of LLMs into our society and economy. But when talking about future growth potential, I don't think you can get around making assumptions about the shape of the growth function.
    - danielmarkbruce 2y ago
      You can't write a nice article about it. But you can talk about it with a simple "things are going to get a loooot better". The problem with thinking about progress in functions is it suggests some underlying law exists, and there isn't one. Even if someone could point to a function and say "10x better" by 2030 - what does that even mean in the context of an LLM for example?
blackbear_ 2y ago
https://xkcd.com/605 https://xkcd.com/605

lxgr 2y ago

And I'm surprised how many people think they can confidently tell whether they're looking at an s-curve or an exponential function based on very limited data points. I don't even doubt that superintelligence is a very real possibility! But it might or might not happen, and if it does, it might or might not be based on deep learning. As a counterexample: The maximum speed of travel for the average person for millenia used to be as fast as they could run, then it was as fast as the fastest horse can run, and then within a century it has accelerated to almost the speed of sound – at which it has plateaued. Looking purely at the decades of acceleration, you might have very well concluded from the data that we'd be making significant headway towards getting within double-digit percentages of the speed of light at this point.

danielmarkbruce 2y ago
Maybe don't think of it as curves or functions. Just go through an LLM and think about all the things that could be improved. It's a long list once you get into the details. By and large...sota models are: a) trained on crappy data, including questionable RLHF feedback. b) trained with questionable embedding layers. c) trained with questionable loss functions d) trained with questionable optimizers e) trained at questionable precision (somewhat related to d) f) are very big which stops fast iteration around all the above. It's kinda like semiconductors. You don't have to think of it as a curve - just ask people who are really close to them and they'll have a laundry list of stupid stuff which is currently done and will likely be improved upon over time.
- lxgr 2y ago
  I don't doubt that there's tons of work ahead to even just integrate current-day capabilities of LLMs into our society and economy. But when talking about future growth potential, I don't think you can get around making assumptions about the shape of the growth function.
  - danielmarkbruce 2y ago
    You can't write a nice article about it. But you can talk about it with a simple "things are going to get a loooot better". The problem with thinking about progress in functions is it suggests some underlying law exists, and there isn't one. Even if someone could point to a function and say "10x better" by 2030 - what does that even mean in the context of an LLM for example?

blackbear_ 2y ago

https://xkcd.com/605 https://xkcd.com/605