3 ms·

Can someone provide insight into why there's so much insistence from business that magic happens at scale with LLMs? We're a long way from AGI. The lack of mea

Can someone provide insight into why there's so much insistence from business that magic happens at scale with LLMs? We're a long way from AGI. The lack of meaningful details in these announcements makes me pessimistic.

mewpmewp2 3y ago
How do you know how long away we are?
- digging 3y ago
  I imagine you don't expect a serious answer to that anyway, but to be clear: anyone talking about AI timelines seriously would not be expressing so much certainty because it's not possible to know AI timelines with certainty right now.
skepticATX 3y ago
I don’t think that’s Meta’s viewpoint, considering FAIR is run by Yann LeCun who has been quite vocal about the limitations of what we currently have.
two_in_one 3y ago
> why there's so much insistence from business that magic happens at scale with LLMs It's already happening. See latest Google layoffs. They are automating a lot of things. Most people don't realize it, but the change is going to be dramatic. > We're a long way from AGI This is a big question, what is AGI? LLMs are quite generic and 'intelligent'. Not human-like, but. Next is going to be incremental evolution. Till we find other, non-verbal, ways of 'thinking' and put them together. That's going to be a breakthrough. Interesting, terminator-like embodiment isn't a requirement for AGI, nor is stable 'personality'.
- x86x87 3y ago
  Boy, we must have a totally different understanding of whay AGI is. Intelligence is not about parroting an answer you've seen before. It's about using your environment to gain an evolutionary advantage.
  - digging 3y ago
    > It's about using your environment to gain an evolutionary advantage. I don't think that's right either (it sounds like a description of adaptation), and I don't think your description of LLMs is fair, even though I'm fairly sure they're not AGI and won't scale to AGI. Intelligence is more like the ability to generalize skills, applying knowledge gained in one scenario to another scenario.
    - x86x87 3y ago
      https://www.merriam-webster.com/dictionary/intelligence https://www.merriam-webster.com/dictionary/intelligence the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests) Manipulating the environment afaik is to increase your chances of survival.
    - two_in_one 3y ago
      > Intelligence is more like the ability to generalize skills, applying knowledge gained in one scenario to another scenario. Hmm.. you are talking about LLMs.. They are the most generic thing we have right now (Jan 2024) LLMs have limitations, like learning on the fly isn't their strong side. But the same with brain, it consists of limited components, that's only together they work well. LLMs can be a part of the solution, if we can't find something better.
  - two_in_one 3y ago
    > using your environment to gain an evolutionary advantage. That's more like robotics. Except for evolution part. Does AGI require breeding? Software can easily multiply itself. That's hardware is the problem then.
    - x86x87 3y ago
      Think about what incentives do you have to live - it may sound rough but pain and ultimately death are the ultimate things everyone is trying to avoid. Intelligence emerges as you are trying to survive longer. Reproduction is the ultimate way of cheating death. Environment does not necessarily needs to mean physical environment, but until the "AI" does not recognize that it is in danger of nonexisting and starts to behave in ways to avoid it it cannot, imho, make the leap to AGI and it's just a really sophisticated tool.
- LudwigNagasena 3y ago
  > See latest Google layoffs. They are automating a lot of things. Most people don't realize it, but the change is going to be dramatic. What exactly did they automate?
  - sublinear 3y ago
    > What exactly did they automate? Their expenses haha
  - two_in_one 3y ago
    My guess paperwork. As they cut jobs in ads, where may things can be done programmatically now.
dwaltrip 3y ago
LLM scaling laws are pretty well established at this point. They probably won’t hold forever but we aren’t at the breaking point yet. Some more pressing questions are: * What new capabilities emerge as the models get better and better at predicting (i.e. loss goes down)? * How much will it cost to train increasingly large models? And to run inference on them? * How difficult will it be to find or generate more and more high quality data?
- gardenhedge 3y ago
  > LLM scaling laws are pretty well established at this point what are they then? I thought everyone was firmly in the "let's train with more data and see what happens" camp
  - esafak 3y ago
    https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_scaling_(Hoffmann,_et_al,_2022) https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_...
  - dwaltrip 3y ago
    https://openai.com/research/scaling-laws-for-neural-language-models https://openai.com/research/scaling-laws-for-neural-language... https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361
  - kilotaras 3y ago
    Scaling laws in terms of loss are well established. How loss translates into higher-level capabilities is anyone's guess.

mewpmewp2 3y ago

How do you know how long away we are?

digging 3y ago
I imagine you don't expect a serious answer to that anyway, but to be clear: anyone talking about AI timelines seriously would not be expressing so much certainty because it's not possible to know AI timelines with certainty right now.

skepticATX 3y ago

I don’t think that’s Meta’s viewpoint, considering FAIR is run by Yann LeCun who has been quite vocal about the limitations of what we currently have.

two_in_one 3y ago

> why there's so much insistence from business that magic happens at scale with LLMs It's already happening. See latest Google layoffs. They are automating a lot of things. Most people don't realize it, but the change is going to be dramatic. > We're a long way from AGI This is a big question, what is AGI? LLMs are quite generic and 'intelligent'. Not human-like, but. Next is going to be incremental evolution. Till we find other, non-verbal, ways of 'thinking' and put them together. That's going to be a breakthrough. Interesting, terminator-like embodiment isn't a requirement for AGI, nor is stable 'personality'.

x86x87 3y ago
Boy, we must have a totally different understanding of whay AGI is. Intelligence is not about parroting an answer you've seen before. It's about using your environment to gain an evolutionary advantage.
- digging 3y ago
  > It's about using your environment to gain an evolutionary advantage. I don't think that's right either (it sounds like a description of adaptation), and I don't think your description of LLMs is fair, even though I'm fairly sure they're not AGI and won't scale to AGI. Intelligence is more like the ability to generalize skills, applying knowledge gained in one scenario to another scenario.
  - x86x87 3y ago
    https://www.merriam-webster.com/dictionary/intelligence https://www.merriam-webster.com/dictionary/intelligence the ability to apply knowledge to manipulate one's environment or to think abstractly as measured by objective criteria (such as tests) Manipulating the environment afaik is to increase your chances of survival.
  - two_in_one 3y ago
    > Intelligence is more like the ability to generalize skills, applying knowledge gained in one scenario to another scenario. Hmm.. you are talking about LLMs.. They are the most generic thing we have right now (Jan 2024) LLMs have limitations, like learning on the fly isn't their strong side. But the same with brain, it consists of limited components, that's only together they work well. LLMs can be a part of the solution, if we can't find something better.
- two_in_one 3y ago
  > using your environment to gain an evolutionary advantage. That's more like robotics. Except for evolution part. Does AGI require breeding? Software can easily multiply itself. That's hardware is the problem then.
  - x86x87 3y ago
    Think about what incentives do you have to live - it may sound rough but pain and ultimately death are the ultimate things everyone is trying to avoid. Intelligence emerges as you are trying to survive longer. Reproduction is the ultimate way of cheating death. Environment does not necessarily needs to mean physical environment, but until the "AI" does not recognize that it is in danger of nonexisting and starts to behave in ways to avoid it it cannot, imho, make the leap to AGI and it's just a really sophisticated tool.
LudwigNagasena 3y ago
> See latest Google layoffs. They are automating a lot of things. Most people don't realize it, but the change is going to be dramatic. What exactly did they automate?
- sublinear 3y ago
  > What exactly did they automate? Their expenses haha
- two_in_one 3y ago
  My guess paperwork. As they cut jobs in ads, where may things can be done programmatically now.

dwaltrip 3y ago

LLM scaling laws are pretty well established at this point. They probably won’t hold forever but we aren’t at the breaking point yet. Some more pressing questions are: * What new capabilities emerge as the models get better and better at predicting (i.e. loss goes down)? * How much will it cost to train increasingly large models? And to run inference on them? * How difficult will it be to find or generate more and more high quality data?

gardenhedge 3y ago
> LLM scaling laws are pretty well established at this point what are they then? I thought everyone was firmly in the "let's train with more data and see what happens" camp
- esafak 3y ago
  https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_scaling_(Hoffmann,_et_al,_2022) https://en.wikipedia.org/wiki/Neural_scaling_law#Chinchilla_...
- dwaltrip 3y ago
  https://openai.com/research/scaling-laws-for-neural-language-models https://openai.com/research/scaling-laws-for-neural-language... https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361
- kilotaras 3y ago
  Scaling laws in terms of loss are well established. How loss translates into higher-level capabilities is anyone's guess.