4 ms·

Most people didn't think we were anywhere close to LLM's five years ago. The capabilities we have now were expected to be a decades away, depending on who you t

Most people didn't think we were anywhere close to LLM's five years ago. The capabilities we have now were expected to be a decades away, depending on who you talked to. [EDIT: sorry, I should have said 10 years ago... recent years get too compressed in my head and stuff from 2020 still feels like it was 2 years ago!] So I think a lot of people now don't see what the path is to AGI, but also realize they hadn't seen the path to LLM's, and innovation is coming fast and furious. So the most honest answer seems to be, it's entirely plausible that AGI just depends on another couple conceptual breakthroughs that are imminent... and it's also entirely plausible that AGI will require 20 different conceptual breakthroughs all working together that we'll only figure out decades from now. True honesty requires acknowledging that we truly have no idea. Progress in AI is happening faster than ever before, but nobody has the slightest idea how much progress is needed to get to AGI.

airstrike 1y ago
Notwithstanding the fact that AGI is a significantly higher bar than "LLM", this argument is illogical. Nobody thought we were anywhere closer to me jumping off the Empire State Building and flying across the globe 5 years ago, but I'm sure I will. Wish me luck as I take that literal leap of faith tomorrow.
- JoelMcCracken 1y ago
  what's super weird to me is how people seem to look at LLM output and see: "oh look it can think! but then it fails sometimes! how strange, we need to fix the bug that makes the thinking no workie" instead of: "oh, this is really weird. Its like a crazy advanced pattern recognition and completion engine that works better than I ever imagined such a thing could. But, it also clearly isn't _thinking_, so it seems like we are perhaps exactly as far from thinking machines as we were before LLMs"
  - famouswaffles 1y ago
    By that logic, I can conclude humans don't think, because of all the numerous times out 'thinking fails'. I don't know what else to tell you other than this infallible logic automaton you imagine must exist before it is 'real intelligence' does not exist and has never existed except in the realm of fiction.
    - JoelMcCracken 1y ago
      You’re absolutely right!
  - hackinthebochs 1y ago
    Why should LLM failures trump successes when determining if it thinks/understands? Yes, they have a lot of inhuman failure modes. But so what, they aren't human. Their training regimes are very dissimilar to ours and so we should expect alien failure modes owing to this. This doesn't strike me as good reason to think they don't understand anything in the face of examples that presumably demonstrate understanding.
    - saalweachter 1y ago
      Because there's no difference between a success and failure as far as an LLM is concerned. Nothing went wrong when the LLM produced a false statement. Nothing went right when the LLM produced a true statement. It produced a statement. The lexical structure of the statement is highly congruent with its training data and the previous statements.
      - hackinthebochs 1y ago
        This argument is vacuous. Truth is always external to the system. Nothing goes wrong inside the human when he makes an unintentionally false claim. He is simply reporting on what he believes to be true. There are failures leading up to the human making a false claim. But the same can be said for the LLM in terms of insufficient training data. >The lexical structure of the statement is highly congruent with its training data and the previous statements. This doesn't accurately capture how LLMs work. LLMs have an ability to generalize that undermines the claim of their responses being "highly congruent with training data".
  - wholinator2 1y ago
    Well the difference between those two statements is obvious. One looks and feels, the other processes and analyzes. Most people can process and analyze some things, they're not complete idiots most of the time. But also most people cannot think and analyze the most ground breaking technological advancement they might've personally ever witnessed, that requires college level math and computer science to understand. It's how people have been forever, electricity, the telephone, computers, even barcodes. People just don't understand new technologies. It would be much weirder if the populace suddenly knew exactly what was going on. And to the "most groundbreaking blah blah blah", i could argue that the difference between no computer and computer requires you to actually understand the computer, which almost no one actually does. It just makes peoples work more confusing and frustrating most of the time. While the difference between computer that can't talk to you and "the voice of god answering directly all questions you can think of" is a sociological catastrophic change.
rapind 1y ago
> Most people didn't think we were anywhere close to LLM's five years ago. That's very ambiguous. "Most people" don't know most things. If we're talking about people that have been working in the industry though, my understanding is that the concept of our modern day LLMs aren't magical at all. In fact, the idea has been around for quite a while. The breakthroughs in processing power and networking (data) were the hold up. The result definitely feels magical to "most people" though for sure. Right now we're "iterating" right? I'm not sure anyone really see's a clear path to AGI if what we're actually talking about is the singularity. There are a lot of unknown unknowns right?
- fadedsignal 1y ago
  I 100% agree with this. I suggest the other guy to check history of NLP.
  - crazygringo 1y ago
    Not sure what history you're suggesting I check? I've been following NLP for decades. Sure, neural nets have been around for many decades. Deep learning in this century. But the explosive success of what LLM's can do now came as a huge surprise. Transformers date to just 2017, and the idea that they would be this successful just with throwing gargantuan amounts of data and processing at them -- this was not a common viewpoint. So I stand by the main point of my original comment, except I did just now edit it to say 10 years ago rather than 5... the point is, it really did seem to come out of nowhere.
- dkdcio 1y ago
  I worked in Microsoft's AI platform from 2018-2022. people were very aware of LLMs & AI in general. it's not magical AGI is a silly concept
  - pixl97 1y ago
    AGI is a poorly defined concept because intelligence is a poorly defined concept. Everyone knows what intelligence is... until we attempt to agree on a common definition.
gravity13 1y ago
At this point, AGI seems to be more of a marketing beacon than any sort of non-vague deterministic classification. We all thought about a future where AI just woke up one day, when realistically, we got philosophical debates over whether the ability to finally order a pizza constitutes true intelligence.
- noir_lord 1y ago
  We can order the pizza, it just hallucinated and I'm not entirely sure why my pizza has seahorses instead of anchovies.
armonster 1y ago
I think what is much more plausible is that companies such as this one benefit greatly from being viewed as being close to, or on the way to AGI.
- travelalberta 1y ago
  > Once AGI is declared by OpenAI, that declaration will now be verified by an independent expert panel. I always like the phrase, "follow the money", in situations like this. Are OpenAI or Microsoft close to AGI? Who knows... Is there a monetary incentive to making you believe they are close to AGI? Absolutely. Take in this was the first bullet point in Microsoft's blog post.
fadedsignal 1y ago
I don't think AGI will happen with LLMs. For example, can an LLM drive a car?? I know it's a silly question but it's a fact.
- famouswaffles 1y ago
  Well yeah https://wayve.ai/thinking/lingo-2-driving-with-language/ https://wayve.ai/thinking/lingo-2-driving-with-language/
- JoelMcCracken 1y ago
  this is something I think about. state of the art in self driving cars still makes mistakes that humans wouldn't make, despite all the investment into this specific problem. This bodes very poorly for AGI in the near term, IMO
- torginus 1y ago
  It can? If you use 'multimodal transformer' instead of LLM (which most SOTA models are), I don't think there's any reason why a transformer arch couldn't be trained to drive a car, in fact I'm sure that's what Tesla and co. are using in their cars right now. I'm sure self-driving will become good enough to be commercially viable in the next couple years (with some limitations), that doesn't mean it's AGI.
  - tsimionescu 1y ago
    There is a vast gulf between "GPT-5 can drive a car" and "a neural network using the transformer architecture can be trained to drive a car". And I see no proof whatsoever that we can, today, train a single model that can both write a play and drive a car. Even less so one that could do both at the same time, as a generally intelligent being should be able to. If someone wants to claim that, say, GPT-5 is AGI, then it is on them to connect GPT-5 to a car control system and inputs and show that it can drive a car decently well. After all, it has consumed all of the literature on driving and physics ever produced, plus untold numbers of hours of video of people driving.
    - 1y ago
voidfunc 1y ago
Also possible we get something "close enough" to AGI and it's really fucking useful. AGI is the end-game. There's a lot of room between current LLMs and AGI.
- oldestofsports 1y ago
  Sure, but then OpenAI should not claim it is AGI, even if it is ”close enough”
dreamcompiler 1y ago
In 1900 we didn't see a viable path to climb Mount Everest or to go to the moon. This does not make the two tasks equally difficult.
nodja 1y ago
GPT3 existed 5 years ago, and the trajectory was set with the transformers paper. Everything from the transformer paper to GPT3 was pretty much speculated in the paper, it just took people spending the effort and compute to make it reality. The only real surprise was how fast openai producterized an LLM into a chat interface with chatgpt, before then we had finetuned GPT3 models doing specific tasks (translation, summarization, etc.)
jlarocco 1y ago
What people thought about LLMs five years ago, and how close we are to AGI right now are unrelated, and it's not logially sound to say "We were close to LLMs then, so we are close to AGI now." It's also a misleading view of the history. It's true "most people" weren't thinking about LLMs five years ago, but a lot of the underpinnings had been studied since the 70s and 80s. The ideas had been worked out, but the hardware wasn't able to handle the processing. > True honesty requires acknowledging that we truly have no idea. Progress in AI is happening faster than ever before, but nobody has the slightest idea how much progress is needed to get to AGI. Maybe, but don't tell that to OpenAI's investors.
- deleted 1y ago
  [deleted]
mandeepj 1y ago
> Most people didn't think we were anywhere close to LLM's five years ago. Well, Google had LLMs ready by 2017, which was almost 9 years ago. https://en.wikipedia.org/wiki/Large_language_model https://en.wikipedia.org/wiki/Large_language_model
timdiggerm 1y ago
> Progress in AI is happening faster than ever before Is it happening faster than it was six months ago? a year ago?

airstrike 1y ago

Notwithstanding the fact that AGI is a significantly higher bar than "LLM", this argument is illogical. Nobody thought we were anywhere closer to me jumping off the Empire State Building and flying across the globe 5 years ago, but I'm sure I will. Wish me luck as I take that literal leap of faith tomorrow.

JoelMcCracken 1y ago
what's super weird to me is how people seem to look at LLM output and see: "oh look it can think! but then it fails sometimes! how strange, we need to fix the bug that makes the thinking no workie" instead of: "oh, this is really weird. Its like a crazy advanced pattern recognition and completion engine that works better than I ever imagined such a thing could. But, it also clearly isn't _thinking_, so it seems like we are perhaps exactly as far from thinking machines as we were before LLMs"
- famouswaffles 1y ago
  By that logic, I can conclude humans don't think, because of all the numerous times out 'thinking fails'. I don't know what else to tell you other than this infallible logic automaton you imagine must exist before it is 'real intelligence' does not exist and has never existed except in the realm of fiction.
  - JoelMcCracken 1y ago
    You’re absolutely right!
- hackinthebochs 1y ago
  Why should LLM failures trump successes when determining if it thinks/understands? Yes, they have a lot of inhuman failure modes. But so what, they aren't human. Their training regimes are very dissimilar to ours and so we should expect alien failure modes owing to this. This doesn't strike me as good reason to think they don't understand anything in the face of examples that presumably demonstrate understanding.
  - saalweachter 1y ago
    Because there's no difference between a success and failure as far as an LLM is concerned. Nothing went wrong when the LLM produced a false statement. Nothing went right when the LLM produced a true statement. It produced a statement. The lexical structure of the statement is highly congruent with its training data and the previous statements.
    - hackinthebochs 1y ago
      This argument is vacuous. Truth is always external to the system. Nothing goes wrong inside the human when he makes an unintentionally false claim. He is simply reporting on what he believes to be true. There are failures leading up to the human making a false claim. But the same can be said for the LLM in terms of insufficient training data. >The lexical structure of the statement is highly congruent with its training data and the previous statements. This doesn't accurately capture how LLMs work. LLMs have an ability to generalize that undermines the claim of their responses being "highly congruent with training data".
- wholinator2 1y ago
  Well the difference between those two statements is obvious. One looks and feels, the other processes and analyzes. Most people can process and analyze some things, they're not complete idiots most of the time. But also most people cannot think and analyze the most ground breaking technological advancement they might've personally ever witnessed, that requires college level math and computer science to understand. It's how people have been forever, electricity, the telephone, computers, even barcodes. People just don't understand new technologies. It would be much weirder if the populace suddenly knew exactly what was going on. And to the "most groundbreaking blah blah blah", i could argue that the difference between no computer and computer requires you to actually understand the computer, which almost no one actually does. It just makes peoples work more confusing and frustrating most of the time. While the difference between computer that can't talk to you and "the voice of god answering directly all questions you can think of" is a sociological catastrophic change.

rapind 1y ago

> Most people didn't think we were anywhere close to LLM's five years ago. That's very ambiguous. "Most people" don't know most things. If we're talking about people that have been working in the industry though, my understanding is that the concept of our modern day LLMs aren't magical at all. In fact, the idea has been around for quite a while. The breakthroughs in processing power and networking (data) were the hold up. The result definitely feels magical to "most people" though for sure. Right now we're "iterating" right? I'm not sure anyone really see's a clear path to AGI if what we're actually talking about is the singularity. There are a lot of unknown unknowns right?

fadedsignal 1y ago
I 100% agree with this. I suggest the other guy to check history of NLP.
- crazygringo 1y ago
  Not sure what history you're suggesting I check? I've been following NLP for decades. Sure, neural nets have been around for many decades. Deep learning in this century. But the explosive success of what LLM's can do now came as a huge surprise. Transformers date to just 2017, and the idea that they would be this successful just with throwing gargantuan amounts of data and processing at them -- this was not a common viewpoint. So I stand by the main point of my original comment, except I did just now edit it to say 10 years ago rather than 5... the point is, it really did seem to come out of nowhere.
dkdcio 1y ago
I worked in Microsoft's AI platform from 2018-2022. people were very aware of LLMs & AI in general. it's not magical AGI is a silly concept
- pixl97 1y ago
  AGI is a poorly defined concept because intelligence is a poorly defined concept. Everyone knows what intelligence is... until we attempt to agree on a common definition.

gravity13 1y ago

At this point, AGI seems to be more of a marketing beacon than any sort of non-vague deterministic classification. We all thought about a future where AI just woke up one day, when realistically, we got philosophical debates over whether the ability to finally order a pizza constitutes true intelligence.

noir_lord 1y ago
We can order the pizza, it just hallucinated and I'm not entirely sure why my pizza has seahorses instead of anchovies.

armonster 1y ago

I think what is much more plausible is that companies such as this one benefit greatly from being viewed as being close to, or on the way to AGI.

travelalberta 1y ago
> Once AGI is declared by OpenAI, that declaration will now be verified by an independent expert panel. I always like the phrase, "follow the money", in situations like this. Are OpenAI or Microsoft close to AGI? Who knows... Is there a monetary incentive to making you believe they are close to AGI? Absolutely. Take in this was the first bullet point in Microsoft's blog post.

fadedsignal 1y ago

I don't think AGI will happen with LLMs. For example, can an LLM drive a car?? I know it's a silly question but it's a fact.

famouswaffles 1y ago
Well yeah https://wayve.ai/thinking/lingo-2-driving-with-language/ https://wayve.ai/thinking/lingo-2-driving-with-language/
JoelMcCracken 1y ago
this is something I think about. state of the art in self driving cars still makes mistakes that humans wouldn't make, despite all the investment into this specific problem. This bodes very poorly for AGI in the near term, IMO
torginus 1y ago
It can? If you use 'multimodal transformer' instead of LLM (which most SOTA models are), I don't think there's any reason why a transformer arch couldn't be trained to drive a car, in fact I'm sure that's what Tesla and co. are using in their cars right now. I'm sure self-driving will become good enough to be commercially viable in the next couple years (with some limitations), that doesn't mean it's AGI.
- tsimionescu 1y ago
  There is a vast gulf between "GPT-5 can drive a car" and "a neural network using the transformer architecture can be trained to drive a car". And I see no proof whatsoever that we can, today, train a single model that can both write a play and drive a car. Even less so one that could do both at the same time, as a generally intelligent being should be able to. If someone wants to claim that, say, GPT-5 is AGI, then it is on them to connect GPT-5 to a car control system and inputs and show that it can drive a car decently well. After all, it has consumed all of the literature on driving and physics ever produced, plus untold numbers of hours of video of people driving.
  - 1y ago

voidfunc 1y ago

Also possible we get something "close enough" to AGI and it's really fucking useful. AGI is the end-game. There's a lot of room between current LLMs and AGI.

oldestofsports 1y ago
Sure, but then OpenAI should not claim it is AGI, even if it is ”close enough”

dreamcompiler 1y ago

In 1900 we didn't see a viable path to climb Mount Everest or to go to the moon. This does not make the two tasks equally difficult.

nodja 1y ago

GPT3 existed 5 years ago, and the trajectory was set with the transformers paper. Everything from the transformer paper to GPT3 was pretty much speculated in the paper, it just took people spending the effort and compute to make it reality. The only real surprise was how fast openai producterized an LLM into a chat interface with chatgpt, before then we had finetuned GPT3 models doing specific tasks (translation, summarization, etc.)

jlarocco 1y ago

What people thought about LLMs five years ago, and how close we are to AGI right now are unrelated, and it's not logially sound to say "We were close to LLMs then, so we are close to AGI now." It's also a misleading view of the history. It's true "most people" weren't thinking about LLMs five years ago, but a lot of the underpinnings had been studied since the 70s and 80s. The ideas had been worked out, but the hardware wasn't able to handle the processing. > True honesty requires acknowledging that we truly have no idea. Progress in AI is happening faster than ever before, but nobody has the slightest idea how much progress is needed to get to AGI. Maybe, but don't tell that to OpenAI's investors.

deleted 1y ago
[deleted]

mandeepj 1y ago

> Most people didn't think we were anywhere close to LLM's five years ago. Well, Google had LLMs ready by 2017, which was almost 9 years ago. https://en.wikipedia.org/wiki/Large_language_model https://en.wikipedia.org/wiki/Large_language_model

timdiggerm 1y ago

> Progress in AI is happening faster than ever before Is it happening faster than it was six months ago? a year ago?