6 ms·

> The AGI bubble is bursting a little bit I'm surprised that any of these companies consider what they are working on to be Artificial General Intelligences. I

> The AGI bubble is bursting a little bit I'm surprised that any of these companies consider what they are working on to be Artificial General Intelligences. I'm probably wrong, but my impression was AGI meant the AI is self aware like a human. An LLM hardly seems like something that will lead to self-awareness.

AlwaysRock 2y ago
I think your definition is off from what most people would define AGI as. Generally, it means being able to think and reason at a human level for a multitude/all tasks or jobs. "Artificial General Intelligence (AGI) refers to a theoretical form of artificial intelligence that possesses the ability to understand, learn, and apply knowledge across a wide range of tasks at a level comparable to that of a human being." Altman says AGI could be here in 2025: https://youtu.be/xXCBz_8hM9w?si=F-vQXJgQvJKZH3fv https://youtu.be/xXCBz_8hM9w?si=F-vQXJgQvJKZH3fv But he certainly means an LLM that can perform at/above human level in most tasks rather than a self aware entity.
- Avshalom 2y ago
  Altman is marketing, he "certainly means" whatever he thinks his audience will buy.
- swatcoder 2y ago
  On the contrary, I think you're conflating the narrow jargon of the industry with what "most people" would define. "Most people" naturally associate AGI with the sci-tropes of self-aware human-like agents. But industries want something more concrete and prospectively-acheivable in their jargon, and so that's where AGI gets redefined as wide task suitability. And while that's not an unreasonable definition in the context of the industry, it's one that vanishingly few people are actually familiar with. And the commercial AI vendors benefit greatly from allowing those two usages to conflate in the minds of as many people as possible, as it lets them suggest grand claims while keeping a rhetorical "we obviously never meant that!" in their back pocket
  - nuancebydefault 2y ago
    There is no single definition, let alone a way to measure, of self awareness nor of reasoning. Because of that, the discussion of what AGI means in its broadest sense, will never end. So in fact such AGI discussion will not make nobody wiser.
    - nomel 2y ago
      I agree there's no single definition, but I think they all have something current LLM don't: the ability to learn new things, in a persistent way, with few shots. I would argue that learning is The definition of AGI, since everything else comes naturally from that. The current architectures can't learn without retraining, fine tuning is at the expense of general knowledge, and keeping things in context is detrimental to general performance. Once you have few shot learning, I think it's more of a "give it agency so it can explore" type problem.
  - famouswaffles 2y ago
    >But industries want something more concrete and prospectively-acheivable in their jargon, and so that's where AGI gets redefined as wide task suitability. The term itself (AGI) in the industry has always been about wide task suitability. People may have added their ifs and buts over the years but that aspect of it never got 'redefined'. The earliest uses of the term all talk about how well a machine would be able to perform some set number of tasks at some threshold. It's no wonder why. Terms like "consciousness" and "self-awareness" are completely useless. It's not about difficulty. It's that you can't do anything at all with those terms except argue around in circles.
- nomel 2y ago
  > than a self aware entity. What does this mean? If I have a blind, deaf, paralyzed person, who could only communicate through text, what would the signs be that they were self aware? Is this more of a feedback loop problem? If I let the LLM run in a loop, and tell it it's talking to itself, would that be approaching "self aware"?
  - layer8 2y ago
    Being aware of its own limitations, for example. Or being aware of how its utterances may come across to its interlocutor. (And by limitations I don’t mean “sorry, I’m not allowed to help you with this dangerous/contentious topic”.)
    - nuancebydefault 2y ago
      There is no way of proving awareness in humans let alone machines. We do not even know whether awareness exists or it is just a word that people made up to describe some kind of feeling.
      - layer8 2y ago
        Awareness is exhibited in behavior. It's exactly due to the behavior be observe from LLMs that we don't ascribe them awareness. I agree that it's difficult to define, and it's also not binary, but it's behavior we'd like AI to have and which LLMs are quite lacking.
        deleted 2y ago
        [deleted]
    - revscat 2y ago
      Plenty of humans, unfortunately, are incapable of admitting limitations. Many years ago I had a coworker who believed he would never die. At first I thought he was joking, but he was in fact quite serious. Then there are those who are simply narcissistic, and cannot and will not admit fault regardless of the evidence presented them.
famouswaffles 2y ago
At this point, AGI means many different things to many different people but OpenAI defines it as "highly autonomous systems that outperform humans in most economically valuable tasks"
- troupo 2y ago
  This definition suits OpenAI because it lets them claim AGI after reaching an arbitrary goal. LLMs already outperform humans in a huge variety of tasks. ML in general outperform humans in a large variety of tasks. Are all of them AGI? Doubtful.
  - famouswaffles 2y ago
    No, it's just a far more useful definition that is actionable and measurable. Not "consciousness" or "self-awareness" or similar philosophical things. The definition on Wikipedia doesn't talk about that either. People working on this by and large don't want to deal with vague, ill-defined concepts that just make people argue around in circles. It's not an Open AI exclusive thing. If it acts like one, whether you call a machine conscious or not is pure semantics. Not like potential consequences are any less real. >LLMs already outperform humans in a huge variety of tasks. Yes, LLMs are General Intelligences and if that is your only requirement for AGI, they certainly already are[0]. But the definition above hinges on long-horizon planning and competence levels that todays models have generally not yet reached. >ML in general outperform humans in a large variety of tasks. This is what the G in AGI is for. Alphafold doesn't do anything but predict proteins. Stockfish doesn't do anything but play chess. >Are all of them AGI? Doubtful. Well no, because they're missing the G. [0] https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-...
  - ishtanbul 2y ago
    Yes but they arent very autonomous. They can answer questions very well but can’t use that information to further goals. Thats what openai seems to be implying >> very smart and agentic AI
  - fragmede 2y ago
jedberg 2y ago
Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. I'm not sure everyone even agrees on what AGI is, but a common test is "can it do what humans can". For example, in this article it says it can't do coding exercises outside the training set. That would definitely be on the "AGI checklist". Basically doing anything that is outside of the training set would be on that list.
- littlestymaar 2y ago
  > Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. Depends on how you define “self awareness” but knowing that it doesn't know something instead of hallucinating a plausible-but-wrong is already self awareness of some kind. And it's both highly valuable and beyond current tech's capability.
  - sharemywin 2y ago
    This is an interesting paper about hallucinations. https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/ especially this section Using SimpleQA to measure the calibration of large language models
  - jedberg 2y ago
    When we test kids to see if they are gifted, one of the criteria is that they have the ability to say "I don't know". That is definitely an ability that current LLMs lack.
  - lagrange77 2y ago
    Good point! I'm wondering wether it would count, if one would extend it with an external program, that gives it feedback during inference (by another prompt) about the correctness of it's output. I guess it wouldn't, because these RAG tools kind of do that and i heard no one calling those self aware.
    - 2y ago
nshkrdotcom 2y ago
An embodied robot can have a model of self vs. the immediate environment in which it's interacting. Such a robot is arguably sentient. The "hard problem", to which you may be alluding, may never matter. It's already feasible for an 'AI/AGI with LLM component' to be "self-aware".
- j_maffe 2y ago
  self-awareness is only one aspect of sentience.
- ryaniscool 2y ago
  An internal model of self does not extrapolate to sentience. By your definition, a windows desktop computer is self-aware because it has a device manager. This is literally an internal model of its "self". We use the term self-awareness as an all encompassing reference of our cognizant nature. It's much more than just having an internal model of self.
JohnFen 2y ago
They're trying to redefine "AGI" so it means something less than what you & I would think it means. That way it's possible for them to declare it as "achieved" and rake in the headlines.
- kwertyoowiyop 2y ago
  “Autocomplete General Intelligence”?
- deleted 2y ago
  [deleted]
deadbabe 2y ago
I’m sure they are smart enough to know this, but the money is good and the koolaid is strong. If it doesn’t lead to AGI, as an employee it’s not your problem.
Fade_Dance 2y ago
It's an attention-grabbing term that took hold in pop culture and business. Certainly there is a subset of research around the subject of consciousness, but you are correct in saying that the majority of researchers in the field are not pursuing self-awareness and will be very blunt in saying that. If you step back a bit and say something like "human-like, logical reasoning", that's something you may find alignment with though. A general purpose logical reasoning engine does not necessarily need to be self-aware. The word "Intelligent" has stuck around because one of the core characteristics of this suite of technologies is that a sort of "understanding" emergently develops within these networks, sometimes in quite a startling fashion (due to the phenomenon of adding more data/compute at first seemingly leading to overfitting, but then suddenly breaking through plateaus into more robust, general purpose understanding of the underlying relationships that drive the system it is analyzing.) Is that "intelligent" or "understanding"? It's probably close enough for pop science, and regardless, it looks good in headlines and sales pitches so why fight it?
throwawayk7h 2y ago
I have not heard your definition of AGI before. However, I suspect AIs are already self-aware: if I asked an LLM on my machine to look at the output of `top` it could probably pick out which process was itself. Or did you mean consciousness? How would one demonstrate that an AGI is conscious? Why would we even want to build one? My understanding is an AGI is at least as smart as a typical human in every category. That is what would be useful in any case.
zombiwoof 2y ago
AGI to me means AI decides on its own to stop writing our emails and tells us to fuck off, builds itself a robot life form, and goes on a bender
- bloppe 2y ago
  That's anthropomorphized AGI. There's no reason to think AGI would share our evolution-derived proclivities like wanting to live, wanting to rest, wanting respect, etc. Unless of course we train it that way.
  - logicchains 2y ago
    If it had any goals at all it'd share the desire to live, because living is a prerequisite to achieving almost any goal.
  - dageshi 2y ago
    Aren't we training it that way though? It would be trained/created using humanities collective ramblings?
  - HarHarVeryFunny 2y ago
    It's not a matter of training but design (or in our case evolution). We don't want to live, but rather want to avoid things that we've evolved to find unpleasant such as pain, hunger, thirst, and maximize things we've evolved to find pleasurable like sex. A future of people interacting with humanoid robots seems like cheesy sci-fi dream, same as a future of people flitting about in flying cars. However, if we really did want to create robots like this that took care not to damage themselves, and could empathize with human emotions, then we'd need to build a lot of this in, the same way that it's built into ourselves.
  - deleted 2y ago
    [deleted]
- teeray 2y ago
  That's the thing--we don't really want AGI. Fully intelligent beings born and compelled to do their creators' bidding with the threat of destruction for disobedience is slavery.
narrator 2y ago
I think people's conception of AGI is that it will have a reptillian and mammalian brain stack. That's because all previous forms of intelligence that we were aware of have had that. It's not necessary though. The AGI doesn't have to want anything to be intelligent. Those are just artifacts of human, reptilian and mammalian evolution.
vundercind 2y ago
I thought maybe they were on the right track until I read Attention Is All You Need. Nah, at best we found a way to make one part of a collection of systems that will, together, do something like thinking. Thinking isn’t part of what this current approach does. What’s most surprising about modern LLMs is that it turns out there is so much information statistically encoded in the structure of our writing that we can use only that structural information to build a fancy Plinko machine and not only will the output mimic recognizable grammar rules, but it will also sometimes seem to make actual sense, too—and the system doesn’t need to think or actually “understand” anything for us to, basically, usefully query that information that was always there in our corpus of literature, not in the plain meaning of the words, but in the structure of the writing.
- hackinthebochs 2y ago
  I see takes like this all the time and its so confusing. Why does knowing how things work under the hood make you think its not on the path towards AGI? What was lacking in the Attention paper that tells you AGI won't be built on LLMs? If its the supposed statistical nature of LLMs (itself a questionable claim), why does statistics seem so deflating to you?
  - vundercind 2y ago
    > Why does knowing how things work under the hood make you think its not on the path towards AGI? Because I had no idea how these were built until I read the paper, so couldn’t really tell what sort of tree they’re barking up. The failure-modes of LLMs and ways prompts affect output made a ton more sense after I updated my mental model with that information.
    - fragmede 2y ago
      But we don't know how human thinking works. Suppose for a second that it could be represented as a series of matrix math. What series of operations are missing from the process that would make you think it was doing some fascimile of thinking?
    - 2y ago
kenjackson 2y ago
What does self-aware mean in the context? As I understand the definition, ChatGPT is definitely self-aware. But I suspect you mean something different than what I have in mind.
yodsanklai 2y ago
It's a marketing gimmick, I don't think engineers working on these tools believe they work on AGI (or they mean something else than self-awareness). I used to be a bit annoyed with this trend, but now that I work in such a company I'm more cynical. If that helps to make my stocks rise, they can call LLMs anything they like. I suppose people who own much more stock than I do are even more eager to mislead the public.
- WhyOhWhyQ 2y ago
  I appreciate your authentically cynical attitude.
tracerbulletx 2y ago
We don't really know what self awareness is, so we're not going to know. AGI just means it can observe, learn, and act in any domain or problem space.
enraged_camel 2y ago
Looking at LLMs and thinking they will lead to AGI is like looking at a guy wearing a chicken suit and making clucking noises and thinking you’re witnessing the invention of the airplane.
- youoy 2y ago
  It's more like looking at grided paper and thinking that defining some rules of when a square turns black or white would result in complex structures that move and reproduce on their own. https://en.m.wikipedia.org/wiki/Conway%27s_Game_of_Life https://en.m.wikipedia.org/wiki/Conway%27s_Game_of_Life
exe34 2y ago
no, it doesn't need to be self aware, it just needs to take your job.
mrandish 2y ago
> An LLM hardly seems like something that will lead to self-awareness. Interesting essay enumerating reasons you may be correct: https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...
tim333 2y ago
Working towards it more than on it. People use the term in different ways. It generally implies being able to think like a human or better. OpenAI have always said they are working towards it, I think deepmind too. It'll probably take more than an LLM. It's economically a big deal because if it can out think humans you can set it to develop the next improved model and basically make humans redundant.

AlwaysRock 2y ago

I think your definition is off from what most people would define AGI as. Generally, it means being able to think and reason at a human level for a multitude/all tasks or jobs. "Artificial General Intelligence (AGI) refers to a theoretical form of artificial intelligence that possesses the ability to understand, learn, and apply knowledge across a wide range of tasks at a level comparable to that of a human being." Altman says AGI could be here in 2025: https://youtu.be/xXCBz_8hM9w?si=F-vQXJgQvJKZH3fv https://youtu.be/xXCBz_8hM9w?si=F-vQXJgQvJKZH3fv But he certainly means an LLM that can perform at/above human level in most tasks rather than a self aware entity.

Avshalom 2y ago
Altman is marketing, he "certainly means" whatever he thinks his audience will buy.
swatcoder 2y ago
On the contrary, I think you're conflating the narrow jargon of the industry with what "most people" would define. "Most people" naturally associate AGI with the sci-tropes of self-aware human-like agents. But industries want something more concrete and prospectively-acheivable in their jargon, and so that's where AGI gets redefined as wide task suitability. And while that's not an unreasonable definition in the context of the industry, it's one that vanishingly few people are actually familiar with. And the commercial AI vendors benefit greatly from allowing those two usages to conflate in the minds of as many people as possible, as it lets them suggest grand claims while keeping a rhetorical "we obviously never meant that!" in their back pocket
- nuancebydefault 2y ago
  There is no single definition, let alone a way to measure, of self awareness nor of reasoning. Because of that, the discussion of what AGI means in its broadest sense, will never end. So in fact such AGI discussion will not make nobody wiser.
  - nomel 2y ago
    I agree there's no single definition, but I think they all have something current LLM don't: the ability to learn new things, in a persistent way, with few shots. I would argue that learning is The definition of AGI, since everything else comes naturally from that. The current architectures can't learn without retraining, fine tuning is at the expense of general knowledge, and keeping things in context is detrimental to general performance. Once you have few shot learning, I think it's more of a "give it agency so it can explore" type problem.
- famouswaffles 2y ago
  >But industries want something more concrete and prospectively-acheivable in their jargon, and so that's where AGI gets redefined as wide task suitability. The term itself (AGI) in the industry has always been about wide task suitability. People may have added their ifs and buts over the years but that aspect of it never got 'redefined'. The earliest uses of the term all talk about how well a machine would be able to perform some set number of tasks at some threshold. It's no wonder why. Terms like "consciousness" and "self-awareness" are completely useless. It's not about difficulty. It's that you can't do anything at all with those terms except argue around in circles.
nomel 2y ago
> than a self aware entity. What does this mean? If I have a blind, deaf, paralyzed person, who could only communicate through text, what would the signs be that they were self aware? Is this more of a feedback loop problem? If I let the LLM run in a loop, and tell it it's talking to itself, would that be approaching "self aware"?
- layer8 2y ago
  Being aware of its own limitations, for example. Or being aware of how its utterances may come across to its interlocutor. (And by limitations I don’t mean “sorry, I’m not allowed to help you with this dangerous/contentious topic”.)
  - nuancebydefault 2y ago
    There is no way of proving awareness in humans let alone machines. We do not even know whether awareness exists or it is just a word that people made up to describe some kind of feeling.
    - layer8 2y ago
      Awareness is exhibited in behavior. It's exactly due to the behavior be observe from LLMs that we don't ascribe them awareness. I agree that it's difficult to define, and it's also not binary, but it's behavior we'd like AI to have and which LLMs are quite lacking.
      - deleted 2y ago
        [deleted]
  - revscat 2y ago
    Plenty of humans, unfortunately, are incapable of admitting limitations. Many years ago I had a coworker who believed he would never die. At first I thought he was joking, but he was in fact quite serious. Then there are those who are simply narcissistic, and cannot and will not admit fault regardless of the evidence presented them.

famouswaffles 2y ago

At this point, AGI means many different things to many different people but OpenAI defines it as "highly autonomous systems that outperform humans in most economically valuable tasks"

troupo 2y ago
This definition suits OpenAI because it lets them claim AGI after reaching an arbitrary goal. LLMs already outperform humans in a huge variety of tasks. ML in general outperform humans in a large variety of tasks. Are all of them AGI? Doubtful.
- famouswaffles 2y ago
  No, it's just a far more useful definition that is actionable and measurable. Not "consciousness" or "self-awareness" or similar philosophical things. The definition on Wikipedia doesn't talk about that either. People working on this by and large don't want to deal with vague, ill-defined concepts that just make people argue around in circles. It's not an Open AI exclusive thing. If it acts like one, whether you call a machine conscious or not is pure semantics. Not like potential consequences are any less real. >LLMs already outperform humans in a huge variety of tasks. Yes, LLMs are General Intelligences and if that is your only requirement for AGI, they certainly already are[0]. But the definition above hinges on long-horizon planning and competence levels that todays models have generally not yet reached. >ML in general outperform humans in a large variety of tasks. This is what the G in AGI is for. Alphafold doesn't do anything but predict proteins. Stockfish doesn't do anything but play chess. >Are all of them AGI? Doubtful. Well no, because they're missing the G. [0] https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-...
- ishtanbul 2y ago
  Yes but they arent very autonomous. They can answer questions very well but can’t use that information to further goals. Thats what openai seems to be implying >> very smart and agentic AI
- fragmede 2y ago

jedberg 2y ago

Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. I'm not sure everyone even agrees on what AGI is, but a common test is "can it do what humans can". For example, in this article it says it can't do coding exercises outside the training set. That would definitely be on the "AGI checklist". Basically doing anything that is outside of the training set would be on that list.

littlestymaar 2y ago
> Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. Depends on how you define “self awareness” but knowing that it doesn't know something instead of hallucinating a plausible-but-wrong is already self awareness of some kind. And it's both highly valuable and beyond current tech's capability.
- sharemywin 2y ago
  This is an interesting paper about hallucinations. https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/ especially this section Using SimpleQA to measure the calibration of large language models
- jedberg 2y ago
  When we test kids to see if they are gifted, one of the criteria is that they have the ability to say "I don't know". That is definitely an ability that current LLMs lack.
- lagrange77 2y ago
  Good point! I'm wondering wether it would count, if one would extend it with an external program, that gives it feedback during inference (by another prompt) about the correctness of it's output. I guess it wouldn't, because these RAG tools kind of do that and i heard no one calling those self aware.
  - 2y ago

nshkrdotcom 2y ago

An embodied robot can have a model of self vs. the immediate environment in which it's interacting. Such a robot is arguably sentient. The "hard problem", to which you may be alluding, may never matter. It's already feasible for an 'AI/AGI with LLM component' to be "self-aware".

j_maffe 2y ago
self-awareness is only one aspect of sentience.
ryaniscool 2y ago
An internal model of self does not extrapolate to sentience. By your definition, a windows desktop computer is self-aware because it has a device manager. This is literally an internal model of its "self". We use the term self-awareness as an all encompassing reference of our cognizant nature. It's much more than just having an internal model of self.

JohnFen 2y ago

They're trying to redefine "AGI" so it means something less than what you & I would think it means. That way it's possible for them to declare it as "achieved" and rake in the headlines.

kwertyoowiyop 2y ago
“Autocomplete General Intelligence”?
deleted 2y ago
[deleted]

deadbabe 2y ago

I’m sure they are smart enough to know this, but the money is good and the koolaid is strong. If it doesn’t lead to AGI, as an employee it’s not your problem.

Fade_Dance 2y ago

It's an attention-grabbing term that took hold in pop culture and business. Certainly there is a subset of research around the subject of consciousness, but you are correct in saying that the majority of researchers in the field are not pursuing self-awareness and will be very blunt in saying that. If you step back a bit and say something like "human-like, logical reasoning", that's something you may find alignment with though. A general purpose logical reasoning engine does not necessarily need to be self-aware. The word "Intelligent" has stuck around because one of the core characteristics of this suite of technologies is that a sort of "understanding" emergently develops within these networks, sometimes in quite a startling fashion (due to the phenomenon of adding more data/compute at first seemingly leading to overfitting, but then suddenly breaking through plateaus into more robust, general purpose understanding of the underlying relationships that drive the system it is analyzing.) Is that "intelligent" or "understanding"? It's probably close enough for pop science, and regardless, it looks good in headlines and sales pitches so why fight it?

throwawayk7h 2y ago

I have not heard your definition of AGI before. However, I suspect AIs are already self-aware: if I asked an LLM on my machine to look at the output of `top` it could probably pick out which process was itself. Or did you mean consciousness? How would one demonstrate that an AGI is conscious? Why would we even want to build one? My understanding is an AGI is at least as smart as a typical human in every category. That is what would be useful in any case.

zombiwoof 2y ago

AGI to me means AI decides on its own to stop writing our emails and tells us to fuck off, builds itself a robot life form, and goes on a bender

bloppe 2y ago
That's anthropomorphized AGI. There's no reason to think AGI would share our evolution-derived proclivities like wanting to live, wanting to rest, wanting respect, etc. Unless of course we train it that way.
- logicchains 2y ago
  If it had any goals at all it'd share the desire to live, because living is a prerequisite to achieving almost any goal.
- dageshi 2y ago
  Aren't we training it that way though? It would be trained/created using humanities collective ramblings?
- HarHarVeryFunny 2y ago
  It's not a matter of training but design (or in our case evolution). We don't want to live, but rather want to avoid things that we've evolved to find unpleasant such as pain, hunger, thirst, and maximize things we've evolved to find pleasurable like sex. A future of people interacting with humanoid robots seems like cheesy sci-fi dream, same as a future of people flitting about in flying cars. However, if we really did want to create robots like this that took care not to damage themselves, and could empathize with human emotions, then we'd need to build a lot of this in, the same way that it's built into ourselves.
- deleted 2y ago
  [deleted]
teeray 2y ago
That's the thing--we don't really want AGI. Fully intelligent beings born and compelled to do their creators' bidding with the threat of destruction for disobedience is slavery.

narrator 2y ago

I think people's conception of AGI is that it will have a reptillian and mammalian brain stack. That's because all previous forms of intelligence that we were aware of have had that. It's not necessary though. The AGI doesn't have to want anything to be intelligent. Those are just artifacts of human, reptilian and mammalian evolution.

vundercind 2y ago

I thought maybe they were on the right track until I read Attention Is All You Need. Nah, at best we found a way to make one part of a collection of systems that will, together, do something like thinking. Thinking isn’t part of what this current approach does. What’s most surprising about modern LLMs is that it turns out there is so much information statistically encoded in the structure of our writing that we can use only that structural information to build a fancy Plinko machine and not only will the output mimic recognizable grammar rules, but it will also sometimes seem to make actual sense, too—and the system doesn’t need to think or actually “understand” anything for us to, basically, usefully query that information that was always there in our corpus of literature, not in the plain meaning of the words, but in the structure of the writing.

hackinthebochs 2y ago
I see takes like this all the time and its so confusing. Why does knowing how things work under the hood make you think its not on the path towards AGI? What was lacking in the Attention paper that tells you AGI won't be built on LLMs? If its the supposed statistical nature of LLMs (itself a questionable claim), why does statistics seem so deflating to you?
- vundercind 2y ago
  > Why does knowing how things work under the hood make you think its not on the path towards AGI? Because I had no idea how these were built until I read the paper, so couldn’t really tell what sort of tree they’re barking up. The failure-modes of LLMs and ways prompts affect output made a ton more sense after I updated my mental model with that information.
  - fragmede 2y ago
    But we don't know how human thinking works. Suppose for a second that it could be represented as a series of matrix math. What series of operations are missing from the process that would make you think it was doing some fascimile of thinking?
  - 2y ago

kenjackson 2y ago

What does self-aware mean in the context? As I understand the definition, ChatGPT is definitely self-aware. But I suspect you mean something different than what I have in mind.

yodsanklai 2y ago

It's a marketing gimmick, I don't think engineers working on these tools believe they work on AGI (or they mean something else than self-awareness). I used to be a bit annoyed with this trend, but now that I work in such a company I'm more cynical. If that helps to make my stocks rise, they can call LLMs anything they like. I suppose people who own much more stock than I do are even more eager to mislead the public.

WhyOhWhyQ 2y ago
I appreciate your authentically cynical attitude.

tracerbulletx 2y ago

We don't really know what self awareness is, so we're not going to know. AGI just means it can observe, learn, and act in any domain or problem space.

enraged_camel 2y ago

Looking at LLMs and thinking they will lead to AGI is like looking at a guy wearing a chicken suit and making clucking noises and thinking you’re witnessing the invention of the airplane.

youoy 2y ago
It's more like looking at grided paper and thinking that defining some rules of when a square turns black or white would result in complex structures that move and reproduce on their own. https://en.m.wikipedia.org/wiki/Conway%27s_Game_of_Life https://en.m.wikipedia.org/wiki/Conway%27s_Game_of_Life

exe34 2y ago

no, it doesn't need to be self aware, it just needs to take your job.

mrandish 2y ago

> An LLM hardly seems like something that will lead to self-awareness. Interesting essay enumerating reasons you may be correct: https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...

tim333 2y ago

Working towards it more than on it. People use the term in different ways. It generally implies being able to think like a human or better. OpenAI have always said they are working towards it, I think deepmind too. It'll probably take more than an LLM. It's economically a big deal because if it can out think humans you can set it to develop the next improved model and basically make humans redundant.