5 ms·

Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. I'm not sure everyone

Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. I'm not sure everyone even agrees on what AGI is, but a common test is "can it do what humans can". For example, in this article it says it can't do coding exercises outside the training set. That would definitely be on the "AGI checklist". Basically doing anything that is outside of the training set would be on that list.

littlestymaar 2y ago
> Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. Depends on how you define “self awareness” but knowing that it doesn't know something instead of hallucinating a plausible-but-wrong is already self awareness of some kind. And it's both highly valuable and beyond current tech's capability.
- sharemywin 2y ago
  This is an interesting paper about hallucinations. https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/ especially this section Using SimpleQA to measure the calibration of large language models
- jedberg 2y ago
  When we test kids to see if they are gifted, one of the criteria is that they have the ability to say "I don't know". That is definitely an ability that current LLMs lack.
- lagrange77 2y ago
  Good point! I'm wondering wether it would count, if one would extend it with an external program, that gives it feedback during inference (by another prompt) about the correctness of it's output. I guess it wouldn't, because these RAG tools kind of do that and i heard no one calling those self aware.
  - littlestymaar 2y ago
    > if one would extend it with an external program, that gives it feedback If you have an external program, then by defining it's not self-awareness ;). Also, it's not about correctness per se, but about the model's ability to assess its own knowledge (making a mistake because the model was exposed to mistakes in the training data is fine, hallucinating isn't).
    - lagrange77 2y ago
      Yes, but that's essentially my point. Where to draw the system boundary? The brain is also composed of multiple components and does IO with external components, that are definitely not considered part of it.
Filligree 2y ago
Let me modify that a little, because humans can't do things outside their training set either. A crucial element of AGI would be the ability to self-train on self-generated data, online. So it's not really AGI if there is a hard distinction between training and inference (though it may still be very capable), and it's not really AGI if it can't work its way through novel problems on its own. The ability to immediately solve a problem it's never seen before is too high a bar, I think. And yes, my definition still excludes a lot of humans in a lot of fields. That's a bullet I'm willing to bite.
- lxgr 2y ago
  Are you arguing that writing, doing math, going to the moon etc. were all in the "original training set" of humans in some way?
  - layer8 2y ago
    Not in the original training set (GP is saying), but the necessary skills became part of the training set over time. In other words, human are fine with the training set being a changing moving target, whereas ML models are to a significant extent “stuck” with their original training set. (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.)
    - Jensson 2y ago
      > (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.) The lifetime is the context window, the model/training is the DNA. A human in the moment isn't general intelligent, but a human over his lifetime is, the first is so much easier to try to replicate though but that is a bad target since humans aren't born like that.
- HarHarVeryFunny 2y ago
  > Let me modify that a little, because humans can't do things outside their training set either. That's not true. Humans can learn. An LLM is just a tool. If it can't do what you want then too bad.
norir 2y ago
Here is an example of a task that I do not believe this generation of LLMs can ever do but that is possible for a human: design a Turing complete programming language that is both human and machine readable and implement a self hosted compiler in this language that self compiles on existing hardware faster than any known language implementation that also self compiles. Additionally, for any syntactically or semantically invalid program, the compiler must provide an error message that points exactly to the source location of the first error that occurs in the program. I will get excited for/scared of LLMs when they can tackle this kind of problem. But I don't believe they can because of the fundamental nature of their design, which is both backward looking (thus not better than the human state of the art) and lacks human intuition and self awareness. Or perhaps rather I believe that the prompt that would be required to get an LLM to produce such a program is a problem of at least equivalent complexity to implementing the program without an LLM.
- Xenoamorphous 2y ago
  > Here is an example of a task that I do not believe this generation of LLMs can ever do but that is possible for a human That’s possible for a highly intelligent, extensively trained, very small subset of humans.
  - hatefulmoron 2y ago
    If you took the intersection of every human's abilities you'd be left with a very unimpressive set. That also ignores the fact that the small set of humans capable of building programming languages and compilers is a consequence of specialization and lack of interest. There are plenty of humans that are capable of learning how to do it. LLMs, on the other hand, are both specialized for the task and aren't lazy or uninterested.
  - luckydata 2y ago
    does it mean people that can build languages and compilers are not humans? What is the point you're trying to make?
    - 2y ago
sourcepluck 2y ago
Searle's Chinese Room Argument springs to mind: https://plato.stanford.edu/entries/chinese-room/ The idea that "human-like" behaviour will lead to self-awareness is both unproven (it can't be proven until it happens) and impossible to disprove (like Russell's teapot). Yet, one common assumption of many people running these companies or investing in them, or of some developers investing their time in these technologies, is precisely that some sort of explosion of superintelligence is likely, or even inevitable. It surely is possible, but stretching that to likely seems a bit much if you really think how imperfectly we understand things like consciousness and the mind. Of course there are people who have essentially religious reactions to the notion that there may be limits to certain domains of knowledge. Nonetheless, I think that's the reality we're faced with here.
- abeppu 2y ago
  > The idea that "human-like" behaviour will lead to self-awareness is both unproven (it can't be proven until it happens) and impossible to disprove (like Russell's teapot). I think Searle's view was that: - while it cannot be dis-_proven_, the Chinese Room argument was meant to provide reasons against believing it - the "it can't be proven until it happens" part is misunderstanding: you won't know if it happens because the objective, externally available attributes don't indicate whether self-awareness (or indeed awareness at all) is present
  - sourcepluck 2y ago
    The short version of this is that I don't disagree with your interpretation of Searle, and my paragraphs immediately following the link weren't meant to be a direct description of his point with the Chinese Room thought experiment. > while it cannot be dis-_proven_, the Chinese Room argument was meant to provide reasons against believing it Yes, like Russell's teapot. I also think that's what Searle means. > the "it can't be proven until it happens" part is misunderstanding: you won't know if it happens because the objective, externally available attributes don't indicate whether self-awareness (or indeed awareness at all) is present Yes, agreed, I believe that's what Searle is saying too. I think I was maybe being ambiguous here - I wanted to say that even if you forgave the AI maximalists for ignoring all relevant philosophical work, the notion that "appearing human-like" inevitably tends to what would actually be "consciousness" or "intelligence" is more than a big claim. Searle goes further, and I'm not sure if I follow him all the way, personally, but it's a side point.
olalonde 2y ago
I feel the test for AGI should be more like: "go find a job and earn money" or "start a profitable business" or "pick a bachelor degree and complete it", etc.
- rodgerd 2y ago
  An LLM doing crypto spam/scamming has been making money by tricking Marc Andressen into boosting it. So to the degree that "scamming gullible billionaires and their fans" is a job, that's been done.
  - rsanek 2y ago
    source? didn't find anything online about this.
    - rodgerd 2y ago
      Goatseus Maximus is what you're after.
  - olalonde 2y ago
    That story was a bit blown out of proportion. He gave a research grant to the bot's creator: https://x.com/pmarca/status/1846374466101944629 https://x.com/pmarca/status/1846374466101944629
- jedberg 2y ago
  Can most humans do that? Find a job and earn money, probably. The other two? Not so much.
- eichi 2y ago
  This is people's true desire. Make something like that while handling critisisms and fitting products to the market.

littlestymaar 2y ago

> Whether self awareness is a requirement for AGI definitely gets more into the Philosophy department than the Computer Science department. Depends on how you define “self awareness” but knowing that it doesn't know something instead of hallucinating a plausible-but-wrong is already self awareness of some kind. And it's both highly valuable and beyond current tech's capability.

sharemywin 2y ago
This is an interesting paper about hallucinations. https://openai.com/index/introducing-simpleqa/ https://openai.com/index/introducing-simpleqa/ especially this section Using SimpleQA to measure the calibration of large language models
jedberg 2y ago
When we test kids to see if they are gifted, one of the criteria is that they have the ability to say "I don't know". That is definitely an ability that current LLMs lack.
lagrange77 2y ago
Good point! I'm wondering wether it would count, if one would extend it with an external program, that gives it feedback during inference (by another prompt) about the correctness of it's output. I guess it wouldn't, because these RAG tools kind of do that and i heard no one calling those self aware.
- littlestymaar 2y ago
  > if one would extend it with an external program, that gives it feedback If you have an external program, then by defining it's not self-awareness ;). Also, it's not about correctness per se, but about the model's ability to assess its own knowledge (making a mistake because the model was exposed to mistakes in the training data is fine, hallucinating isn't).
  - lagrange77 2y ago
    Yes, but that's essentially my point. Where to draw the system boundary? The brain is also composed of multiple components and does IO with external components, that are definitely not considered part of it.

Filligree 2y ago

Let me modify that a little, because humans can't do things outside their training set either. A crucial element of AGI would be the ability to self-train on self-generated data, online. So it's not really AGI if there is a hard distinction between training and inference (though it may still be very capable), and it's not really AGI if it can't work its way through novel problems on its own. The ability to immediately solve a problem it's never seen before is too high a bar, I think. And yes, my definition still excludes a lot of humans in a lot of fields. That's a bullet I'm willing to bite.

lxgr 2y ago
Are you arguing that writing, doing math, going to the moon etc. were all in the "original training set" of humans in some way?
- layer8 2y ago
  Not in the original training set (GP is saying), but the necessary skills became part of the training set over time. In other words, human are fine with the training set being a changing moving target, whereas ML models are to a significant extent “stuck” with their original training set. (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.)
  - Jensson 2y ago
    > (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.) The lifetime is the context window, the model/training is the DNA. A human in the moment isn't general intelligent, but a human over his lifetime is, the first is so much easier to try to replicate though but that is a bad target since humans aren't born like that.
HarHarVeryFunny 2y ago
> Let me modify that a little, because humans can't do things outside their training set either. That's not true. Humans can learn. An LLM is just a tool. If it can't do what you want then too bad.

norir 2y ago

Here is an example of a task that I do not believe this generation of LLMs can ever do but that is possible for a human: design a Turing complete programming language that is both human and machine readable and implement a self hosted compiler in this language that self compiles on existing hardware faster than any known language implementation that also self compiles. Additionally, for any syntactically or semantically invalid program, the compiler must provide an error message that points exactly to the source location of the first error that occurs in the program. I will get excited for/scared of LLMs when they can tackle this kind of problem. But I don't believe they can because of the fundamental nature of their design, which is both backward looking (thus not better than the human state of the art) and lacks human intuition and self awareness. Or perhaps rather I believe that the prompt that would be required to get an LLM to produce such a program is a problem of at least equivalent complexity to implementing the program without an LLM.

Xenoamorphous 2y ago
> Here is an example of a task that I do not believe this generation of LLMs can ever do but that is possible for a human That’s possible for a highly intelligent, extensively trained, very small subset of humans.
- hatefulmoron 2y ago
  If you took the intersection of every human's abilities you'd be left with a very unimpressive set. That also ignores the fact that the small set of humans capable of building programming languages and compilers is a consequence of specialization and lack of interest. There are plenty of humans that are capable of learning how to do it. LLMs, on the other hand, are both specialized for the task and aren't lazy or uninterested.
- luckydata 2y ago
  does it mean people that can build languages and compilers are not humans? What is the point you're trying to make?
  - 2y ago

sourcepluck 2y ago

Searle's Chinese Room Argument springs to mind: https://plato.stanford.edu/entries/chinese-room/ The idea that "human-like" behaviour will lead to self-awareness is both unproven (it can't be proven until it happens) and impossible to disprove (like Russell's teapot). Yet, one common assumption of many people running these companies or investing in them, or of some developers investing their time in these technologies, is precisely that some sort of explosion of superintelligence is likely, or even inevitable. It surely is possible, but stretching that to likely seems a bit much if you really think how imperfectly we understand things like consciousness and the mind. Of course there are people who have essentially religious reactions to the notion that there may be limits to certain domains of knowledge. Nonetheless, I think that's the reality we're faced with here.

abeppu 2y ago
> The idea that "human-like" behaviour will lead to self-awareness is both unproven (it can't be proven until it happens) and impossible to disprove (like Russell's teapot). I think Searle's view was that: - while it cannot be dis-_proven_, the Chinese Room argument was meant to provide reasons against believing it - the "it can't be proven until it happens" part is misunderstanding: you won't know if it happens because the objective, externally available attributes don't indicate whether self-awareness (or indeed awareness at all) is present
- sourcepluck 2y ago
  The short version of this is that I don't disagree with your interpretation of Searle, and my paragraphs immediately following the link weren't meant to be a direct description of his point with the Chinese Room thought experiment. > while it cannot be dis-_proven_, the Chinese Room argument was meant to provide reasons against believing it Yes, like Russell's teapot. I also think that's what Searle means. > the "it can't be proven until it happens" part is misunderstanding: you won't know if it happens because the objective, externally available attributes don't indicate whether self-awareness (or indeed awareness at all) is present Yes, agreed, I believe that's what Searle is saying too. I think I was maybe being ambiguous here - I wanted to say that even if you forgave the AI maximalists for ignoring all relevant philosophical work, the notion that "appearing human-like" inevitably tends to what would actually be "consciousness" or "intelligence" is more than a big claim. Searle goes further, and I'm not sure if I follow him all the way, personally, but it's a side point.

olalonde 2y ago

I feel the test for AGI should be more like: "go find a job and earn money" or "start a profitable business" or "pick a bachelor degree and complete it", etc.

rodgerd 2y ago
An LLM doing crypto spam/scamming has been making money by tricking Marc Andressen into boosting it. So to the degree that "scamming gullible billionaires and their fans" is a job, that's been done.
- rsanek 2y ago
  source? didn't find anything online about this.
  - rodgerd 2y ago
    Goatseus Maximus is what you're after.
- olalonde 2y ago
  That story was a bit blown out of proportion. He gave a research grant to the bot's creator: https://x.com/pmarca/status/1846374466101944629 https://x.com/pmarca/status/1846374466101944629
jedberg 2y ago
Can most humans do that? Find a job and earn money, probably. The other two? Not so much.
eichi 2y ago
This is people's true desire. Make something like that while handling critisisms and fitting products to the market.