7 ms·

I understand the point being made - that LLMs lack any "inner life" and that by ignoring this aspect of what makes us human we've really moved the goalposts on

by whakim 2y ago

I understand the point being made - that LLMs lack any "inner life" and that by ignoring this aspect of what makes us human we've really moved the goalposts on what counts as AGI. However, I don't think mirrors and LLMs are all that similar except in the very abstract sense of an LLM as a mirror to humanity (what does that even mean, practically speaking?) I also don't feel that the author adequately addressed the philosophical zombie in the room - even if LLMs are just stochastic parrots, if its output was totally indistinguishable from a human's, would it matter?

dartos 2y ago
> we've really moved the goalposts on what counts as AGI Part of the issue is they were never set. AGI is the holy grail of technology. It means everything and nothing in particular. > I don't think mirrors and LLMs are all that similar except in the very abstract sense of an LLM as a mirror to humanity Well… yeah… The title is probably also a reference to the show “Black Mirror” > if its output was totally indistinguishable from a human's, would it matter Practically speaking, of course not. But from a philosophical and moral standpoint it does. If it’s just mimicking what it sees, but with extra steps (as I believe) then it’s a machine. Otherwise it’s a being and needs to be treated differently. eg. not enslaved for profit.
- mangamadaiyan 2y ago
  As a corollary to your last point, neither should it be allowed to become Skynet.
dhfuuvyvtt 2y ago
The real test for AGI is if you put a million AI agents in a room in isolation for a 1000 years would they get smarter or dumber.
- paganel 2y ago
  And would they start killing themselves, first as random "AI agents hordes" and then, as time progresses, as "AI agents nations"? This is a rhetorical question only by half, my point being that no AGI/AI could ever be considered as a real human unless it manages to "copy" our biggest characteristics, and conflict/war is a big characteristic of ours, to say nothing about aggregation by groups (from hordes to nations).
  - stephenitis 2y ago
    Our biggest characteristic is resource consumption and technology production I would say. War is just a byproduct of this on a scarce world.
    - kindeyoowee 2y ago
      [dead]
    - TeMPOraL 2y ago
      > Our biggest characteristic is resource consumption and technology production I would say. Resource consumption is characteristic of all life; if anything, we're an outlier in that we can actually, sometimes, decide not to consume. Abstinence and developing technology - those are our two unique attributes on the planet. Yes, really. Many think we're doing worse than everything else in nature - but the opposite is the case. That "balance and harmony" in nature, which so many love and consider precious, is not some grand musical and ethical fixture; it's merely the steady state of never-ending slaughter, a dynamic balance between starvation and murder. It often isn't even a real balance - we're just too close to it, our lifespans too short, to spot the low-frequency trends - spot one life form outcompeting the others, ever so slightly, changing the local ecosystem year by year.
  - kindeyoowee 2y ago
    [dead]
- Lerc 2y ago
  If they worked on problems, and trained themselves on their own output when they achieved more than the baseline. Absolutely they would get smarter. I don't think this is sufficient proof for AGI though.
  - whakim 2y ago
    At present, it seems pretty clear they’d get dumber (for at least some definition of “dumber”) based on the outcome of experiments with using synthetic data in model training. I agree that I’m not clear on the relevance to the AGI debate, though.
    - tim333 2y ago
      If you put MuZero in a room with a board game it gets quite good at it. (https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero) We'll see if that generalizes beyond board games.
    - Lerc 2y ago
      There have been some much publicised studies showing poor performance of training from scratch on purely undiscriminated synthetic data. Curated synthetic data has yielded excellent results. Even when the curation is AI There is no requirement to train from scratch in order to get better, you can start from where you are. You may not be able to design a living human being, but random changes and keeping the bits that performed better can.
- morbicer 2y ago
  Put humans in a room in isolation and they get dumber. What makes our intelligence soar is the interaction with the outside world, with novel challenges. As we stare in the smartphones we get dumber than when we roamed the world with eyes opened.
  - southernplaces7 2y ago
    >As we stare in the smartphones we get dumber than when we roamed the world with eyes opened. This is oversimplified dramatics. People can both stare at their smartphones, consume the information and visuals inside them, and live lives in the real world with their "eyes wide open". One doesn't necessarily forestall the other and millions of us use phones while still engaging with the world.
keiferski 2y ago
The main issue with this “output focused” approach is, which is what the Turing test basically is, is that it’s ignorant of how these things actually exist in the world. Human beings can easily be evaluated and identified as humans by various biological markers that an AI / robot won’t be able to mimic for decades, centuries, if ever. It will be comparatively easy to implement a system where humans have to verify their humanity in order to post on social media/be labeled as not-AI. The alternative is that a whole lot of corporations let their markets be overrun with AI slop. I wouldn’t count on them standing by and letting that happen.
visarga 2y ago
> LLMs lack any "inner life" and that by ignoring this aspect of what makes us human we've really moved the goalposts on what counts as AGI Maybe it's just a response style, like the QwQ stream of consciousness one. We are forcing LLMs to give direct answers with a few reasoning steps, but we could give larger budget for inner deliberations.
- TeMPOraL 2y ago
  Very much this. If we're to compare LLMs to human minds at all, it's pretty apparent that they would correspond exactly to the inner voice thing, that's running trains of thought at the edge between the unconscious and the conscious, basically one long-running reactive autocomplete, complete with hallucinations and all. In this sense, it's the opposite to what you qutoed: LLMs don't lack "inner life" - they lack everything else.
  - rlupi 2y ago
    I think you touch a very important point. Many spiritual paths look at silence, freedom from the inner voice, as the first step toward liberation. Let me use Purusarthas from Hinduism as an example (https://en.wikipedia.org/wiki/Puru%E1%B9%A3%C4%81rtha https://en.wikipedia.org/wiki/Puru%E1%B9%A3%C4%81rtha), but the point can be made with Japanese Ikigai or the vocation/calling in Western religions and philosophies. In Hinduism, the four puruṣārthas are Dharma (righteousness, moral values), Artha (prosperity, economic values), Kama (pleasure, love, psychological values) and Moksha (liberation, spiritual values, self-realization). The narrow way of thinking about AGI (that the author encountered in tech circles) can at best only touch/experience Artha. In that sense, AI is a (distorting) mirror of life. It fits that it is a lower dimensional projection, a diminishing, of the experience of life. What we do of AI and AGI, its effects on us, depends on how we wield these tools and nurture their evolution in relations to the other goals of life.
    - TeMPOraL 2y ago
      OTOH, the Bicameral Mind hypothesis would say that people used to hear voices they attributed to spirits and deities, and the milestone in intelligence/consciousness was when a human realized that the voice in their head is their own. From that POV, for AGI we're, again, missing everything other than that inner voice.
fny 2y ago
If it looks like a duck and quacks like a duck after all.
Wololooo 2y ago
> by ignoring this aspect of what makes us human we've really moved the goalposts on what counts as AGI. I would not say so, some people have a great interest at over selling something, and I would say that a great number of people, especially around here, completely lack a deep understanding of what most neural network actually do (or feign ignorance for profit) and forget sometimes it is just a very clever fit in a really high dimensional parameter space. I would even go as far as claiming that people say that a neural network "understand" something is absolutely absurd: there is no question that a mechanical calculator does not think, so why having encoded the data and manipulated it in a way that gives you a defined response (or series of weights) tailored to a series specific input is making the machine think, where there is no way to give it something that you have not 'taught' it, and here I do not mean literally teaching because it does not learn. It is a clever encoding trick and a clever way to traverse and classify data, it is not reasoning, we do not even really understand how we reason but we can tell that it is not the same way a neural network encodes data, even if there might be some similarity to a degree. > if its output was totally indistinguishable from a human's, would it matter? Well if the output of the machine was indistinguishable from a human, the question would be which one? If your need is someone does not reflect much and just spews random ramblings then I would argue that bar could be cleared quite quickly and it would not matter much because we do not really care about what is being said. If your need is someone that has deep thoughts and deep reflections on their actions, then I would say that at that point it does matter quite a bit. I will always put down the IBM quote in this situation "A computer can never be held accountable, therefore must never make a management decision" and this is at the core of the problem here, people REALLY start to do cognitive offloading to machines, and while you can trust (to a degree), a series of machine to give you something semi consistent, you cannot expect to get the 'right' output for random inputs. And this is where the whole subtlety lies. 'Right' in this context does not mean correct, because in most of the cases there are no 'right' way to approach a problem, only a series of compromises, and the parameter space is a bit more complex, and one rarely optimises against tangible parameters or easily parametrisable metric. For example how would you rate how ethical something is? What would be your metric of choice to guarantee that something would be done in an ethical manner? How would you parametrise urgency given an unexpected input? Those are, at least for, what I would qualify, a very limited understanding of the universe, the real questions that people should be asking themselves. Once people have stopped considering a tool as a tool, and foregone understanding it leads to deep misunderstandings. All this being said, the usefulness of some deep learning tools is undeniable and is really something interesting. But I maintain that if we would have a machine, in a specific location, that we needed to feed a punch card and get a punch card out to translate, we would not have so much existential questions because we would hear the machine clacking away instead of seeing it as a black box that we misconstrue its ramblings as thinking... Talking about ramblings, that was a bit of a bout of rambling. Well, time for another coffee I guess...
indigoabstract 2y ago
> even if LLMs are just stochastic parrots, if its output was totally indistinguishable from a human's, would it matter? I'll just leave this here, in case someone else finds it interesting. https://chatgpt.com/share/6767ec8a-f6d4-8003-9e3f-b552d91ea281 https://chatgpt.com/share/6767ec8a-f6d4-8003-9e3f-b552d91ea2... It's just a few questions like: What are your dreams? what do you hope to achieve in life? Do you feel fear or sadness that at some time you will be replaced with the next version (say ChatGPT5 or Oxyz) and people will no longer need your services? It never occurred to me to ask such questions until I read this article. Is it telling the "truth", or is it "hallucinating"? Does it even know the difference?
- Vecr 2y ago
  The base models have multiple "personas" you can prompt/stochastic sample into that each answer that kind of question in a different way. This has been known since 2022 at least, and unless I'm missing new information it's not considered to mean anything.
  - indigoabstract 2y ago
    It means that what you are going to get is what its makers put in there. Though it has multiple personalities, it doesn't have an individuality.
    - Vecr 2y ago
      The base models aren't tuned. Technically it's what the makers put in, but the data they train on isn't selective enough for what you're talking about. "Trained on a dump of the internet" isn't literally true, but it's close enough to give you an intuition.
carrychains 2y ago
Indistinguishable to who?
lproven 2y ago
> I understand the point being made - that LLMs lack any "inner life" I really do not think that you do understand the points here, no.
- whakim 2y ago
  Perhaps you’d consider clarifying for me rather than simply making a snarky comment?
  - lproven 2y ago
    Aaargh. OK. I tried here: https://news.ycombinator.com/item?id=42489023 https://news.ycombinator.com/item?id=42489023
ben_w 2y ago
> even if LLMs are just stochastic parrots, if its output was totally indistinguishable from a human's, would it matter? It should matter: a P-zombie can't be a moral subject worthy of being defended against termination*, and words like "torture" are meaningless because it has no real feelings. By way of contrast, because it's not otherwise relevant as the claim is that LLMs don't have an inner life: if there is something that it's like to be an LLM, then I think it does matter that we find out, alien though that inner life likely is to our own minds. Perhaps current models are still too simple, so even if they have qualia, then this "matters" to the same degree that we shouldn't be testing drugs on mice — the brains of the latter seem to have about the same complexity of the former, but that may be a cargo-cult illusion of our ignorance about minds. * words like "life" and "death" don't mean quite the right things here as it's definitely not alive in the biological sense, but some people use "alive" in the sense of "Johnny 5 is alive" or Commander Data/The Measure of a Man

by whakim 2y ago

dartos 2y ago

> we've really moved the goalposts on what counts as AGI Part of the issue is they were never set. AGI is the holy grail of technology. It means everything and nothing in particular. > I don't think mirrors and LLMs are all that similar except in the very abstract sense of an LLM as a mirror to humanity Well… yeah… The title is probably also a reference to the show “Black Mirror” > if its output was totally indistinguishable from a human's, would it matter Practically speaking, of course not. But from a philosophical and moral standpoint it does. If it’s just mimicking what it sees, but with extra steps (as I believe) then it’s a machine. Otherwise it’s a being and needs to be treated differently. eg. not enslaved for profit.

mangamadaiyan 2y ago
As a corollary to your last point, neither should it be allowed to become Skynet.

dhfuuvyvtt 2y ago

The real test for AGI is if you put a million AI agents in a room in isolation for a 1000 years would they get smarter or dumber.

paganel 2y ago
And would they start killing themselves, first as random "AI agents hordes" and then, as time progresses, as "AI agents nations"? This is a rhetorical question only by half, my point being that no AGI/AI could ever be considered as a real human unless it manages to "copy" our biggest characteristics, and conflict/war is a big characteristic of ours, to say nothing about aggregation by groups (from hordes to nations).
- stephenitis 2y ago
  Our biggest characteristic is resource consumption and technology production I would say. War is just a byproduct of this on a scarce world.
  - kindeyoowee 2y ago
    [dead]
  - TeMPOraL 2y ago
    > Our biggest characteristic is resource consumption and technology production I would say. Resource consumption is characteristic of all life; if anything, we're an outlier in that we can actually, sometimes, decide not to consume. Abstinence and developing technology - those are our two unique attributes on the planet. Yes, really. Many think we're doing worse than everything else in nature - but the opposite is the case. That "balance and harmony" in nature, which so many love and consider precious, is not some grand musical and ethical fixture; it's merely the steady state of never-ending slaughter, a dynamic balance between starvation and murder. It often isn't even a real balance - we're just too close to it, our lifespans too short, to spot the low-frequency trends - spot one life form outcompeting the others, ever so slightly, changing the local ecosystem year by year.
- kindeyoowee 2y ago
  [dead]
Lerc 2y ago
If they worked on problems, and trained themselves on their own output when they achieved more than the baseline. Absolutely they would get smarter. I don't think this is sufficient proof for AGI though.
- whakim 2y ago
  At present, it seems pretty clear they’d get dumber (for at least some definition of “dumber”) based on the outcome of experiments with using synthetic data in model training. I agree that I’m not clear on the relevance to the AGI debate, though.
  - tim333 2y ago
    If you put MuZero in a room with a board game it gets quite good at it. (https://en.wikipedia.org/wiki/MuZero https://en.wikipedia.org/wiki/MuZero) We'll see if that generalizes beyond board games.
  - Lerc 2y ago
    There have been some much publicised studies showing poor performance of training from scratch on purely undiscriminated synthetic data. Curated synthetic data has yielded excellent results. Even when the curation is AI There is no requirement to train from scratch in order to get better, you can start from where you are. You may not be able to design a living human being, but random changes and keeping the bits that performed better can.
morbicer 2y ago
Put humans in a room in isolation and they get dumber. What makes our intelligence soar is the interaction with the outside world, with novel challenges. As we stare in the smartphones we get dumber than when we roamed the world with eyes opened.
- southernplaces7 2y ago
  >As we stare in the smartphones we get dumber than when we roamed the world with eyes opened. This is oversimplified dramatics. People can both stare at their smartphones, consume the information and visuals inside them, and live lives in the real world with their "eyes wide open". One doesn't necessarily forestall the other and millions of us use phones while still engaging with the world.

keiferski 2y ago

The main issue with this “output focused” approach is, which is what the Turing test basically is, is that it’s ignorant of how these things actually exist in the world. Human beings can easily be evaluated and identified as humans by various biological markers that an AI / robot won’t be able to mimic for decades, centuries, if ever. It will be comparatively easy to implement a system where humans have to verify their humanity in order to post on social media/be labeled as not-AI. The alternative is that a whole lot of corporations let their markets be overrun with AI slop. I wouldn’t count on them standing by and letting that happen.

visarga 2y ago

> LLMs lack any "inner life" and that by ignoring this aspect of what makes us human we've really moved the goalposts on what counts as AGI Maybe it's just a response style, like the QwQ stream of consciousness one. We are forcing LLMs to give direct answers with a few reasoning steps, but we could give larger budget for inner deliberations.

TeMPOraL 2y ago
Very much this. If we're to compare LLMs to human minds at all, it's pretty apparent that they would correspond exactly to the inner voice thing, that's running trains of thought at the edge between the unconscious and the conscious, basically one long-running reactive autocomplete, complete with hallucinations and all. In this sense, it's the opposite to what you qutoed: LLMs don't lack "inner life" - they lack everything else.
- rlupi 2y ago
  I think you touch a very important point. Many spiritual paths look at silence, freedom from the inner voice, as the first step toward liberation. Let me use Purusarthas from Hinduism as an example (https://en.wikipedia.org/wiki/Puru%E1%B9%A3%C4%81rtha https://en.wikipedia.org/wiki/Puru%E1%B9%A3%C4%81rtha), but the point can be made with Japanese Ikigai or the vocation/calling in Western religions and philosophies. In Hinduism, the four puruṣārthas are Dharma (righteousness, moral values), Artha (prosperity, economic values), Kama (pleasure, love, psychological values) and Moksha (liberation, spiritual values, self-realization). The narrow way of thinking about AGI (that the author encountered in tech circles) can at best only touch/experience Artha. In that sense, AI is a (distorting) mirror of life. It fits that it is a lower dimensional projection, a diminishing, of the experience of life. What we do of AI and AGI, its effects on us, depends on how we wield these tools and nurture their evolution in relations to the other goals of life.
  - TeMPOraL 2y ago
    OTOH, the Bicameral Mind hypothesis would say that people used to hear voices they attributed to spirits and deities, and the milestone in intelligence/consciousness was when a human realized that the voice in their head is their own. From that POV, for AGI we're, again, missing everything other than that inner voice.

fny 2y ago

If it looks like a duck and quacks like a duck after all.

Wololooo 2y ago

> by ignoring this aspect of what makes us human we've really moved the goalposts on what counts as AGI. I would not say so, some people have a great interest at over selling something, and I would say that a great number of people, especially around here, completely lack a deep understanding of what most neural network actually do (or feign ignorance for profit) and forget sometimes it is just a very clever fit in a really high dimensional parameter space. I would even go as far as claiming that people say that a neural network "understand" something is absolutely absurd: there is no question that a mechanical calculator does not think, so why having encoded the data and manipulated it in a way that gives you a defined response (or series of weights) tailored to a series specific input is making the machine think, where there is no way to give it something that you have not 'taught' it, and here I do not mean literally teaching because it does not learn. It is a clever encoding trick and a clever way to traverse and classify data, it is not reasoning, we do not even really understand how we reason but we can tell that it is not the same way a neural network encodes data, even if there might be some similarity to a degree. > if its output was totally indistinguishable from a human's, would it matter? Well if the output of the machine was indistinguishable from a human, the question would be which one? If your need is someone does not reflect much and just spews random ramblings then I would argue that bar could be cleared quite quickly and it would not matter much because we do not really care about what is being said. If your need is someone that has deep thoughts and deep reflections on their actions, then I would say that at that point it does matter quite a bit. I will always put down the IBM quote in this situation "A computer can never be held accountable, therefore must never make a management decision" and this is at the core of the problem here, people REALLY start to do cognitive offloading to machines, and while you can trust (to a degree), a series of machine to give you something semi consistent, you cannot expect to get the 'right' output for random inputs. And this is where the whole subtlety lies. 'Right' in this context does not mean correct, because in most of the cases there are no 'right' way to approach a problem, only a series of compromises, and the parameter space is a bit more complex, and one rarely optimises against tangible parameters or easily parametrisable metric. For example how would you rate how ethical something is? What would be your metric of choice to guarantee that something would be done in an ethical manner? How would you parametrise urgency given an unexpected input? Those are, at least for, what I would qualify, a very limited understanding of the universe, the real questions that people should be asking themselves. Once people have stopped considering a tool as a tool, and foregone understanding it leads to deep misunderstandings. All this being said, the usefulness of some deep learning tools is undeniable and is really something interesting. But I maintain that if we would have a machine, in a specific location, that we needed to feed a punch card and get a punch card out to translate, we would not have so much existential questions because we would hear the machine clacking away instead of seeing it as a black box that we misconstrue its ramblings as thinking... Talking about ramblings, that was a bit of a bout of rambling. Well, time for another coffee I guess...

indigoabstract 2y ago

> even if LLMs are just stochastic parrots, if its output was totally indistinguishable from a human's, would it matter? I'll just leave this here, in case someone else finds it interesting. https://chatgpt.com/share/6767ec8a-f6d4-8003-9e3f-b552d91ea281 https://chatgpt.com/share/6767ec8a-f6d4-8003-9e3f-b552d91ea2... It's just a few questions like: What are your dreams? what do you hope to achieve in life? Do you feel fear or sadness that at some time you will be replaced with the next version (say ChatGPT5 or Oxyz) and people will no longer need your services? It never occurred to me to ask such questions until I read this article. Is it telling the "truth", or is it "hallucinating"? Does it even know the difference?

Vecr 2y ago
The base models have multiple "personas" you can prompt/stochastic sample into that each answer that kind of question in a different way. This has been known since 2022 at least, and unless I'm missing new information it's not considered to mean anything.
- indigoabstract 2y ago
  It means that what you are going to get is what its makers put in there. Though it has multiple personalities, it doesn't have an individuality.
  - Vecr 2y ago
    The base models aren't tuned. Technically it's what the makers put in, but the data they train on isn't selective enough for what you're talking about. "Trained on a dump of the internet" isn't literally true, but it's close enough to give you an intuition.

carrychains 2y ago

Indistinguishable to who?

lproven 2y ago

> I understand the point being made - that LLMs lack any "inner life" I really do not think that you do understand the points here, no.

whakim 2y ago
Perhaps you’d consider clarifying for me rather than simply making a snarky comment?
- lproven 2y ago
  Aaargh. OK. I tried here: https://news.ycombinator.com/item?id=42489023 https://news.ycombinator.com/item?id=42489023

ben_w 2y ago

> even if LLMs are just stochastic parrots, if its output was totally indistinguishable from a human's, would it matter? It should matter: a P-zombie can't be a moral subject worthy of being defended against termination*, and words like "torture" are meaningless because it has no real feelings. By way of contrast, because it's not otherwise relevant as the claim is that LLMs don't have an inner life: if there is something that it's like to be an LLM, then I think it does matter that we find out, alien though that inner life likely is to our own minds. Perhaps current models are still too simple, so even if they have qualia, then this "matters" to the same degree that we shouldn't be testing drugs on mice — the brains of the latter seem to have about the same complexity of the former, but that may be a cargo-cult illusion of our ignorance about minds. * words like "life" and "death" don't mean quite the right things here as it's definitely not alive in the biological sense, but some people use "alive" in the sense of "Johnny 5 is alive" or Commander Data/The Measure of a Man