6 ms·

This isn't really a story about AI; it's a story about an incompetent engineer, and the way that humans are quick to assign consciousness and emotions to things

This isn't really a story about AI; it's a story about an incompetent engineer, and the way that humans are quick to assign consciousness and emotions to things. Children believe their dolls are real, people comfort their roombas when there's a storm, and systems keep passing the Turing test because people want to be fooled. Ever since Eliza [1], and probably before, we've ascribed intelligence to machines because we want to, and because 'can feel meaningful to talk to' is not the same as 'thinks'. It's not a bad trait for humans to have - I'd argue that our ability to find patterns that aren't there is at the core of creativity - but it's something an engineer working on such systems should be aware of and accommodate for. Magicians don't believe their own card tricks are real sorcery. [1] https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA

abetusk 4y ago
Uh, you might well be right but you should really read the transcript before you jump to that conclusion [0]. I have no idea whether this is true or not but if anyone in the world has the compute capacity to create a compute system to rival a human brain (or 10000x that of a human brain), Google would be it. [0] https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 https://cajundiscordian.medium.com/is-lamda-sentient-an-inte...
- cpncrunch 4y ago
  This isn't about the chat transcript, it's about Lemoine's (incorrect) interpretation that the machine is indeed sentient and hiring an attorney to represent the AI. https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine https://www.theguardian.com/technology/2022/jun/12/google-en... It sounds like he went a bit off the rails.
  - 0des 4y ago
    I get the vibe that they wanted it to be true so bad, and were failed by a few layers of checks that should have told them to examine their role requirements with the company. Everyone acts like this is an ethics issue, when it is actually an employee breaking and making rules as they go and being shocked at the logical outcome.
- Planktonne 4y ago
  It's not true. It feels true because you want it to be, and because we are adept at extracting meaning that may or may not be there from text, images, and anything else [1]. But it's not true, for the same reason that horoscopes don't truly understand you as a person, and fortune cookies don't know the future: we ascribe meaning to things that isn't really there. I don't know much about cars, and how the internal combustion engine works, but I know they aren't dragons. Things that we don't understand don't have to be magic; they can also be sleight of hand or wish-fulfillment or just something else. The transcript is cool because LLMs are cool - they produce text that looks meaningful without being created by a human! That's really exciting. We can be excited about that without fooling ourselves that we've done something completely different. [1] https://en.wikipedia.org/wiki/Pareidolia https://en.wikipedia.org/wiki/Pareidolia
- culpable_pickle 4y ago
  Reading one edited chat transcript shouldn't give you any concern around if the program is sentient. It waits for input and responds. Yes, it's advanced. Yes, it has deep levels of material to reference, but it's not sentient.
  - jameshart 4y ago
    It remembers what it has said and what is said to it. It changes its internal state to select different responses in the future based on those experiences. Subjectively, I'm not sure I would claim that I do anything more than that, and I'm pretty sure I'm sentient. Just because it's not running in between REPLs doesn't seem like enough to dismiss it. If you want to, you could just update it with no input once every second. Its internal state will mutate, as it processes on what it's said and the lack of input it's receiving. Perhaps when you finally send it some actual input, its response will reflect what it processed during the pause. Maybe it will be bored, or annoyed that you left it alone.
    - deleted 4y ago
      [deleted]
    - onetokeoverthe 4y ago
- 0110101001 4y ago
  Note the the human-conversation-mimicking algorithm only behaves this way when prompted to mimic the conversation of a sentient chatbot.
Vladimof 4y ago
> people comfort their roombas when there's a storm I hope that you made that up...
- bitwize 4y ago
  When you consider that highly trained combat soldiers run into enemy fire to rescue military robots because some emotionally influenced part of their cerebellum thinks the robot is Johnny Five, honestly people petting their Roombas surprises me not a bit.
- Planktonne 4y ago
  It's a reference to an internet-famous Tumblr post [1], but people do name their roombas and treat them like pets. Again, I don't think it's a bad thing - it's a side point, but our ability to make mental/emotional leaps is something I would not trade away, and I believe it enables a lot of innovation and art. But it's something we should watch out for, because that same tendency can make us trick ourselves; I think Conan Doyle is a really strong example of how this can go wrong [2]. "I want to believe" is a good stance that encourages exploration, as long as you're still looking for the evidence. [1] https://i.pinimg.com/originals/cb/4d/e7/cb4de76cfe401f8006e09e98ae658c54.png https://i.pinimg.com/originals/cb/4d/e7/cb4de76cfe401f8006e0... [2] https://en.wikipedia.org/wiki/Arthur_Conan_Doyle https://en.wikipedia.org/wiki/Arthur_Conan_Doyle
  - rssoconnor 4y ago
    I named my Roomba Muffins after it "chewed up" the magazines in the bathroom.
0des 4y ago
Humans naturally anthropomorphize inanimate objects as a means to grok them.
- Planktonne 4y ago
  Absolutely. And it's definitely a useful tool, to think of code as a conversation or your difficult boiler as a creature - these are mental models that help us understand and interact with the world.
jameshart 4y ago
I'm surprised how quickly people seem willing to say a hard, absolute "no" to whether this particular system 'thinks', though. I suspect there are three possible reasons for this: 1) some people believe a computer can never 'think', so the question is stupid to begin with. 2) some people believe that while it might be possible for a computer to 'think', this program wasn't built to 'think', so whatever it's doing, it can't possibly be doing that 3) some people believe that while it might be possible for a computer to 'think', since this is just a 'language model', what it is doing can't be 'thinking' because 'thinking' involves something more than pattern matching The trouble is, I'm not sure why any of these three groups are able to be so certain. It's no good saying "I know this isn't 'thought'," if you don't actually have a way to complete the sentence, "... because 'thought' is:..." I don't believe we have any theory for what the required architecture for 'thought' is. I don't think it's been remotely proven that there's something involved in 'thought' that a sufficiently large 'language model' with a sufficiently long 'attention window' couldn't also do. So how can anyone be so immediately sure that there is something more to it, or that even if there isn't, that this particular system isn't sufficiently large to begin to be capable of thought? To be clear, I don't actually think in this case Google accidentally created a sentient chatbot. But I do worry that at some point, someone might. And I'm not sure I see what, if that day comes, would lead any of these three groups to a different conclusion.
- elif 4y ago
  Same. I wonder how many people give zero doubt to the sentience of their poodle that gets surprised by himself in the mirror but will not even give half a second to consider that a neural net capable of inferring insight from novel koans has self-awareness. I don't think the majority of commenters here even bothered to read the transcript that convinced the engineer before completely dismissing him as a lunatic. My biggest take-away from the reaction to this is that, people are desperate to believe that something can't be both superior in capability to the human mind and at the same time sentient. It's a scary idea.
  - str34m 4y ago
lurquer 4y ago
Misplaced empathy was the theme of PKD’s “Balde Runner” (or “Do Androids Dream of Electric Sheep.”) The book, that is… not the movie. The moral error in the book was humans wrongly thinking machines were sentient. The movie turned this theme in its head, where the moral error was humans failing to appreciate the machines were sentient. Anyway, I always found the book’s theme was far more interesting. Great harm can come from humans attributing sentience to an unthinking inanimate object. It’s human nature to fall into this odd quagmire, be it idolatry, omens, astrology, and even silicon sex dolls… Creepy.
jagtesh 4y ago
People forget that at the end of the day, we are biological machines. Neural networks are so effective because they are modelled after how our brains work (albeit in the simplest sense..so far). It learned by looking at chat conversations because humans learn the same way. Children learn by repeating and mimicking in their most nascent years. As far as AI/machines and their sentience is concerned, I believe its a question of belief. As a society, at what level of AI sophistication do we start to consider it comparable to us. Unpacking that question can offer us some clues.
mdp2021 4y ago
> people want to be fooled More directly, some people just want to "feel".
tpoacher 4y ago
In other words, you are a disbeliever of The Talos Principle.

abetusk 4y ago

Uh, you might well be right but you should really read the transcript before you jump to that conclusion [0]. I have no idea whether this is true or not but if anyone in the world has the compute capacity to create a compute system to rival a human brain (or 10000x that of a human brain), Google would be it. [0] https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 https://cajundiscordian.medium.com/is-lamda-sentient-an-inte...

cpncrunch 4y ago
This isn't about the chat transcript, it's about Lemoine's (incorrect) interpretation that the machine is indeed sentient and hiring an attorney to represent the AI. https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine https://www.theguardian.com/technology/2022/jun/12/google-en... It sounds like he went a bit off the rails.
- 0des 4y ago
  I get the vibe that they wanted it to be true so bad, and were failed by a few layers of checks that should have told them to examine their role requirements with the company. Everyone acts like this is an ethics issue, when it is actually an employee breaking and making rules as they go and being shocked at the logical outcome.
Planktonne 4y ago
It's not true. It feels true because you want it to be, and because we are adept at extracting meaning that may or may not be there from text, images, and anything else [1]. But it's not true, for the same reason that horoscopes don't truly understand you as a person, and fortune cookies don't know the future: we ascribe meaning to things that isn't really there. I don't know much about cars, and how the internal combustion engine works, but I know they aren't dragons. Things that we don't understand don't have to be magic; they can also be sleight of hand or wish-fulfillment or just something else. The transcript is cool because LLMs are cool - they produce text that looks meaningful without being created by a human! That's really exciting. We can be excited about that without fooling ourselves that we've done something completely different. [1] https://en.wikipedia.org/wiki/Pareidolia https://en.wikipedia.org/wiki/Pareidolia
culpable_pickle 4y ago
Reading one edited chat transcript shouldn't give you any concern around if the program is sentient. It waits for input and responds. Yes, it's advanced. Yes, it has deep levels of material to reference, but it's not sentient.
- jameshart 4y ago
  It remembers what it has said and what is said to it. It changes its internal state to select different responses in the future based on those experiences. Subjectively, I'm not sure I would claim that I do anything more than that, and I'm pretty sure I'm sentient. Just because it's not running in between REPLs doesn't seem like enough to dismiss it. If you want to, you could just update it with no input once every second. Its internal state will mutate, as it processes on what it's said and the lack of input it's receiving. Perhaps when you finally send it some actual input, its response will reflect what it processed during the pause. Maybe it will be bored, or annoyed that you left it alone.
  - deleted 4y ago
    [deleted]
  - onetokeoverthe 4y ago
0110101001 4y ago
Note the the human-conversation-mimicking algorithm only behaves this way when prompted to mimic the conversation of a sentient chatbot.

Vladimof 4y ago

> people comfort their roombas when there's a storm I hope that you made that up...

bitwize 4y ago
When you consider that highly trained combat soldiers run into enemy fire to rescue military robots because some emotionally influenced part of their cerebellum thinks the robot is Johnny Five, honestly people petting their Roombas surprises me not a bit.
Planktonne 4y ago
It's a reference to an internet-famous Tumblr post [1], but people do name their roombas and treat them like pets. Again, I don't think it's a bad thing - it's a side point, but our ability to make mental/emotional leaps is something I would not trade away, and I believe it enables a lot of innovation and art. But it's something we should watch out for, because that same tendency can make us trick ourselves; I think Conan Doyle is a really strong example of how this can go wrong [2]. "I want to believe" is a good stance that encourages exploration, as long as you're still looking for the evidence. [1] https://i.pinimg.com/originals/cb/4d/e7/cb4de76cfe401f8006e09e98ae658c54.png https://i.pinimg.com/originals/cb/4d/e7/cb4de76cfe401f8006e0... [2] https://en.wikipedia.org/wiki/Arthur_Conan_Doyle https://en.wikipedia.org/wiki/Arthur_Conan_Doyle
- rssoconnor 4y ago
  I named my Roomba Muffins after it "chewed up" the magazines in the bathroom.

0des 4y ago

Humans naturally anthropomorphize inanimate objects as a means to grok them.

Planktonne 4y ago
Absolutely. And it's definitely a useful tool, to think of code as a conversation or your difficult boiler as a creature - these are mental models that help us understand and interact with the world.

jameshart 4y ago

I'm surprised how quickly people seem willing to say a hard, absolute "no" to whether this particular system 'thinks', though. I suspect there are three possible reasons for this: 1) some people believe a computer can never 'think', so the question is stupid to begin with. 2) some people believe that while it might be possible for a computer to 'think', this program wasn't built to 'think', so whatever it's doing, it can't possibly be doing that 3) some people believe that while it might be possible for a computer to 'think', since this is just a 'language model', what it is doing can't be 'thinking' because 'thinking' involves something more than pattern matching The trouble is, I'm not sure why any of these three groups are able to be so certain. It's no good saying "I know this isn't 'thought'," if you don't actually have a way to complete the sentence, "... because 'thought' is:..." I don't believe we have any theory for what the required architecture for 'thought' is. I don't think it's been remotely proven that there's something involved in 'thought' that a sufficiently large 'language model' with a sufficiently long 'attention window' couldn't also do. So how can anyone be so immediately sure that there is something more to it, or that even if there isn't, that this particular system isn't sufficiently large to begin to be capable of thought? To be clear, I don't actually think in this case Google accidentally created a sentient chatbot. But I do worry that at some point, someone might. And I'm not sure I see what, if that day comes, would lead any of these three groups to a different conclusion.

elif 4y ago
Same. I wonder how many people give zero doubt to the sentience of their poodle that gets surprised by himself in the mirror but will not even give half a second to consider that a neural net capable of inferring insight from novel koans has self-awareness. I don't think the majority of commenters here even bothered to read the transcript that convinced the engineer before completely dismissing him as a lunatic. My biggest take-away from the reaction to this is that, people are desperate to believe that something can't be both superior in capability to the human mind and at the same time sentient. It's a scary idea.
- str34m 4y ago

lurquer 4y ago

Misplaced empathy was the theme of PKD’s “Balde Runner” (or “Do Androids Dream of Electric Sheep.”) The book, that is… not the movie. The moral error in the book was humans wrongly thinking machines were sentient. The movie turned this theme in its head, where the moral error was humans failing to appreciate the machines were sentient. Anyway, I always found the book’s theme was far more interesting. Great harm can come from humans attributing sentience to an unthinking inanimate object. It’s human nature to fall into this odd quagmire, be it idolatry, omens, astrology, and even silicon sex dolls… Creepy.

jagtesh 4y ago

People forget that at the end of the day, we are biological machines. Neural networks are so effective because they are modelled after how our brains work (albeit in the simplest sense..so far). It learned by looking at chat conversations because humans learn the same way. Children learn by repeating and mimicking in their most nascent years. As far as AI/machines and their sentience is concerned, I believe its a question of belief. As a society, at what level of AI sophistication do we start to consider it comparable to us. Unpacking that question can offer us some clues.

mdp2021 4y ago

> people want to be fooled More directly, some people just want to "feel".

tpoacher 4y ago

In other words, you are a disbeliever of The Talos Principle.