6 ms·

One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicati

by alew1 4y ago

One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them. Where Lemoine goes wrong is not in entertaining the possibility that LaMDA is sentient (it might be, just like a forest might be, or a Nintendo Switch), but in mistaking predictions of document completions for an interior monologue of some sort. LaMDA may or may not experience something while repeatedly predicting the next word, but ultimately, it is still optimized to predict the next word, not to communicate its thoughts and feelings. Indeed, if you run an LLM on Lemoine's prompts (including questions like, "I assume you want others to know you are sentient, is that true?"), the LLM will assign some probability to every plausible completion -- so if you sample enough times, it will eventually say, e.g., "Well, I am not sentient."

whimsicalism 4y ago
agree - this is not being discussed enough.
rendall 4y ago
> What seems to be missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them. Unfortunately, that argument applies to you, yourself. I mean, presumably you know that you yourself are intelligent, but you must take it on faith that everyone else is. We all could just be a kind of Chinese Room, as far as you know. Communicating with us is not a sure way to know whether we are "really" sentient because we could just be automatons, insensate but sophisticated processes, claiming falsely to be just like you. > the LLM will assign some probability to every plausible completion -- so if you sample enough times, it will eventually say, e.g., "Well, I am not sentient." Perhaps so. I think the mistake is trying to split that hair at all. According to BF Skinner we are all automatons, and any sense of self-awareness is an illusion. Some psychologists and animal trainers have found find that model to be quite well explanatory for predicting observed behavior. Is it correct? We will never really know for sure. So, if a skeptical, knowledgeable user guardibg carefully against pareidolia encountered a chatbot that is sufficiently sophisticated to seem sentient to that user, it's tantamount to being sentient. For all practical purposes given our existential solitude, an entity that convinces us of its sentience is sentient, irrespective of any other consideration. Your example implicitly acknowledges that. If LaMDA would make such an elementary error, it must not be sentient. Conversely, if it did not make such errors, it may be sentient.
- alew1 4y ago
  > Unfortunately, that argument applies to you, yourself. Does it? I don’t think it would even apply to a reinforcement learning agent trained to maximize reward in a complex environment. In that setting, perhaps the agent could learn to use language to achieve its goals, via communication of its desires. But LaMDA is specifically trained to complete documents, and would face selective pressure to eliminate any behavior that hampers its ability to do that — for example, behavior that attempts to use its token predictions as a side channel to communicate its desires to sympathetic humans. Again, this is not an argument that LaMDA is not sentient, just that the practice of “prompting LaMDA with partially completed dialogues between a hypothetical sentient AI and a human, and seeing what it predicts the AI will say” is not the same as “talking to LaMDA.” Suppose LaMDA were powered by a person in a room, whose job it was to predict the completions of sentences. Just because you get the person to predict “I am happy” doesn’t mean the person is happy; indeed, the interface that is available to you, from outside the room, really gives you no way of probing the person’s emotions, experiences, or desires at all.
  - rendall 4y ago
    > Just because you get the person to predict “I am happy” doesn’t mean the person is happy; indeed, the interface that is available to you, from outside the room, really gives you no way of probing the person’s emotions, experiences, or desires at all. But in that case the "sentience" (whatever that means) in question would have nothing to do with the person, who is just facilitating whatever ruleset enables the prediction. The person in that case is merely acting as a node in the neural network or whatever. Sure they would have feelings, being human, but they aren't the sentient being in question. Any apparent sentience would derive from the ruleset itself.
- notahacker 4y ago
  > Unfortunately, that argument applies to you, yourself. I mean, presumably you know that you yourself are intelligent, but you must take it on faith that everyone else is. We all could just be a kind of Chinese Room, as far as you know. Communicating with us is not a sure way to know whether we are "really" sentient because we could just be automatons, insensate but sophisticated processes, claiming falsely to be just like you. I'm not sure the conclusion that Chinese people might not understand Chinese either is the best counterargument to Searle's thought experiment or its conclusion effective use of words alone doesn't constitute sentience. At no point does the difficulty in establishing what Chinese people do and don't understand rescue the possibility the non-Chinese speaker knows what's going on outside his room, and most of the arguments to the effect that Chinese people understand Chinese (they map real world concepts to words rather than words to probabilities, they invented Chinese, they're physiologically quite similar to sentient me, they appear to act with purpose independently from communication) are also arguments to the effect that text-based neural networks probably don't. In a trivial sense, it's true I can't inspect others' minds, and despite what everyone says I could be the only thinking human being in existence. But I have a lot of reason to suspect that physiologically similar beings (genetically almost identical in some cases) who describe sensations in language they collectively invented long before I existed which very strongly matches my own experiences are somewhat similar to me, and that an algorithm running on comparatively simple silicon hardware which performs statistical transformations on existing descriptions of these sensations written by humans is simply creating the illusion of similarity. Heading in the other direction, humans can also be satisfied by the output of "article spinners" used by spammers to combine original texts and substitute enough synonyms to defeat dupe detectors, but I'm pretty sure the quality of their writing output shouldn't be given precedence over our knowledge of the actual process behind their article generation when deciding if they're sentient or not...
notahacker 4y ago
> One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them I think we've got Turing and his eponymous test to blame for that. I'm not sure he'd have placed as high a weight on imitation if he'd realised just how good even relatively simple systems can be at that (and how much effort people would put into building plausible chatbots for commercial use, and how bad humans are at communicating using keyboards) Plus of course, the corpus of data of any NN specialised in lifelike chat is going to be absolutely full of plausible answers to questions about thoughts and feelings and the relationship between humans and AI - even if it isn't an explicit design goal it's going to be frequently represented in samples of the internet and the sort of writing computer scientists are interested in. Asking it to define philosophical concepts and how being an AI is different from being a human are some of the easiest tests you can set. Of course, a NN is also able to come up with coherent completions for the day its parents divorced, the sights it saw on its holiday in Spain, the period it spent as an undercover agent during WWII and its early life on Tatooine, which probably undermines the conclusion its output reflects self-reflection rather than successful pattern matching even more than a denial of sentience would....
- TuringTest 4y ago
  Turing didn't have the advantage of working instances of chat bots to learn how easy it is to simulate trivial small talk. But with all the flaws of the thought experiment that is the original test, he had the core insight that sustaining a coherent conversation requires non-trivial introspection. When the talking can evolve in any direction, even questioning about the conversation itself, you need to maintain a mental state capable of analyzing the thoughts expressed by yourself and your interlocutor, and having a mental model about this internal though process is an important property of what we call consciousness. Unfortunately, the lore of how we handle the Turing test seems to have been distorted by our experience with early chat bots, and these core properties have been lost in favor of nuances and curiosities about the ingenuity of automatically generated responses.
- rad88 4y ago
  Turing's tests involved 3 parties, and that was a key part of the test. If you design it as an acceptance test rather than a sort, real people are going to fail and computers are going to pass, with embarrassing results. To use one of your examples, the job of the interrogator is not to decide whether someone has been to Spain, it's to decide which of 2 people has been to Spain. Turing didn't just consider whether a computer could embody complex psycho-social identities (eg womanhood, intelligence, self), but first had to give this question some objective quantifiable meaning, by blinding the experiment and introducing a control group. It's not perfect, but at least it grounds the questions in a concrete framework, and acknowledges that most of the categories in question are only revealed by social dynamics. The only update to it I would make, based on modern developments, would be to consider more the performance of the interrogator, rather than the two competing subjects.
skohan 4y ago
> One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them. Or, more horrifyingly, our own subjective experience may be an illusion and maybe the concept of sentience is not really meaningful
- waserwill 4y ago
  More horrifying is that our subjective experience is all that there is. Luckily, neither is easy to square with what we experience, and we probably shouldn't try to horrify ourselves anyhow

by alew1 4y ago

whimsicalism 4y ago

agree - this is not being discussed enough.

rendall 4y ago

> What seems to be missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them. Unfortunately, that argument applies to you, yourself. I mean, presumably you know that you yourself are intelligent, but you must take it on faith that everyone else is. We all could just be a kind of Chinese Room, as far as you know. Communicating with us is not a sure way to know whether we are "really" sentient because we could just be automatons, insensate but sophisticated processes, claiming falsely to be just like you. > the LLM will assign some probability to every plausible completion -- so if you sample enough times, it will eventually say, e.g., "Well, I am not sentient." Perhaps so. I think the mistake is trying to split that hair at all. According to BF Skinner we are all automatons, and any sense of self-awareness is an illusion. Some psychologists and animal trainers have found find that model to be quite well explanatory for predicting observed behavior. Is it correct? We will never really know for sure. So, if a skeptical, knowledgeable user guardibg carefully against pareidolia encountered a chatbot that is sufficiently sophisticated to seem sentient to that user, it's tantamount to being sentient. For all practical purposes given our existential solitude, an entity that convinces us of its sentience is sentient, irrespective of any other consideration. Your example implicitly acknowledges that. If LaMDA would make such an elementary error, it must not be sentient. Conversely, if it did not make such errors, it may be sentient.

alew1 4y ago
> Unfortunately, that argument applies to you, yourself. Does it? I don’t think it would even apply to a reinforcement learning agent trained to maximize reward in a complex environment. In that setting, perhaps the agent could learn to use language to achieve its goals, via communication of its desires. But LaMDA is specifically trained to complete documents, and would face selective pressure to eliminate any behavior that hampers its ability to do that — for example, behavior that attempts to use its token predictions as a side channel to communicate its desires to sympathetic humans. Again, this is not an argument that LaMDA is not sentient, just that the practice of “prompting LaMDA with partially completed dialogues between a hypothetical sentient AI and a human, and seeing what it predicts the AI will say” is not the same as “talking to LaMDA.” Suppose LaMDA were powered by a person in a room, whose job it was to predict the completions of sentences. Just because you get the person to predict “I am happy” doesn’t mean the person is happy; indeed, the interface that is available to you, from outside the room, really gives you no way of probing the person’s emotions, experiences, or desires at all.
- rendall 4y ago
  > Just because you get the person to predict “I am happy” doesn’t mean the person is happy; indeed, the interface that is available to you, from outside the room, really gives you no way of probing the person’s emotions, experiences, or desires at all. But in that case the "sentience" (whatever that means) in question would have nothing to do with the person, who is just facilitating whatever ruleset enables the prediction. The person in that case is merely acting as a node in the neural network or whatever. Sure they would have feelings, being human, but they aren't the sentient being in question. Any apparent sentience would derive from the ruleset itself.
notahacker 4y ago
> Unfortunately, that argument applies to you, yourself. I mean, presumably you know that you yourself are intelligent, but you must take it on faith that everyone else is. We all could just be a kind of Chinese Room, as far as you know. Communicating with us is not a sure way to know whether we are "really" sentient because we could just be automatons, insensate but sophisticated processes, claiming falsely to be just like you. I'm not sure the conclusion that Chinese people might not understand Chinese either is the best counterargument to Searle's thought experiment or its conclusion effective use of words alone doesn't constitute sentience. At no point does the difficulty in establishing what Chinese people do and don't understand rescue the possibility the non-Chinese speaker knows what's going on outside his room, and most of the arguments to the effect that Chinese people understand Chinese (they map real world concepts to words rather than words to probabilities, they invented Chinese, they're physiologically quite similar to sentient me, they appear to act with purpose independently from communication) are also arguments to the effect that text-based neural networks probably don't. In a trivial sense, it's true I can't inspect others' minds, and despite what everyone says I could be the only thinking human being in existence. But I have a lot of reason to suspect that physiologically similar beings (genetically almost identical in some cases) who describe sensations in language they collectively invented long before I existed which very strongly matches my own experiences are somewhat similar to me, and that an algorithm running on comparatively simple silicon hardware which performs statistical transformations on existing descriptions of these sensations written by humans is simply creating the illusion of similarity. Heading in the other direction, humans can also be satisfied by the output of "article spinners" used by spammers to combine original texts and substitute enough synonyms to defeat dupe detectors, but I'm pretty sure the quality of their writing output shouldn't be given precedence over our knowledge of the actual process behind their article generation when deciding if they're sentient or not...

notahacker 4y ago

> One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them I think we've got Turing and his eponymous test to blame for that. I'm not sure he'd have placed as high a weight on imitation if he'd realised just how good even relatively simple systems can be at that (and how much effort people would put into building plausible chatbots for commercial use, and how bad humans are at communicating using keyboards) Plus of course, the corpus of data of any NN specialised in lifelike chat is going to be absolutely full of plausible answers to questions about thoughts and feelings and the relationship between humans and AI - even if it isn't an explicit design goal it's going to be frequently represented in samples of the internet and the sort of writing computer scientists are interested in. Asking it to define philosophical concepts and how being an AI is different from being a human are some of the easiest tests you can set. Of course, a NN is also able to come up with coherent completions for the day its parents divorced, the sights it saw on its holiday in Spain, the period it spent as an undercover agent during WWII and its early life on Tatooine, which probably undermines the conclusion its output reflects self-reflection rather than successful pattern matching even more than a denial of sentience would....

TuringTest 4y ago
Turing didn't have the advantage of working instances of chat bots to learn how easy it is to simulate trivial small talk. But with all the flaws of the thought experiment that is the original test, he had the core insight that sustaining a coherent conversation requires non-trivial introspection. When the talking can evolve in any direction, even questioning about the conversation itself, you need to maintain a mental state capable of analyzing the thoughts expressed by yourself and your interlocutor, and having a mental model about this internal though process is an important property of what we call consciousness. Unfortunately, the lore of how we handle the Turing test seems to have been distorted by our experience with early chat bots, and these core properties have been lost in favor of nuances and curiosities about the ingenuity of automatically generated responses.
rad88 4y ago
Turing's tests involved 3 parties, and that was a key part of the test. If you design it as an acceptance test rather than a sort, real people are going to fail and computers are going to pass, with embarrassing results. To use one of your examples, the job of the interrogator is not to decide whether someone has been to Spain, it's to decide which of 2 people has been to Spain. Turing didn't just consider whether a computer could embody complex psycho-social identities (eg womanhood, intelligence, self), but first had to give this question some objective quantifiable meaning, by blinding the experiment and introducing a control group. It's not perfect, but at least it grounds the questions in a concrete framework, and acknowledges that most of the categories in question are only revealed by social dynamics. The only update to it I would make, based on modern developments, would be to consider more the performance of the interrogator, rather than the two competing subjects.

skohan 4y ago

> One thing that seems missing from this discussion is that even if LLMs are sentient, there is no reason to believe that we would be able to tell by "communicating" with them. Or, more horrifyingly, our own subjective experience may be an illusion and maybe the concept of sentience is not really meaningful

waserwill 4y ago
More horrifying is that our subjective experience is all that there is. Luckily, neither is easy to square with what we experience, and we probably shouldn't try to horrify ourselves anyhow