8 ms·

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this. I cannot see that it

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this. I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. It only matters if you probe beyond the realm at which you previously decided it matters (e.g roasting and eating it), at which point you are also insisting that it walk, quack and TASTE like a duck. So then you quantify that, change the goalposts, and assess every prospective duck against that. And if one comes along that matches all of those but doesn’t have wings, then if you deny it to be a duck FOR ALL INTENTS AND PURPOSES it simply means you didn’t specify your requirements. I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument. It seems to me to posit that to understand requires that the understandee is human. If that’s the case we just pick another word for it and move on with our lives.

JieJie 4y ago
I literally lost a friend of thirty years yesterday because she is wedded to the Chinese Room analogy so fiercely, she refuses to engage on the subject at all. For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with. (edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)
- bulkprotocol 4y ago
  You blew up a 30 year friendship over an...analogy?
  - JieJie 4y ago
    I didn't! Someone else did it to me. I was trying desperately not to. (edit: This is the kind of stuff I think my friends are watching and being informed by [0] as it was what they are posting in our common areas.) [0]: https://youtu.be/ro130m-f_yk https://youtu.be/ro130m-f_yk
    - roflyear 4y ago
      Why does it upset you if they disagree with you on this?
      - JieJie 4y ago
        It's not that they disagree with me, as there were many things we agreed on. I have read over it and it seems that they don't respect what I believe so much that their example of how they felt was this angry shouting man. But again, I was not the person who ended the friendship. In short, I'm upset that a person I thought was a friend disrespected my opinion about something so much, she stopped being my friend.
- brazzy 4y ago
  How can an disagreement over GPT possibly be something to end a friendship over? This sounds like you said "I cannot possibly be friends with someone who does not believe that LLMs are emerging AGI!", and people read it like that and are downvoting you. I'm gonna assume the situation was more complex, but still find it hard to imagine, how a disagreement over such an academic topic could end up destroying a friendship.
  - JieJie 4y ago
    It was a complex situation to be sure. I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently. If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear. I can barely speak with my artist friends on the issue these days due to their generative AI fears. Their emotions are completely intractable on the subject: AI art is theft. Period.
    - brazzy 4y ago
      If are is a major part of someone's identity (strongly implied by calling them an "artist"), then it's hardly surprising that they would feel threatened and react emotionally to something that massively disrupts the creation of art and the viability of making a living from it.
      - JieJie 4y ago
        I'm old enough to have had my design and publishing career evaporate underneath my feet because newsprint has mostly been replaced by the internet. I'm even old enough that I was part of replacing other people's careers with technology during the desktop publishing revolution. I remember when graphics tablets became a thing and how traditional pen and brush artists cried, "unfair!" I always felt empathy for them, even back then. I feel empathy for my artist friends as well. How can we ever have a better future if we keep trying to prevent change from disrupting our careers? In my experience, I've seen a lot of wailing and gnashing of teeth, but these changes have always come no matter how hard people fought. On the other hand, I see people who try to adapt to these changes do quite well. It's the part where I watch them hide their head in the sand that bothers me, not their anger. I want them to succeed. I think they have great eyes for color and line and those things are still super-valid long after the generative AI has replaced the pen and brush. I don't want to watch my artist friends suffer. I am old enough to know that the effort to stop generative AI is almost certainly going to fail (just as all the previous efforts to prevent change failed), and it is better to light a candle than to curse the darkness. The only constant is change.
rwmj 4y ago
Isn't the whole point of Searle raising the Chinese Rooom argument exactly so that these kinds of things can be debated?
mhink 4y ago
In my understanding of the Chinese Room example, the resolution to the argument is that the *human* may not understand Chinese, but the *system as a whole* can be said to understand it. With this in mind, I think asking whether ChatGPT *in and of itself* is "conscious" or has "agency" is sort of like asking if the speech center of a particular human's brain is "conscious" or has "agency": it's not really a question that makes sense, because the speech center of a brain is just one part of a densely interconnected system that we only interpret as a "mind" when considered in its totality.
- deleted 4y ago
  [deleted]
- pegasus 4y ago
  Good point, that very much vibes with my thoughts on this matter. Lately, I've been contemplating the analogy between the role LLMs might take within society with that of the brain's language center* in human behavior. There's definitely a way in which we resemble these models. More than some might like to admit. The cleverness, but also the hallucinating, gaslighting and other such behaviors. And on the other hand, any way you'd slice it, it seems to me LLMs - and software systems in general - necessarily lack intrinsic motivation. By definition, any goal it has can only be the goal of whoever designed that system. Even if its maker decides - "let it pick goals randomly", those randomly picked goals are just intermediate steps toward the enacting of the programmer's original goal. Robert Miles' YouTube videos on alignment shed light on these issues also. For example: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo Another relevant source on these issues is the book "The Master and his Emissary", which discusses how basically the language center can, in some way - I'm simplifying a lot, fall prey to the illusion that "it" is the entirety of human consciousness. * or at least some subsystems of that language center, it's important to remember how little we still understand of human cognition
  - brycedriesenga 4y ago
    What goals do we have that aren't essentially all boiled down to whatever evolution, genetics, and our environment have sorted of molded into us?
iliane5 4y ago
What I find really entertaining is the "just predicting the next token" argument. If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.
- tines 4y ago
  > If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until the recent internet age, in which garbage survives as well as the gold)). > then maybe there's a bit of hubris in how smart we think we actually are I feel like you could say this if ChatGPT or whatever obtained its knowledge some other way than direct guidance from humans, but since we hand-fed it the answers, it falls a little flat for me. I'm open to persuasion.
  - autokad 4y ago
    chatgpt doesnt just feed us back answers we already taught it. It learned relationships and semantics so it can apply that knowledge to do something novel. For instance, I took the basic of a dream and told it to turn it into a short story. The short story wasn't bad. I said make it more exciting, it updated the story such that one of the cars exploded. I guess chatgpt learned excitement from michael bay.
    - tines 4y ago
      (I'm going to be brusque for the sake of the argument, I very much could be wrong and I don't even know how much I believe of the argument I'm making.) > chatgpt doesnt just feed us back answers we already taught it True, there is some structure to the answers we already taught it that it statistically mimics as well. > It learned relationships and semantics so it can apply that knowledge to do something novel Can you provide an example of this novelty? I think we underestimate the depth and variety of things that humans have written about and put on the internet, and so while anything you ask ChatGPT to do might be outside of your own experience, it's highly likely that it's already been thought before and uploaded to the internet, and that ChatGPT is just parrotting back something to you that is very similar to what it has already seen. This effect of ChatGPT having so much more experience/training data than any single human being such that it can convince any single human that it is original is an interesting one. This is why I think, for example, that image generation will result in (a period of) "artistic inbreeding." Because there is so much that other humans have done that is outside of any individual's experience, we will accept e.g. Midjourney's output as something moving and original, when in reality it's just a slight variation on something that someone else has done before that we haven't seen. (Again apologies for any rudeness, I respect your opinion and experiences and am enjoying the conversation.)
bnralt 4y ago
> I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument. I imagine many definitions are initially rather broad and only get refined down over time. Laertius gives us a classic example: > Plato defined man thus: “Man is a two-footed, featherless animal,” and was much praised for the definition; so Diogenes plucked a cock and brought it into his school, and said, “This is Plato’s man.” On which account this addition was made to the definition, “With broad flat nails.” I don’t think it’s correct to think of that as infinitely moving goalposts, however. More that the weakness of definitions isn’t always immediately transparent.
barrysteve 4y ago
I have tried multiple times to use Chatgpt to generate Unreal c++ code. It does not do. It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type. For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it. We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform. Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT. It must understand/have structure, or at least use a crutch to get it over the finish line..
- i-use-nixos-btw 4y ago
  Yes, in this instance I understand failings of today (though copilot has a much better hit rate, and at the moment it’s a great augmentation to coding if you treat it like an enthusiastic intern). My question is about the future. The argument goes that a machine can never understand Chinese, even if it is capable of interpreting Chinese and responding to or acting on the input perfectly every time. My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding. It’s kind of like the whole string theory vs SUSY vs… argument in physics. If the only outcomes are things that agree with the Standard Model in all measurable aspects, and don’t provide any measurable distinction, then for all intents and purposes they don’t matter. That’s why their active areas of research are looking for the measurable distinctions. FWIW, supersymmetry models predict measurable things (that so far have only ruled out those models when tested) but have applications elsewhere. String theory research has had implications in mathematics, condensed matter, and a bunch of other places. They’re useful. But that’s beside the point, because the premise of the Chinese room problem is that there exists a machine that passes all scenarios, where no measurable difference can be found, and that this machine does not understand Chinese.
  - missingrib 4y ago
    > My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding. I'm not sure if you understood the argument. The argument isn't asserting that there is a measurable way of distinguishing it, it's actually claiming that regardless of how well it seems like it understands Chinese, it doesn't actually understand Chinese. It's about intentionality and consciousness.
tim333 4y ago
I'm not sure many people take the Chinese Room argument seriously these days. I mean it basically proposes a room that is equivalent to a lookup table and says it can't understand much hence computers in general can't understand. The "hence..." bit makes no sense really.
kmod 4y ago
Here's an example that I think garners more agreement that properties of a limit ("really understanding") don't necessarily mean that any path towards that limit has the properties of the limit. I think there's a lot of room for disagreement about whether this is a factually-accurate analogy and I'm not trying to argue either way on that, just trying to answer your question about how one might make these sorts of arguments if one has a certain belief about the facts. Let's say we're trying to build a calculator that only needs to do integer addition. And we decide to build it by building a giant if-else chain that hardcodes the answer to each and every possible addition. And due to finite resources, we're going to hardcode all the additions of integers up to absolute value N, but we will increase N over time. Everything you said applies equally to this situation: it quacks like a duck, and when we talk about things it can't do we have to continually move the goalposts each time a new version comes out. It also has the property that there is a "scaling law" that says that each time you double N you get predictably better performance from the system, and you can do this without bound and continually approach a limit where it can answer any question indistinguishably from something we might call "true understanding". But I think it's a bit easier to agree that in this case that it's not "really doing" addition and is a bit short of our wish to have an artificial addition system. And if someone touts this system as the way to automate addition we might feel a bit irritated. Again, many people will say that this is a bad analogy because LLMs operate quite differently, and I'm not trying to argue for or against that. Just trying to give my explanation for how a certain understanding of the facts can imply the kind of conclusion that you are trying to understand.
- brotchie 4y ago
  re: Calculator examples, I kinda see them both as information systems that achieve a result but there's a phase difference in where the information is stored. Similar to how space is 4D such that with relativity going faster in a spatial dimension kind of "borrows" from the time dimension (in a hand wavy way). By analogy, you can have something that's purely a lookup table, or on the other hand, completely based on an algorithm, and the full lookup table is kind of "borrowing" from the algorithmic dimension of the information system space and vice-verse the fully algorithmic version is borrowing from the hardcoded dimension of the information system space. Under the condition that you're adding integers below N, then if you consider BOTH the (hardcoded, algorithmic) as a singular space (as with 4D space time) then they are equivalent. Need to work on this theory further to make it more understandable, but I think this way about intelligence. Intelligence sits as a pattern in the information system space that can range anywhere from hardcoded to algorithmic (if we choose to orthogonalize the space this way). But what actually matters is the system's future impact on it's local laws of physics, and for that purpose both implementations are equivalent. Edit: Conversation with GPT-4 about this https://sharegpt.com/c/Sbs4XgI https://sharegpt.com/c/Sbs4XgI
d0mine 4y ago
Here's GPT output that demonstrates that it doesn't reflect on its text: «GPT Chat: You are right, I apologize for the confusion in my previous responses. The corrected explanation is: The pattern is counting the number of closed spaces in each letter of the spelled-out number. A closed space is any enclosed space in a letter, such as in the letters "a", "b", "d", "e", etc. Following the pattern: - one -> 2 (there are closed spaces in the letters "n" and "e") - two -> 1 (there is a closed space in the letter "o") - three -> 2 (there are closed spaces in the letters "h" and "e") - four -> 1 (there is a closed space in the letter "o") - five -> 1 (there is a closed space in the letter "e") - six -> 0 (there are no closed spaces in the letters) - seven -> 2 (there are closed spaces in the letters "e" and "n") - eight -> 1 (there is a closed space in the letter "g") - nine -> 1 (there is a closed space in the letter "e") - ten -> 1 (there is a closed space in the letter "b") - eleven -> 3 (there are closed spaces in the letters "e", "l", and "v") - twelve -> 2 (there are closed spaces in the letters "b" and "d") - thirteen -> 2 (there are closed spaces in the letters "b" and "d") Each item follows the pattern, as the number of closed spaces in their letters matches the corresponding number in the pattern. The whole sequence is: one -> 2 two -> 1 three -> 2 four -> 1 five -> 1 six -> 0 seven -> 2 eight -> 1 nine -> 1 ten -> 1 eleven -> 3 twelve -> 2 thirteen -> 2 ...» It is clear the model doesn't know what it is talking about.
- int_19h 4y ago
  Anything having to do with shapes or spatial relationships seems to befuddle GPT-3.5. GPT-4 is significantly better but it's clear that it still struggles with it. As far as shapes of the letters in particular go, you can ask it to draw them using SVG or <canvas> and see what comes out. Here's a different example involving dataset analysis with GPT-4 that required it to analyze its own previous outputs to find and correct mistakes and form a new hypothesis: https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21dfd https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21...
dvt 4y ago
> It seems to me to posit that to understand requires that the understandee is human. Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding? Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sadness, depression, love, etc.), a sense of what philosophers like Elizabeth Anscombe call I-ness, etc. Most of this stuff, to me, seems like is language-agnostic. Even a baby that can't speak feels pain or happiness. Even a dog feels anxiety or affection. LLMs are a cute parlor trick, but a phantasm nonetheless.
- iliane5 4y ago
  There's no denying LLMs are anything but sentient however is sentience really needed for intelligence? I feel like if we can have machines that are X% smarter than a human could ever get for any given task, it'd be a much better outcome for us if they were not sentient.
- lordnacho 4y ago
  Does that mean sentient beings need to have had the evolutionary history that people have had? After all why would aliens have feelings like we have feelings? Don't we have feelings because either it was useful to our ancestors, or they are a spandrel of something that was?
  - pegasus 4y ago
    Maybe that's all there is to it. But maybe not. These questions are not settled. It might seem obvious to you, but it doesn't seem so to me. Check out some of my other comments in this thread if you'd like to understand where my reservations come from.
root_axis 4y ago
> if it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. This isn't true. If a plane flies like a bird and you only need it for flying it doesn't then follow that a plane is a bird "for all intents and purposes".
- i-use-nixos-btw 4y ago
  The emphasis on “for all intents and purposes” is important. If you specify requirements you specify them for your intent and for your purpose. Requiring that something fly and that something be a plane are two different things with only minor overlap. If all you require is something that flies, then a dragonfly matches your requirements exactly as much as an apache helicopter does.
eternalban 4y ago
> I cannot see that it matters if a computer understands something. I am not sure they can, but the difference is profound and material. A machine that actually understands, like a human being, is not going to be (can not be) entirely truthful or transparent. There will be private inner thoughts, idea formation, and possibly even willful intent, as a direct consequence of understanding. And the nature of interactions, regardless of superficial similarity, shifts from one of utility to relationship. For example, we would care to know if e.g. the systems entrusted with apocalyptic deterent forces are mechanisms or organisms. Please note that not a single one of us has ever interacted with any intelligent life form lacking a sense of self, or an ego. Thus, all our sensory registers of another 'intelligent being' are learned in a context of the implicit 'this other is like me'. We are not equipped to distinguish or articulate intelligence (in the abstract) merely based on sensory information. Note that even non-verbal communication, such as jabbing a friend in the ribs, are all learned to have a certain meaning in that very same context of implicits, and any mechanism that mimicks them (via training) will be afforded the same projection of the implicit. I do not believe there is, in fact, any non-destructive test of determining 'consciousness' in an entity. (Destructive, since there may be long running tests of a subject than can be shown to be probably accurate, possibly via creating situational problems involving survival, and unexpected circumstances.) Ask yourself what is it that convinces you that the last person you spoke with (in real life) was actually conscious? I assert that the entire matter is a 'fictional certainty' based on assumption of shared nature. "They are conscious because I am".
jacksnipe 4y ago
This is also my interpretation of the Chinese Room. Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html https://norvig.com/chomsky.html
user_named 4y ago
You're talking about intelligence - GPT doesn't quack at all. It clearly is not a duck.
lsy 4y ago
I don't know if I take the Chinese Room scenario seriously, it's a little hard to think about. For me the key is that an LLM (and maybe the Chinese room?) is only trained on text, and therefore its entire "universe" is merely representational. To the LLM there is no difference between the color blue and the word "blue", or a dog and the word "dog". People understand that a word is a provisional marker for something with almost infinite complexity, and when a word fails to meet the goals of the person, the word is discarded, amended, or elaborated until it expresses the thing which has heretofore been unexpressed. For an LLM there is no deeper world to access beyond text. I don't think an intelligence needs to be human, and it should be physically possible to create an intelligence which is synthetic. But in order to call the intelligence "general", and to rely on it for the purposes that designation implies, it would need to be able to successfully navigate the world, which requires access to that world and the use of the world as its own model, rather than the much simpler and coarser intermediary of text. In order to claim that an LLM can fully navigate the world after being trained on pure text, we would have to believe that all our writings across history have exhausted what there is to say about the world. This is not to say an LLM cannot be useful for some purposes, but there will be key ways in which they fail because they have no sense of meaning or what the world is like. Whether consciousness is required to solve this I don't know, but we simply haven't begun to approach a system that can meaningfully address the world as a world.

JieJie 4y ago

I literally lost a friend of thirty years yesterday because she is wedded to the Chinese Room analogy so fiercely, she refuses to engage on the subject at all. For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with. (edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)

bulkprotocol 4y ago
You blew up a 30 year friendship over an...analogy?
- JieJie 4y ago
  I didn't! Someone else did it to me. I was trying desperately not to. (edit: This is the kind of stuff I think my friends are watching and being informed by [0] as it was what they are posting in our common areas.) [0]: https://youtu.be/ro130m-f_yk https://youtu.be/ro130m-f_yk
  - roflyear 4y ago
    Why does it upset you if they disagree with you on this?
    - JieJie 4y ago
      It's not that they disagree with me, as there were many things we agreed on. I have read over it and it seems that they don't respect what I believe so much that their example of how they felt was this angry shouting man. But again, I was not the person who ended the friendship. In short, I'm upset that a person I thought was a friend disrespected my opinion about something so much, she stopped being my friend.
brazzy 4y ago
How can an disagreement over GPT possibly be something to end a friendship over? This sounds like you said "I cannot possibly be friends with someone who does not believe that LLMs are emerging AGI!", and people read it like that and are downvoting you. I'm gonna assume the situation was more complex, but still find it hard to imagine, how a disagreement over such an academic topic could end up destroying a friendship.
- JieJie 4y ago
  It was a complex situation to be sure. I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently. If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear. I can barely speak with my artist friends on the issue these days due to their generative AI fears. Their emotions are completely intractable on the subject: AI art is theft. Period.
  - brazzy 4y ago
    If are is a major part of someone's identity (strongly implied by calling them an "artist"), then it's hardly surprising that they would feel threatened and react emotionally to something that massively disrupts the creation of art and the viability of making a living from it.
    - JieJie 4y ago
      I'm old enough to have had my design and publishing career evaporate underneath my feet because newsprint has mostly been replaced by the internet. I'm even old enough that I was part of replacing other people's careers with technology during the desktop publishing revolution. I remember when graphics tablets became a thing and how traditional pen and brush artists cried, "unfair!" I always felt empathy for them, even back then. I feel empathy for my artist friends as well. How can we ever have a better future if we keep trying to prevent change from disrupting our careers? In my experience, I've seen a lot of wailing and gnashing of teeth, but these changes have always come no matter how hard people fought. On the other hand, I see people who try to adapt to these changes do quite well. It's the part where I watch them hide their head in the sand that bothers me, not their anger. I want them to succeed. I think they have great eyes for color and line and those things are still super-valid long after the generative AI has replaced the pen and brush. I don't want to watch my artist friends suffer. I am old enough to know that the effort to stop generative AI is almost certainly going to fail (just as all the previous efforts to prevent change failed), and it is better to light a candle than to curse the darkness. The only constant is change.

rwmj 4y ago

Isn't the whole point of Searle raising the Chinese Rooom argument exactly so that these kinds of things can be debated?

mhink 4y ago

In my understanding of the Chinese Room example, the resolution to the argument is that the *human* may not understand Chinese, but the *system as a whole* can be said to understand it. With this in mind, I think asking whether ChatGPT *in and of itself* is "conscious" or has "agency" is sort of like asking if the speech center of a particular human's brain is "conscious" or has "agency": it's not really a question that makes sense, because the speech center of a brain is just one part of a densely interconnected system that we only interpret as a "mind" when considered in its totality.

deleted 4y ago
[deleted]
pegasus 4y ago
Good point, that very much vibes with my thoughts on this matter. Lately, I've been contemplating the analogy between the role LLMs might take within society with that of the brain's language center* in human behavior. There's definitely a way in which we resemble these models. More than some might like to admit. The cleverness, but also the hallucinating, gaslighting and other such behaviors. And on the other hand, any way you'd slice it, it seems to me LLMs - and software systems in general - necessarily lack intrinsic motivation. By definition, any goal it has can only be the goal of whoever designed that system. Even if its maker decides - "let it pick goals randomly", those randomly picked goals are just intermediate steps toward the enacting of the programmer's original goal. Robert Miles' YouTube videos on alignment shed light on these issues also. For example: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo Another relevant source on these issues is the book "The Master and his Emissary", which discusses how basically the language center can, in some way - I'm simplifying a lot, fall prey to the illusion that "it" is the entirety of human consciousness. * or at least some subsystems of that language center, it's important to remember how little we still understand of human cognition
- brycedriesenga 4y ago
  What goals do we have that aren't essentially all boiled down to whatever evolution, genetics, and our environment have sorted of molded into us?

iliane5 4y ago

What I find really entertaining is the "just predicting the next token" argument. If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

tines 4y ago
> If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until the recent internet age, in which garbage survives as well as the gold)). > then maybe there's a bit of hubris in how smart we think we actually are I feel like you could say this if ChatGPT or whatever obtained its knowledge some other way than direct guidance from humans, but since we hand-fed it the answers, it falls a little flat for me. I'm open to persuasion.
- autokad 4y ago
  chatgpt doesnt just feed us back answers we already taught it. It learned relationships and semantics so it can apply that knowledge to do something novel. For instance, I took the basic of a dream and told it to turn it into a short story. The short story wasn't bad. I said make it more exciting, it updated the story such that one of the cars exploded. I guess chatgpt learned excitement from michael bay.
  - tines 4y ago
    (I'm going to be brusque for the sake of the argument, I very much could be wrong and I don't even know how much I believe of the argument I'm making.) > chatgpt doesnt just feed us back answers we already taught it True, there is some structure to the answers we already taught it that it statistically mimics as well. > It learned relationships and semantics so it can apply that knowledge to do something novel Can you provide an example of this novelty? I think we underestimate the depth and variety of things that humans have written about and put on the internet, and so while anything you ask ChatGPT to do might be outside of your own experience, it's highly likely that it's already been thought before and uploaded to the internet, and that ChatGPT is just parrotting back something to you that is very similar to what it has already seen. This effect of ChatGPT having so much more experience/training data than any single human being such that it can convince any single human that it is original is an interesting one. This is why I think, for example, that image generation will result in (a period of) "artistic inbreeding." Because there is so much that other humans have done that is outside of any individual's experience, we will accept e.g. Midjourney's output as something moving and original, when in reality it's just a slight variation on something that someone else has done before that we haven't seen. (Again apologies for any rudeness, I respect your opinion and experiences and am enjoying the conversation.)

bnralt 4y ago

> I’m no philosopher, but if your argument hinges on moving goalposts until purity is reached, and your basic assumption is that the requirements for purity are infinite, then it’s not a very useful argument. I imagine many definitions are initially rather broad and only get refined down over time. Laertius gives us a classic example: > Plato defined man thus: “Man is a two-footed, featherless animal,” and was much praised for the definition; so Diogenes plucked a cock and brought it into his school, and said, “This is Plato’s man.” On which account this addition was made to the definition, “With broad flat nails.” I don’t think it’s correct to think of that as infinitely moving goalposts, however. More that the weakness of definitions isn’t always immediately transparent.

barrysteve 4y ago

I have tried multiple times to use Chatgpt to generate Unreal c++ code. It does not do. It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type. For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it. We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform. Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT. It must understand/have structure, or at least use a crutch to get it over the finish line..

i-use-nixos-btw 4y ago
Yes, in this instance I understand failings of today (though copilot has a much better hit rate, and at the moment it’s a great augmentation to coding if you treat it like an enthusiastic intern). My question is about the future. The argument goes that a machine can never understand Chinese, even if it is capable of interpreting Chinese and responding to or acting on the input perfectly every time. My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding. It’s kind of like the whole string theory vs SUSY vs… argument in physics. If the only outcomes are things that agree with the Standard Model in all measurable aspects, and don’t provide any measurable distinction, then for all intents and purposes they don’t matter. That’s why their active areas of research are looking for the measurable distinctions. FWIW, supersymmetry models predict measurable things (that so far have only ruled out those models when tested) but have applications elsewhere. String theory research has had implications in mathematics, condensed matter, and a bunch of other places. They’re useful. But that’s beside the point, because the premise of the Chinese room problem is that there exists a machine that passes all scenarios, where no measurable difference can be found, and that this machine does not understand Chinese.
- missingrib 4y ago
  > My reply is that, if it acts as if it understands Chinese in every situation, then there’s no measurable way of distinguishing it from understanding. I'm not sure if you understood the argument. The argument isn't asserting that there is a measurable way of distinguishing it, it's actually claiming that regardless of how well it seems like it understands Chinese, it doesn't actually understand Chinese. It's about intentionality and consciousness.

tim333 4y ago

I'm not sure many people take the Chinese Room argument seriously these days. I mean it basically proposes a room that is equivalent to a lookup table and says it can't understand much hence computers in general can't understand. The "hence..." bit makes no sense really.

kmod 4y ago

Here's an example that I think garners more agreement that properties of a limit ("really understanding") don't necessarily mean that any path towards that limit has the properties of the limit. I think there's a lot of room for disagreement about whether this is a factually-accurate analogy and I'm not trying to argue either way on that, just trying to answer your question about how one might make these sorts of arguments if one has a certain belief about the facts. Let's say we're trying to build a calculator that only needs to do integer addition. And we decide to build it by building a giant if-else chain that hardcodes the answer to each and every possible addition. And due to finite resources, we're going to hardcode all the additions of integers up to absolute value N, but we will increase N over time. Everything you said applies equally to this situation: it quacks like a duck, and when we talk about things it can't do we have to continually move the goalposts each time a new version comes out. It also has the property that there is a "scaling law" that says that each time you double N you get predictably better performance from the system, and you can do this without bound and continually approach a limit where it can answer any question indistinguishably from something we might call "true understanding". But I think it's a bit easier to agree that in this case that it's not "really doing" addition and is a bit short of our wish to have an artificial addition system. And if someone touts this system as the way to automate addition we might feel a bit irritated. Again, many people will say that this is a bad analogy because LLMs operate quite differently, and I'm not trying to argue for or against that. Just trying to give my explanation for how a certain understanding of the facts can imply the kind of conclusion that you are trying to understand.

brotchie 4y ago
re: Calculator examples, I kinda see them both as information systems that achieve a result but there's a phase difference in where the information is stored. Similar to how space is 4D such that with relativity going faster in a spatial dimension kind of "borrows" from the time dimension (in a hand wavy way). By analogy, you can have something that's purely a lookup table, or on the other hand, completely based on an algorithm, and the full lookup table is kind of "borrowing" from the algorithmic dimension of the information system space and vice-verse the fully algorithmic version is borrowing from the hardcoded dimension of the information system space. Under the condition that you're adding integers below N, then if you consider BOTH the (hardcoded, algorithmic) as a singular space (as with 4D space time) then they are equivalent. Need to work on this theory further to make it more understandable, but I think this way about intelligence. Intelligence sits as a pattern in the information system space that can range anywhere from hardcoded to algorithmic (if we choose to orthogonalize the space this way). But what actually matters is the system's future impact on it's local laws of physics, and for that purpose both implementations are equivalent. Edit: Conversation with GPT-4 about this https://sharegpt.com/c/Sbs4XgI https://sharegpt.com/c/Sbs4XgI

d0mine 4y ago

Here's GPT output that demonstrates that it doesn't reflect on its text: «GPT Chat: You are right, I apologize for the confusion in my previous responses. The corrected explanation is: The pattern is counting the number of closed spaces in each letter of the spelled-out number. A closed space is any enclosed space in a letter, such as in the letters "a", "b", "d", "e", etc. Following the pattern: - one -> 2 (there are closed spaces in the letters "n" and "e") - two -> 1 (there is a closed space in the letter "o") - three -> 2 (there are closed spaces in the letters "h" and "e") - four -> 1 (there is a closed space in the letter "o") - five -> 1 (there is a closed space in the letter "e") - six -> 0 (there are no closed spaces in the letters) - seven -> 2 (there are closed spaces in the letters "e" and "n") - eight -> 1 (there is a closed space in the letter "g") - nine -> 1 (there is a closed space in the letter "e") - ten -> 1 (there is a closed space in the letter "b") - eleven -> 3 (there are closed spaces in the letters "e", "l", and "v") - twelve -> 2 (there are closed spaces in the letters "b" and "d") - thirteen -> 2 (there are closed spaces in the letters "b" and "d") Each item follows the pattern, as the number of closed spaces in their letters matches the corresponding number in the pattern. The whole sequence is: one -> 2 two -> 1 three -> 2 four -> 1 five -> 1 six -> 0 seven -> 2 eight -> 1 nine -> 1 ten -> 1 eleven -> 3 twelve -> 2 thirteen -> 2 ...» It is clear the model doesn't know what it is talking about.

int_19h 4y ago
Anything having to do with shapes or spatial relationships seems to befuddle GPT-3.5. GPT-4 is significantly better but it's clear that it still struggles with it. As far as shapes of the letters in particular go, you can ask it to draw them using SVG or <canvas> and see what comes out. Here's a different example involving dataset analysis with GPT-4 that required it to analyze its own previous outputs to find and correct mistakes and form a new hypothesis: https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21dfd https://gist.github.com/int19h/cd1d1598f91e8ba92dd8e80bd5d21...

dvt 4y ago

> It seems to me to posit that to understand requires that the understandee is human. Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding? Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sadness, depression, love, etc.), a sense of what philosophers like Elizabeth Anscombe call I-ness, etc. Most of this stuff, to me, seems like is language-agnostic. Even a baby that can't speak feels pain or happiness. Even a dog feels anxiety or affection. LLMs are a cute parlor trick, but a phantasm nonetheless.

iliane5 4y ago
There's no denying LLMs are anything but sentient however is sentience really needed for intelligence? I feel like if we can have machines that are X% smarter than a human could ever get for any given task, it'd be a much better outcome for us if they were not sentient.
lordnacho 4y ago
Does that mean sentient beings need to have had the evolutionary history that people have had? After all why would aliens have feelings like we have feelings? Don't we have feelings because either it was useful to our ancestors, or they are a spandrel of something that was?
- pegasus 4y ago
  Maybe that's all there is to it. But maybe not. These questions are not settled. It might seem obvious to you, but it doesn't seem so to me. Check out some of my other comments in this thread if you'd like to understand where my reservations come from.

root_axis 4y ago

> if it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. This isn't true. If a plane flies like a bird and you only need it for flying it doesn't then follow that a plane is a bird "for all intents and purposes".

i-use-nixos-btw 4y ago
The emphasis on “for all intents and purposes” is important. If you specify requirements you specify them for your intent and for your purpose. Requiring that something fly and that something be a plane are two different things with only minor overlap. If all you require is something that flies, then a dragonfly matches your requirements exactly as much as an apache helicopter does.

eternalban 4y ago

> I cannot see that it matters if a computer understands something. I am not sure they can, but the difference is profound and material. A machine that actually understands, like a human being, is not going to be (can not be) entirely truthful or transparent. There will be private inner thoughts, idea formation, and possibly even willful intent, as a direct consequence of understanding. And the nature of interactions, regardless of superficial similarity, shifts from one of utility to relationship. For example, we would care to know if e.g. the systems entrusted with apocalyptic deterent forces are mechanisms or organisms. Please note that not a single one of us has ever interacted with any intelligent life form lacking a sense of self, or an ego. Thus, all our sensory registers of another 'intelligent being' are learned in a context of the implicit 'this other is like me'. We are not equipped to distinguish or articulate intelligence (in the abstract) merely based on sensory information. Note that even non-verbal communication, such as jabbing a friend in the ribs, are all learned to have a certain meaning in that very same context of implicits, and any mechanism that mimicks them (via training) will be afforded the same projection of the implicit. I do not believe there is, in fact, any non-destructive test of determining 'consciousness' in an entity. (Destructive, since there may be long running tests of a subject than can be shown to be probably accurate, possibly via creating situational problems involving survival, and unexpected circumstances.) Ask yourself what is it that convinces you that the last person you spoke with (in real life) was actually conscious? I assert that the entire matter is a 'fictional certainty' based on assumption of shared nature. "They are conscious because I am".

jacksnipe 4y ago

This is also my interpretation of the Chinese Room. Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html https://norvig.com/chomsky.html

user_named 4y ago

You're talking about intelligence - GPT doesn't quack at all. It clearly is not a duck.

lsy 4y ago

I don't know if I take the Chinese Room scenario seriously, it's a little hard to think about. For me the key is that an LLM (and maybe the Chinese room?) is only trained on text, and therefore its entire "universe" is merely representational. To the LLM there is no difference between the color blue and the word "blue", or a dog and the word "dog". People understand that a word is a provisional marker for something with almost infinite complexity, and when a word fails to meet the goals of the person, the word is discarded, amended, or elaborated until it expresses the thing which has heretofore been unexpressed. For an LLM there is no deeper world to access beyond text. I don't think an intelligence needs to be human, and it should be physically possible to create an intelligence which is synthetic. But in order to call the intelligence "general", and to rely on it for the purposes that designation implies, it would need to be able to successfully navigate the world, which requires access to that world and the use of the world as its own model, rather than the much simpler and coarser intermediary of text. In order to claim that an LLM can fully navigate the world after being trained on pure text, we would have to believe that all our writings across history have exhausted what there is to say about the world. This is not to say an LLM cannot be useful for some purposes, but there will be key ways in which they fail because they have no sense of meaning or what the world is like. Whether consciousness is required to solve this I don't know, but we simply haven't begun to approach a system that can meaningfully address the world as a world.