7 ms·

The Thoughts the Civilized Keep

36 points by arunbahl 5y ago

ctoth 5y ago
> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever. Ah, right, it won't work because it doesn't have a soul. Got it.
- deleted 5y ago
  [deleted]
- jmoss20 5y ago
  I don't agree with the article, but hold on -- you don't need to buy "intelligence -> soul" for this. There are plenty of things that can't (or can't efficiently) be represented in a single agent. This is (for example) Hayek's main critique of central planning: "[price] understanding cannot occur in an isolated computation or behavior"; there are simply too many necessary details to fit into any one head/computer, and therefore we should prefer markets. (Markets being definitionally not located in any one isolated computation/behavior, but inter-agentially in lots of them.)
  - jimmaswell 5y ago
    I don't see any fundamental reason a human brain couldn't be simulated at the atomic level by a turing machine, so it follows intelligence should be able to come from a machine, as far as it can come from a human.
    - majormajor 5y ago
      Simulating that brain will require you to provide some sort of inputs and do something with the outputs. Which gets you into questions of "what does this simulated brain want" once again. How much of human desire and behavior comes from the body? I want to avoid pain (signals that my body sends to my brain). I want to experience pleasure (also signals that my body sends to my brain). If you simulate a body with the same inputs as the existing brain, do you just have a "faster horse" problem, where you have a brain that could possibly perceive/process information faster (or slower) than a real human depending on the speed of the simulation? That could be useful, but sounds potentially rather bleak, especially if we limit what it can do to change its environment. It also seems limited for e.g. central planning - it can do planning faster than any individual, but likely will still have working memory size limitations, if it's just a simulation of a human brain. So it may not be so much better than parallelizing tasks across a bunch of humans. More interesting, probably, then, than the "super human" style, is giving a new sort of intelligence different and additional senses, so that it doesn't want or need the same sort of physical embodiment we have, a la the "set sets" in the Terra Ignota series. But then we have to figure out what separates abstract intelligence from our own senses, and how to usefully augment it with new senses for a data-based world...
      - lolwut987 5y ago
        Yes, I often have to consider the context of the excel spreadsheets on my VMs, because you can’t know how to run a machine without knowing what it wants at a high level.
        jacobr1 5y ago
        Given this is HN, I honestly can't tell if you are being genuine or snarky.
- scandox 5y ago
  I don't see any evidence of dualism in the article. I think it makes sense that as people try to make intelligent machines we will find out more and more about what that means. The goalposts shift themselves until perhaps one day you find yourself there. But at every point you should expect it to turn out to be more complicated than you thought. Surely. As for humans losing the ability to understand - that does sound a little bit panicky.
- jerf 5y ago
  No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well. And I don't just mean this from the old point of view of "if we understand it, it's not AI anymore", I mean, the model has serious fundamental limitations. One of them is that it's static, so, once trained, it doesn't learn from its future input. Another is that when it is doing its "what is the highest probability text output to follow this text" (which, itself, is not exactly the sort of question "Strong AI" is the answer to), it by construction can only use a fixed-size window of text. So, for instance, it is by construction simply impossible for GPT-3 to write a novel, or even a short story, because the only way to get it to keep outputting is to provide it a window of the last X KB of output (I don't recall exactly how big it is) for the next continuation, which means that as it is writing the X+5th KB of output, it has literally completely forgotten the first 5KB of output, as completely as if it never existed. On the flip side, it is clearly very interesting and there is something there. However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).
  - selfhoster11 5y ago
    > However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet). I think that terms like "philosophical zombie" may be close to the truth, but I'm guessing some sci-fi work has a term that we could adopt.
  - TeMPOraL 5y ago
    > there is something there There is indeed. I'll recount my own words after my first proper exposure to the model, in the form of playing custom stories in AI Dungeon: "I played more of it today, and I can confirm that prolonged exposure makes me feel like I'm half awake, and half in a dream - particularly when the real world interrupts, I feel like I didn't fully wake up. What's worse, whenever I tried to talk to a real person today, I've felt like I'm just feeding words to an AI, and anticipated "story" to evolve. Never before has a game messed up with my mind so badly. I'm not sure if it's safe." To which I got a reply perfectly summarizing the phenomenon: "AI Dungeon feels a bit like a fever-dream." -- It's been a year since; thinking back, I feel the trick here is, GPT-3 crosses the threshold of being able to produce plausible and interesting text - where by "interesting" I mean "almost, but not quite, unpredictable". The way other people and their stories are interesting - their words aren't totally random, you have this feeling for the possible trajectories the conversation can take, but you can't quite guess what they're going to say next. GPT-3 does that well enough that it's easy to suspend disbelief and feel as if you're dealing with a mind. But then the feelings that arise in you, that's just you projecting - engaging your mental machinery for dealing with other minds.
- Applejinx 5y ago
  It doesn't come with PURPOSE unless you force it to have some. Which is a perfectly fine avenue to take, I think. But without purpose, what you get is hallucinatory mockeries of meaning: trippy, but if it was consciousness you'd be looking for severe organic damage. Purpose doesn't automatically mean civilized, but you can't get to civilized without purpose. It's an okay word to use for outlining what's missing. AI output is increasingly coherent but MEANINGLESS. If you want to look to meaning and purpose, a better place to look might be emergent behaviors of the YouTube Algorithm, or Facebook feed. Those can easily take toxic swings because they're operating in the absence of a higher sense of what will happen: they're the consciousness of a worm or insect. But they show purpose, where the more coherent and intelligence-like outputs of GPT-3 keep on betraying a lack of meaning. GPT-3 doesn't WANT anything. There's the problem (though at this stage I think it's just as well). Only that will bring real AI… which can be a daunting prospect.
  - sbierwagen 5y ago
    You're proposing that the... facebook feed is "more conscious" than GPT-3?
    - thatjoeoverthr 5y ago
      Seems like an interesting thought. The feeds have a purpose or agenda, actively influence real people in real life, and do so in a way that perpetuates their own existence. As long as they’re effective, they have real world institutions to support them. As it happens, they are perpetually learning and growing. Putting that next to GPT3, it’s like an intelligent rescue dog next to a horse that counts.
    - blamestross 5y ago
      The facebook feed is at least an online system vs a static model, so YES.
    - TeMPOraL 5y ago
      Probably, yes. It's interesting how consciousness could be a matter of drawing boundaries. A well-known example are the ants: individual ants are somewhat dumb, barely above simple chemical seekers. But the ant colony behaves as a single organism, and quite smart at that. Other examples include cities, nations, corporations - it kind of feels like these organizations have minds of their own, with individual humans being cells, and the bureaucracy forming a computational substrate. In the same vein, Facebook feed could be seen as a mind of its own. So could the entire human civilization.
- wnoise 5y ago
  No, the argument is that it won't work because it doesn't have a _body_.
drewcoo 5y ago
Why does this read to me like an overview of a D&D module where our intrepid PCs fight their way to the emotionless lich king at the heart of the keep? Is it the language use? The priming effect of the image?
bob1029 5y ago
> Civilization advances by extending the number of important operations which we can perform without thinking about them. I can't help but read this as "automation". I think there is an interesting philosophical overlap between automation and intelligence. Perhaps some virtuous cycle exists between these things. Like, consciousness is probably only feasible because so much of what happens in our biology occurs without actual intent. If you had to actively process the dilation of your blood vessels or digestion of food, there wouldn't be much room left over for other higher functions. You see this in tech too. For instance, build automation makes it feasible to then go and do certain things that otherwise would have seemed too complex or costly if someone was baking software by hand.
- joshmarlow 5y ago
  > I think there is an interesting philosophical overlap between automation and intelligence. On the flip side - there's an interesting hypothesis that the purpose of consciousness is to mediate conflicting non-conscious impulses. An example would be falling in the water - you have to mediate the impulse to breath with other conflicting impulses (such as the knowledge that you will drown). So I think consciousness/intelligence helps build skills and push them into the subconscious (thus 'automating' them) and then consciousness has to step in again for edge cases the automation can't (yet) handle. There's some fun discussion on the mediation hypothesis (with some Dune tie-ins) here: https://www.rifters.com/crawl/?p=791 https://www.rifters.com/crawl/?p=791
stupidcar 5y ago
> For AI researchers to move past the behaviorist conflation of thought and action, the field needs to drink again from the philosophical waters that fed much AI research in the late 20th century, when the field was theoretically rich, albeit technically floundering. Well, gosh. Fancy a professor of philosophy deciding that AI needs more philosophy. Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying.
- p_j_w 5y ago
  >Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying. What makes you think the author is saying anything like this?
phkahler 5y ago
>> Yet the connections GPT-3 makes are not illusory or concocted from thin air. It and many other machine learning models for natural language processing and generating do, in fact, track and reproduce real features of the symbolic order in which humans express thought. And yet, they do so without needing to have any thoughts to express. Unfortunately there are plenty of people who seem to do that as well. >> But the purpose of thought — what thought is good for — is a question widely neglected today, or else taken to have trivial, self-evident answers. Yet the answers are neither unimportant, nor obvious. I thought the entire purpose of having a brain is to A) predict the future and B) plan for it ahead of time by C) making predictions about outcomes based on our various possible actions. The entire purpose of the brain is to predict and control. That's why projection is so important.
djokkataja 5y ago
> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever. Understanding is not an act but a labor. Labor is entirely irrelevant to a computational model that has no history or trajectory in the world. For more along these lines: https://en.wikipedia.org/wiki/Situated_cognition https://en.wikipedia.org/wiki/Situated_cognition

36 points by arunbahl 5y ago

> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever. Ah, right, it won't work because it doesn't have a soul. Got it.

deleted 5y ago
[deleted]
jmoss20 5y ago
I don't agree with the article, but hold on -- you don't need to buy "intelligence -> soul" for this. There are plenty of things that can't (or can't efficiently) be represented in a single agent. This is (for example) Hayek's main critique of central planning: "[price] understanding cannot occur in an isolated computation or behavior"; there are simply too many necessary details to fit into any one head/computer, and therefore we should prefer markets. (Markets being definitionally not located in any one isolated computation/behavior, but inter-agentially in lots of them.)
- jimmaswell 5y ago
  I don't see any fundamental reason a human brain couldn't be simulated at the atomic level by a turing machine, so it follows intelligence should be able to come from a machine, as far as it can come from a human.
  - majormajor 5y ago
    Simulating that brain will require you to provide some sort of inputs and do something with the outputs. Which gets you into questions of "what does this simulated brain want" once again. How much of human desire and behavior comes from the body? I want to avoid pain (signals that my body sends to my brain). I want to experience pleasure (also signals that my body sends to my brain). If you simulate a body with the same inputs as the existing brain, do you just have a "faster horse" problem, where you have a brain that could possibly perceive/process information faster (or slower) than a real human depending on the speed of the simulation? That could be useful, but sounds potentially rather bleak, especially if we limit what it can do to change its environment. It also seems limited for e.g. central planning - it can do planning faster than any individual, but likely will still have working memory size limitations, if it's just a simulation of a human brain. So it may not be so much better than parallelizing tasks across a bunch of humans. More interesting, probably, then, than the "super human" style, is giving a new sort of intelligence different and additional senses, so that it doesn't want or need the same sort of physical embodiment we have, a la the "set sets" in the Terra Ignota series. But then we have to figure out what separates abstract intelligence from our own senses, and how to usefully augment it with new senses for a data-based world...
    - lolwut987 5y ago
      Yes, I often have to consider the context of the excel spreadsheets on my VMs, because you can’t know how to run a machine without knowing what it wants at a high level.
      - jacobr1 5y ago
        Given this is HN, I honestly can't tell if you are being genuine or snarky.
scandox 5y ago
I don't see any evidence of dualism in the article. I think it makes sense that as people try to make intelligent machines we will find out more and more about what that means. The goalposts shift themselves until perhaps one day you find yourself there. But at every point you should expect it to turn out to be more complicated than you thought. Surely. As for humans losing the ability to understand - that does sound a little bit panicky.
jerf 5y ago
No, if you study how GPT-3 works, it becomes pretty clear that that model can't really produce "understanding", even if we can't clearly define what that is very well. And I don't just mean this from the old point of view of "if we understand it, it's not AI anymore", I mean, the model has serious fundamental limitations. One of them is that it's static, so, once trained, it doesn't learn from its future input. Another is that when it is doing its "what is the highest probability text output to follow this text" (which, itself, is not exactly the sort of question "Strong AI" is the answer to), it by construction can only use a fixed-size window of text. So, for instance, it is by construction simply impossible for GPT-3 to write a novel, or even a short story, because the only way to get it to keep outputting is to provide it a window of the last X KB of output (I don't recall exactly how big it is) for the next continuation, which means that as it is writing the X+5th KB of output, it has literally completely forgotten the first 5KB of output, as completely as if it never existed. On the flip side, it is clearly very interesting and there is something there. However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet).
- selfhoster11 5y ago
  > However, I'd submit that what that something is is a something that we don't have an English word for, because up to this point humanity has never encountered anything with a cognitive model like GPT-3. So, it doesn't have "understanding", it has a quality-that-has-no-name (not to be confused with the "quality without a name" which GPT-3 most certainly does not have yet). I think that terms like "philosophical zombie" may be close to the truth, but I'm guessing some sci-fi work has a term that we could adopt.
- TeMPOraL 5y ago
  > there is something there There is indeed. I'll recount my own words after my first proper exposure to the model, in the form of playing custom stories in AI Dungeon: "I played more of it today, and I can confirm that prolonged exposure makes me feel like I'm half awake, and half in a dream - particularly when the real world interrupts, I feel like I didn't fully wake up. What's worse, whenever I tried to talk to a real person today, I've felt like I'm just feeding words to an AI, and anticipated "story" to evolve. Never before has a game messed up with my mind so badly. I'm not sure if it's safe." To which I got a reply perfectly summarizing the phenomenon: "AI Dungeon feels a bit like a fever-dream." -- It's been a year since; thinking back, I feel the trick here is, GPT-3 crosses the threshold of being able to produce plausible and interesting text - where by "interesting" I mean "almost, but not quite, unpredictable". The way other people and their stories are interesting - their words aren't totally random, you have this feeling for the possible trajectories the conversation can take, but you can't quite guess what they're going to say next. GPT-3 does that well enough that it's easy to suspend disbelief and feel as if you're dealing with a mind. But then the feelings that arise in you, that's just you projecting - engaging your mental machinery for dealing with other minds.
Applejinx 5y ago
It doesn't come with PURPOSE unless you force it to have some. Which is a perfectly fine avenue to take, I think. But without purpose, what you get is hallucinatory mockeries of meaning: trippy, but if it was consciousness you'd be looking for severe organic damage. Purpose doesn't automatically mean civilized, but you can't get to civilized without purpose. It's an okay word to use for outlining what's missing. AI output is increasingly coherent but MEANINGLESS. If you want to look to meaning and purpose, a better place to look might be emergent behaviors of the YouTube Algorithm, or Facebook feed. Those can easily take toxic swings because they're operating in the absence of a higher sense of what will happen: they're the consciousness of a worm or insect. But they show purpose, where the more coherent and intelligence-like outputs of GPT-3 keep on betraying a lack of meaning. GPT-3 doesn't WANT anything. There's the problem (though at this stage I think it's just as well). Only that will bring real AI… which can be a daunting prospect.
- sbierwagen 5y ago
  You're proposing that the... facebook feed is "more conscious" than GPT-3?
  - thatjoeoverthr 5y ago
    Seems like an interesting thought. The feeds have a purpose or agenda, actively influence real people in real life, and do so in a way that perpetuates their own existence. As long as they’re effective, they have real world institutions to support them. As it happens, they are perpetually learning and growing. Putting that next to GPT3, it’s like an intelligent rescue dog next to a horse that counts.
  - blamestross 5y ago
    The facebook feed is at least an online system vs a static model, so YES.
  - TeMPOraL 5y ago
    Probably, yes. It's interesting how consciousness could be a matter of drawing boundaries. A well-known example are the ants: individual ants are somewhat dumb, barely above simple chemical seekers. But the ant colony behaves as a single organism, and quite smart at that. Other examples include cities, nations, corporations - it kind of feels like these organizations have minds of their own, with individual humans being cells, and the bureaucracy forming a computational substrate. In the same vein, Facebook feed could be seen as a mind of its own. So could the entire human civilization.
wnoise 5y ago
No, the argument is that it won't work because it doesn't have a _body_.

drewcoo 5y ago

Why does this read to me like an overview of a D&D module where our intrepid PCs fight their way to the emotionless lich king at the heart of the keep? Is it the language use? The priming effect of the image?

bob1029 5y ago

> Civilization advances by extending the number of important operations which we can perform without thinking about them. I can't help but read this as "automation". I think there is an interesting philosophical overlap between automation and intelligence. Perhaps some virtuous cycle exists between these things. Like, consciousness is probably only feasible because so much of what happens in our biology occurs without actual intent. If you had to actively process the dilation of your blood vessels or digestion of food, there wouldn't be much room left over for other higher functions. You see this in tech too. For instance, build automation makes it feasible to then go and do certain things that otherwise would have seemed too complex or costly if someone was baking software by hand.

joshmarlow 5y ago
> I think there is an interesting philosophical overlap between automation and intelligence. On the flip side - there's an interesting hypothesis that the purpose of consciousness is to mediate conflicting non-conscious impulses. An example would be falling in the water - you have to mediate the impulse to breath with other conflicting impulses (such as the knowledge that you will drown). So I think consciousness/intelligence helps build skills and push them into the subconscious (thus 'automating' them) and then consciousness has to step in again for edge cases the automation can't (yet) handle. There's some fun discussion on the mediation hypothesis (with some Dune tie-ins) here: https://www.rifters.com/crawl/?p=791 https://www.rifters.com/crawl/?p=791

stupidcar 5y ago

> For AI researchers to move past the behaviorist conflation of thought and action, the field needs to drink again from the philosophical waters that fed much AI research in the late 20th century, when the field was theoretically rich, albeit technically floundering. Well, gosh. Fancy a professor of philosophy deciding that AI needs more philosophy. Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying.

p_j_w 5y ago
>Clearly, if the price of making progress in the field is to engage in all this ugly mathematics and -eww- engineering, then it's no price worth paying. What makes you think the author is saying anything like this?

phkahler 5y ago

>> Yet the connections GPT-3 makes are not illusory or concocted from thin air. It and many other machine learning models for natural language processing and generating do, in fact, track and reproduce real features of the symbolic order in which humans express thought. And yet, they do so without needing to have any thoughts to express. Unfortunately there are plenty of people who seem to do that as well. >> But the purpose of thought — what thought is good for — is a question widely neglected today, or else taken to have trivial, self-evident answers. Yet the answers are neither unimportant, nor obvious. I thought the entire purpose of having a brain is to A) predict the future and B) plan for it ahead of time by C) making predictions about outcomes based on our various possible actions. The entire purpose of the brain is to predict and control. That's why projection is so important.

djokkataja 5y ago

> Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated computation or behavior, no matter how clever. Understanding is not an act but a labor. Labor is entirely irrelevant to a computational model that has no history or trajectory in the world. For more along these lines: https://en.wikipedia.org/wiki/Situated_cognition https://en.wikipedia.org/wiki/Situated_cognition