8 ms·

The continued relevance of Weizenbaum’s warnings

131 points by pabs3 4y ago

Radim 4y ago
Article so wordy it could use an AI summary, but this quote stands out: > Writing of the enthusiastic embrace of a fully computerized world, Weizenbaum grumbled, “These people see the technical apparatus underlying Orwell’s 1984 and, like children on seeing the beach, they run for it” > a point to which Weizenbaum added “I wish it were their private excursion, but they demand that we all come along.”
labrador 4y ago
The first time I sensed I was dealing with an alien intelligence was playing a chess machine in 1980. I remember being perplexed that it beat me and aware the machine embodied intelligence in some way that I didn't understand It's not really clear to me if this article is directed at scientists and engineers getting carried away or the general public. If the former, yes they should know better, but the general public is mostly not rational, contrary to what I used to think about the reasoning person. Many people attribute thoughts and feeling to their pets that they don't have. They think crystals have special powers. It's only natural that they will think AI have consciousness. I think as scientists and engineers we have a responsibility to prevent our AI creations from telling people to drink the poisoned Kool Aid or eat glass. Yann LeCun said they shut down Meta's chat AI because it told people that suicide had advantages and one can eat glass. Google is similarly cautious. OpenAI came along and let the cat out of the bag. I wonder if Microsoft has thought through what might happen if they hook it to Bing and it starts telling people crazy things. All it took was a couple of posts on 4chan to get the Q Anon movement off the ground. anthropomorphize attribute human characteristics or behavior to (a god, animal, or object). "people's tendency to anthropomorphize their dogs" numinous having a strong religious or spiritual quality; indicating or suggesting the presence of a divinity. "the strange, numinous beauty of this ancient landmark" What is an example of numinous? Something numinous has a strong religious quality, suggesting the presence of a divine power. When you enter a temple, church, or mosque, you might feel as though you've entered a numinous space. Edit: Write me today's astrology for a Leo ChatGPT: I'm sorry, I am not able to provide astrology predictions as it is not based on scientific evidence. It is a form of divination that is not considered reliable. Is there anything else I can help you with?
- danbruc 4y ago
  It's only natural that they will think AI have consciousness. That seems just a matter of time until they would be doing this rightly so.
  - labrador 4y ago
    You have to explain consciousness in humans before you can tell me it has arisen in silicon chips
    - noorkersz 4y ago
      you expect a written explanation of an incomplete ongoing process but 'written explanations' only really work for done and completed things, or sometimes for future things which we want to repeat; i.e. which are based on things we've done before
    - lamacase 4y ago
      I don't think that's a fair bar. There are arguments for consciousness in non-human animals that don't depend on a full explanation of consciousness in humans. But that also doesn't imply we should suspect any particular AI of being conscious just because AI in general could be conscious in principle. We don't expect worms to be conscious just because animals can be.
      - labrador 4y ago
        I wasn't clear. I'm talking about consciousness in general, humans, animals, octopi... I keep reading but haven't found an explanation that satisfies me. Ray Kurzweil has some interesting things to say about computers and consciousness in his "The Age of Spiritual Machines". I was skeptical about a lot of what he wrote when I read it many years ago. Now I'm not sure it matters. If people imbue their devices with intelligence and emotions then that's what they perceive. I've since learned about Tulpas. Some people have imaginary friends that seem very real to them. People will likely move their Tulpas out of their minds and into their AI's. https://en.wikipedia.org/wiki/Tulpa https://en.wikipedia.org/wiki/Tulpa People on reddit are already talking about Tulpa's and AI https://old.reddit.com/r/Tulpa/ https://old.reddit.com/r/Tulpa/
modzu 4y ago
there are a lot of interesting and deep ideas here, about technology in general. but the critics of ai really need to do better. according to this argument, once you understand how your own brain works, you see human intelligence is also just an "illusion". fundamentally though, there is no reason a computer cannot think. any argument against this requires an understanding of the nature of thought and intelligence, in humans, which is quite outside the wheelhouse of most classical computer scientists. for a more refreshing perspective, read anything by hinton
- qsort 4y ago
  The more simple counterargument, very well within the wheelhouse of any computer scientist, is that the question is completely irrelevant as it's entirely predicated on one's definition of "intelligence". Successful software solves a problem correctly and efficiently. It doesn't matter that GCC doesn't compile code like I would if I had to do that by hand, you still end up with compiled code, which was the original problem to solve. It doesn't matter that the sort() routine doesn't sort numbers like I would, it still does. Playing the imitation game for its own sake is kind of silly. The "illusion" lies in the attribution to a machine of capabilities it does not have.
  - modzu 4y ago
    yes, but we havent pulled off this engineering end run yet
- AlexandrB 4y ago
  > fundamentally though, there is no reason a computer cannot think. This is true, but I think there is also a fundamental reason computers cannot understand us. If I tell a future AGI "I am sick" or "I am happy", the AGI may have a conceptual understanding of what I mean, but cannot empathize with my condition. Many human experiences are intimately tied to our bodies and our hormonal systems. How do you explain a concept like "pain" to an intelligence that has no nerves?
  - noorkersz 4y ago
walleeee 4y ago
> technology does not drive history, people drive history this fails to grasp the changing nature of agency in a world brimming with hybrid intelligence we would perhaps like to think we and we alone "drive" history, and I agree with the author that we ought to try to, but technology carries its own inertia nonhuman agency accretes as we interact with our implements and artifacts this occurs even before we explicitly design autonomy into our tools, which of course we are now doing technology carves ruts in the mind and in the world
lukeschlather 4y ago
I feel like most of the discussion of this topic is muddled by conflation of a few different questions. One is whether you want human or procedural decisionmaking to guide a task. The second question is what distinguishes human and procedural decisionmaking. The third is whether or not computers are capable of executing a given procedure. As for "distinguishing human and procedural decisionmaking" I would define it as basically "human" decisionmaking is when there is some procedure but the human will disregard procedure based on some model of the "intent." You have to trust the human for this to work. So there's a fourth question, which is under what circumstances you could trust a computer to correctly model your intent. But then there's a fifth question, which is "whose intent?" But whose intentions the process serves isn't really intrinsically tied to whether humans or computers are running the procedure. Another thing is that Weizenbaum and the author seem to take it for granted that human decisionmaking is literally magic and a computer is incapable of that kind of magic. But it's unclear and I think it's an important thing to state outright.
- qsort 4y ago
  Also the question of whether or not a technology is neutral is separate from the question of whether or not a particular implementation of that technology is. Which is itself separate from the question of whether or not Mark Zuckerberg is being kind of a dick. The article isn't bad and the reference is worth reading, but it's impossible to engage with this much material at the same time as though it's one conversation. It's poor intellectual discipline, I'm sorry.
- mjburgess 4y ago
  It's not just intention. It's, as some times dismissively put, "common sense". What price is an airline ticket on the first day of a global pandemic? No one knows, and there isn't anything to provide a "procedural system" in order for it to model what our "intention" would be in those cases. This seems obvious with a pandemic, but it is common place. Every day is a genuinely novel occurence, and counts only as a repetition of some superficial patterns in very very limited respects. When this novelty becomes relevant to a problem can never, (obviously), be pre-stated because we don't yet know what that novelty is. There is simply no substitute for being-in-the-world as animals are, that is: moving around, communicating, acting, reacting and participating. Growing, learning, reproducing, and problem-solving in direct causal contact with reality. This isn't magic, but there is no "procedural model" of it; nor should we expect there to be. There aren't procedural models of almost any phenomenon. What we can build a computer to do is "almost nothing", and we shouldnt expect the trivial problem space given by this to exhaust what we actually need machines to do.
agentultra 4y ago
I get the sense that when people want to believe something is true they will willingly ignore evidence to the contrary. They want to believe that consciousness can form from a giant spreadsheet of inferences. And so they will find evidence to support that theory. ChatGPT will change the world! Sure, it already has. If we pay attention to the outcomes and see evidence of how it's changing the world.. it's mostly people doing the changing with their desires, beliefs, and politics. Same as always. The system itself is nothing but an automaton. But aren't we all automatons too, building our self awareness around models of the word? Ah the AI version of the, "You can't prove that it's not consciousness!" Well we can as many scientists studying consciousness in humans and animals can attest: it's complicated, but we're getting better at understanding what it is. ChatGPT ain't it. Definitely worth looking at the motivations of the people building the technology itself.
mjburgess 4y ago
> what largely transformed Weizenbaum into an outspoken critic of AI ... was his revelation that even once the processes were explained many people still bought into the “illusion.” ... that even many people who understood the inner workings of computers quite well could still get swept away as well. Weizenbaum observed that ... “if nothing else, how easy it is to create and maintain the illusion of understanding, hence perhaps of judgement deserving of credibility... A certain danger lurks there.” This speaks to my view exactly, and I haven't seen it put better. The latent (imv) schizophrenic impulse to impart consciousness to any-old-object in the world is here being exploited by peddlers (often charlatans) of technology. This makes the "confirmation bias" here severe and dangerous. People don't just want to believe, their animal psychology primes them for (over-)imparting consciousness. This is a specific cognitive vulnerability that is highly pernicious. Even, trivially, consider "deresponsibilization", that process by which people take whatever-the-machine-says to be "The Answer". Why? Well, the machine "must know". Less trivially, moral bias: "presumably" the machine is "objective" and so is morally superior. How often do we hear this? Too much.
- rhn_mk1 4y ago
  I'm not so quick to attach a judgement to anthropomorphization. Well okay, I am, but not because I think it's wrong. After all, panpsychism has not been defeated as a philosophical standpoint [0]. Perhaps any-old-object in the world is actually conscius, just not all of us can detect it. That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. [0]https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/
  - mjburgess 4y ago
    I think part of the scam is the move from "practical definitions" of words to (pseudo-) philosophical ones. There ought be a name for this type of psychological manipulation, but I don't know it. It is universal in a certain type of "motivational scam". Crypto Scammer: Isn't it bad that finance is so centralised? Crypto Scammer: Let me sell you a decentralized database! Sceptic: But that isn't the same sense of decentarlization? Scammer: Ah! But doesnt power flow from centralization of infrastructure? Sceptic:... maybe, maybe not. Hey, why do you have your hand in my pocket? Here, whether "consciousness" is on a continuum across all physical objects, or whether only a subset of objects have it, is kinda irrelevant: The machine doesnt care about you, has no intentions, has nothing it wishes to say, and doesnt know anything about your or anything else. Its text generation is only a reflection of you, and the average of the superficial data of everyone else around you. In sum: Don't take philosophy classes from sales people.
rhelz 4y ago
C.F. the "AI effect": once you understand how the "magic" is done by very simple individual steps it stops being magical and you tend to think "that's not AI, its just computation." After thinking this for a while, you conclude that AI hasn't gotten anywhere despite decades of effort. Perhaps we are finally thinking that AI is getting somewhere precisely because we don't understand how neural nets really work yet. It keeps the magic alive long enough for us to realize that progress really is being made.
- sumtechguy 4y ago
  The first thing I can think of that was like the 'AI' we have now is seeing the cubic spline method. It was shockingly good at particular points. But between those points it could be 'meh close enough' or wildly bad. AI we have now seems more along the lines of 'meh close enough'. Which for many things is actually good enough. For me I have always looked at the nets as a cubic spline smeared across dozens of nodes with a newtons method for searching out 'optimal'. Bad analogy I know. But that is the closest I can get to explaining 'how they work'.
- int_19h 4y ago
  The other factor here is that we appear to be trying really hard to keep it "magic" wrt ourselves. At the very least, the notion that it might be "just computation" in the brain seems to be very discomforting to some people. Which might affect the way we look at technologies that try to model it, or at least use similar approaches - i.e. we're primed to treat them as more "magic" than they really are, or else to invent some reason why we think ourselves to be different.
cmrdporcupine 4y ago
This is a great essay. I'm going to read it a few times. One thing I have thought about over the years more and more, in this general vein is that it is sometimes to think about "computing technology" as being an evolution less as a specific thing itself and more of a broader "automation technology" which emerges most dramatically with the arrival of the industrial revolution (though there were aspects of it before that.) Thinking with a kind of materialist analysis, we can see computing but even (more broadly) higher level mathematics, logic, etc. as an intellectual effort to chip away at labour costs & the percentage of human effort involved in production/work/business. It's a deeply obscured profit motive. The ideological blindness described in this article I think fits neatly into that: we are all so enamoured with this computing thing, this process, the puzzle and the end results that we often fail to see that it is not some magical force operating under its own motive, with its own natural inevitable destination and we often do not see that there's likely a deeper force underlying all of this... human efforts, labour, driven by profit and hunger. Ultimately computing technology is capitalist automation technology. It reflects the needs of the capitalist market system, and the drive to efficiency that the market economy requires. The 'magic' we seem to pepper overtop is our own flourish. (Maybe when I'm finally done with this software engineering career that has eaten my life, I will go back and finish my philosophy degree I walked away from in the 90s... and then I will write more coherently about this)
- CatWChainsaw 4y ago
  We need more people talking about this seriously now, because we are on a precipice of some sort.
  - cmrdporcupine 4y ago
    I feel like people have never been not talking about it. I probably should have tossed in a reference to Heidegger & his focus on 'instrumentality' etc, too. But the thing is.. how do they say it? "Money talks, bullshit walks?" People follow what fills their stomach. And mechanization/automation/computation, it does that in spades. I mean... I dropped out of my philosophy major decades ago to go do the More Fun To Compute thing... My bank account shows it. In any case... the first major uses of computers were for war. The next, for automation in commerce and finance. That's what computers are really for.
lxgr 4y ago
In my computer science undergrad program, we watched and discussed a movie about Joseph Weizenbaum and Ray Kurzweil. I still vividly remember that discussion. At the time, it seemed obvious to me that Kurzweil got it right and Weizenbaum got it wrong; that clearly, ELIZA was a red herring; that people wouldn't be fooled that easily; that the first AI to truly pass the Turing test would be the real thing (and we'd definitely see one, and with that the Singularity, in my lifetime). I think about Weizenbaum a lot these days.
- throwanem 4y ago
  Was it Plug & Pray? If so, I'll add it to my list.
  - lxgr 4y ago
    Not 100% sure, but that sounds like it.
grantcas 4y ago
[flagged]
- throwanem 4y ago
  This is wildly off topic for a thread from an article that touches nowhere near the question of whether a machine can be made conscious.
  - FreakLegion 4y ago
    If you look at their comment history you'll see this account exclusively regurgitates this same text, over and over. It's the entirety of their comment history.
ccvannorman 4y ago
> “On the one hand the computer makes it possible in principle to live in a world of plenty for everyone, on the other hand we are well on our way to using it to create a world of suffering and chaos. Paradoxical, no?” Um, what? I use a computer and it 100% is responsbile for my "world of plenty." If you're using a computer and your life sucks, you're using it wrong. That's not the computer's fault, and one would do well to abstain from making blanket judgements about the human-computer relationship as if the idea of a computer is to blame.
8bitsrule 4y ago
"And if you are being fooled, it is worth considering who it is that is trying to fool you…" Well worth reading. You'd think we'd be more cautious after learning what the oil companies didn't share with us. And how much we've invested in that we now have to discard. Does GPT provide easily-read citations for each of its assertions? What steps does it take to ensure it's balanced, ethical? Doing that stuff well takes time. Instant gratification? Danger Will Robinson.

131 points by pabs3 4y ago

Article so wordy it could use an AI summary, but this quote stands out: > Writing of the enthusiastic embrace of a fully computerized world, Weizenbaum grumbled, “These people see the technical apparatus underlying Orwell’s 1984 and, like children on seeing the beach, they run for it” > a point to which Weizenbaum added “I wish it were their private excursion, but they demand that we all come along.”

labrador 4y ago

The first time I sensed I was dealing with an alien intelligence was playing a chess machine in 1980. I remember being perplexed that it beat me and aware the machine embodied intelligence in some way that I didn't understand It's not really clear to me if this article is directed at scientists and engineers getting carried away or the general public. If the former, yes they should know better, but the general public is mostly not rational, contrary to what I used to think about the reasoning person. Many people attribute thoughts and feeling to their pets that they don't have. They think crystals have special powers. It's only natural that they will think AI have consciousness. I think as scientists and engineers we have a responsibility to prevent our AI creations from telling people to drink the poisoned Kool Aid or eat glass. Yann LeCun said they shut down Meta's chat AI because it told people that suicide had advantages and one can eat glass. Google is similarly cautious. OpenAI came along and let the cat out of the bag. I wonder if Microsoft has thought through what might happen if they hook it to Bing and it starts telling people crazy things. All it took was a couple of posts on 4chan to get the Q Anon movement off the ground. anthropomorphize attribute human characteristics or behavior to (a god, animal, or object). "people's tendency to anthropomorphize their dogs" numinous having a strong religious or spiritual quality; indicating or suggesting the presence of a divinity. "the strange, numinous beauty of this ancient landmark" What is an example of numinous? Something numinous has a strong religious quality, suggesting the presence of a divine power. When you enter a temple, church, or mosque, you might feel as though you've entered a numinous space. Edit: Write me today's astrology for a Leo ChatGPT: I'm sorry, I am not able to provide astrology predictions as it is not based on scientific evidence. It is a form of divination that is not considered reliable. Is there anything else I can help you with?

danbruc 4y ago
It's only natural that they will think AI have consciousness. That seems just a matter of time until they would be doing this rightly so.
- labrador 4y ago
  You have to explain consciousness in humans before you can tell me it has arisen in silicon chips
  - noorkersz 4y ago
    you expect a written explanation of an incomplete ongoing process but 'written explanations' only really work for done and completed things, or sometimes for future things which we want to repeat; i.e. which are based on things we've done before
  - lamacase 4y ago
    I don't think that's a fair bar. There are arguments for consciousness in non-human animals that don't depend on a full explanation of consciousness in humans. But that also doesn't imply we should suspect any particular AI of being conscious just because AI in general could be conscious in principle. We don't expect worms to be conscious just because animals can be.
    - labrador 4y ago
      I wasn't clear. I'm talking about consciousness in general, humans, animals, octopi... I keep reading but haven't found an explanation that satisfies me. Ray Kurzweil has some interesting things to say about computers and consciousness in his "The Age of Spiritual Machines". I was skeptical about a lot of what he wrote when I read it many years ago. Now I'm not sure it matters. If people imbue their devices with intelligence and emotions then that's what they perceive. I've since learned about Tulpas. Some people have imaginary friends that seem very real to them. People will likely move their Tulpas out of their minds and into their AI's. https://en.wikipedia.org/wiki/Tulpa https://en.wikipedia.org/wiki/Tulpa People on reddit are already talking about Tulpa's and AI https://old.reddit.com/r/Tulpa/ https://old.reddit.com/r/Tulpa/

modzu 4y ago

there are a lot of interesting and deep ideas here, about technology in general. but the critics of ai really need to do better. according to this argument, once you understand how your own brain works, you see human intelligence is also just an "illusion". fundamentally though, there is no reason a computer cannot think. any argument against this requires an understanding of the nature of thought and intelligence, in humans, which is quite outside the wheelhouse of most classical computer scientists. for a more refreshing perspective, read anything by hinton

qsort 4y ago
The more simple counterargument, very well within the wheelhouse of any computer scientist, is that the question is completely irrelevant as it's entirely predicated on one's definition of "intelligence". Successful software solves a problem correctly and efficiently. It doesn't matter that GCC doesn't compile code like I would if I had to do that by hand, you still end up with compiled code, which was the original problem to solve. It doesn't matter that the sort() routine doesn't sort numbers like I would, it still does. Playing the imitation game for its own sake is kind of silly. The "illusion" lies in the attribution to a machine of capabilities it does not have.
- modzu 4y ago
  yes, but we havent pulled off this engineering end run yet
AlexandrB 4y ago
> fundamentally though, there is no reason a computer cannot think. This is true, but I think there is also a fundamental reason computers cannot understand us. If I tell a future AGI "I am sick" or "I am happy", the AGI may have a conceptual understanding of what I mean, but cannot empathize with my condition. Many human experiences are intimately tied to our bodies and our hormonal systems. How do you explain a concept like "pain" to an intelligence that has no nerves?
- noorkersz 4y ago

walleeee 4y ago

> technology does not drive history, people drive history this fails to grasp the changing nature of agency in a world brimming with hybrid intelligence we would perhaps like to think we and we alone "drive" history, and I agree with the author that we ought to try to, but technology carries its own inertia nonhuman agency accretes as we interact with our implements and artifacts this occurs even before we explicitly design autonomy into our tools, which of course we are now doing technology carves ruts in the mind and in the world

lukeschlather 4y ago

I feel like most of the discussion of this topic is muddled by conflation of a few different questions. One is whether you want human or procedural decisionmaking to guide a task. The second question is what distinguishes human and procedural decisionmaking. The third is whether or not computers are capable of executing a given procedure. As for "distinguishing human and procedural decisionmaking" I would define it as basically "human" decisionmaking is when there is some procedure but the human will disregard procedure based on some model of the "intent." You have to trust the human for this to work. So there's a fourth question, which is under what circumstances you could trust a computer to correctly model your intent. But then there's a fifth question, which is "whose intent?" But whose intentions the process serves isn't really intrinsically tied to whether humans or computers are running the procedure. Another thing is that Weizenbaum and the author seem to take it for granted that human decisionmaking is literally magic and a computer is incapable of that kind of magic. But it's unclear and I think it's an important thing to state outright.

qsort 4y ago
Also the question of whether or not a technology is neutral is separate from the question of whether or not a particular implementation of that technology is. Which is itself separate from the question of whether or not Mark Zuckerberg is being kind of a dick. The article isn't bad and the reference is worth reading, but it's impossible to engage with this much material at the same time as though it's one conversation. It's poor intellectual discipline, I'm sorry.
mjburgess 4y ago
It's not just intention. It's, as some times dismissively put, "common sense". What price is an airline ticket on the first day of a global pandemic? No one knows, and there isn't anything to provide a "procedural system" in order for it to model what our "intention" would be in those cases. This seems obvious with a pandemic, but it is common place. Every day is a genuinely novel occurence, and counts only as a repetition of some superficial patterns in very very limited respects. When this novelty becomes relevant to a problem can never, (obviously), be pre-stated because we don't yet know what that novelty is. There is simply no substitute for being-in-the-world as animals are, that is: moving around, communicating, acting, reacting and participating. Growing, learning, reproducing, and problem-solving in direct causal contact with reality. This isn't magic, but there is no "procedural model" of it; nor should we expect there to be. There aren't procedural models of almost any phenomenon. What we can build a computer to do is "almost nothing", and we shouldnt expect the trivial problem space given by this to exhaust what we actually need machines to do.

agentultra 4y ago

I get the sense that when people want to believe something is true they will willingly ignore evidence to the contrary. They want to believe that consciousness can form from a giant spreadsheet of inferences. And so they will find evidence to support that theory. ChatGPT will change the world! Sure, it already has. If we pay attention to the outcomes and see evidence of how it's changing the world.. it's mostly people doing the changing with their desires, beliefs, and politics. Same as always. The system itself is nothing but an automaton. But aren't we all automatons too, building our self awareness around models of the word? Ah the AI version of the, "You can't prove that it's not consciousness!" Well we can as many scientists studying consciousness in humans and animals can attest: it's complicated, but we're getting better at understanding what it is. ChatGPT ain't it. Definitely worth looking at the motivations of the people building the technology itself.

mjburgess 4y ago

> what largely transformed Weizenbaum into an outspoken critic of AI ... was his revelation that even once the processes were explained many people still bought into the “illusion.” ... that even many people who understood the inner workings of computers quite well could still get swept away as well. Weizenbaum observed that ... “if nothing else, how easy it is to create and maintain the illusion of understanding, hence perhaps of judgement deserving of credibility... A certain danger lurks there.” This speaks to my view exactly, and I haven't seen it put better. The latent (imv) schizophrenic impulse to impart consciousness to any-old-object in the world is here being exploited by peddlers (often charlatans) of technology. This makes the "confirmation bias" here severe and dangerous. People don't just want to believe, their animal psychology primes them for (over-)imparting consciousness. This is a specific cognitive vulnerability that is highly pernicious. Even, trivially, consider "deresponsibilization", that process by which people take whatever-the-machine-says to be "The Answer". Why? Well, the machine "must know". Less trivially, moral bias: "presumably" the machine is "objective" and so is morally superior. How often do we hear this? Too much.

rhn_mk1 4y ago
I'm not so quick to attach a judgement to anthropomorphization. Well okay, I am, but not because I think it's wrong. After all, panpsychism has not been defeated as a philosophical standpoint [0]. Perhaps any-old-object in the world is actually conscius, just not all of us can detect it. That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. [0]https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/
- mjburgess 4y ago
  I think part of the scam is the move from "practical definitions" of words to (pseudo-) philosophical ones. There ought be a name for this type of psychological manipulation, but I don't know it. It is universal in a certain type of "motivational scam". Crypto Scammer: Isn't it bad that finance is so centralised? Crypto Scammer: Let me sell you a decentralized database! Sceptic: But that isn't the same sense of decentarlization? Scammer: Ah! But doesnt power flow from centralization of infrastructure? Sceptic:... maybe, maybe not. Hey, why do you have your hand in my pocket? Here, whether "consciousness" is on a continuum across all physical objects, or whether only a subset of objects have it, is kinda irrelevant: The machine doesnt care about you, has no intentions, has nothing it wishes to say, and doesnt know anything about your or anything else. Its text generation is only a reflection of you, and the average of the superficial data of everyone else around you. In sum: Don't take philosophy classes from sales people.

rhelz 4y ago

C.F. the "AI effect": once you understand how the "magic" is done by very simple individual steps it stops being magical and you tend to think "that's not AI, its just computation." After thinking this for a while, you conclude that AI hasn't gotten anywhere despite decades of effort. Perhaps we are finally thinking that AI is getting somewhere precisely because we don't understand how neural nets really work yet. It keeps the magic alive long enough for us to realize that progress really is being made.

sumtechguy 4y ago
The first thing I can think of that was like the 'AI' we have now is seeing the cubic spline method. It was shockingly good at particular points. But between those points it could be 'meh close enough' or wildly bad. AI we have now seems more along the lines of 'meh close enough'. Which for many things is actually good enough. For me I have always looked at the nets as a cubic spline smeared across dozens of nodes with a newtons method for searching out 'optimal'. Bad analogy I know. But that is the closest I can get to explaining 'how they work'.
int_19h 4y ago
The other factor here is that we appear to be trying really hard to keep it "magic" wrt ourselves. At the very least, the notion that it might be "just computation" in the brain seems to be very discomforting to some people. Which might affect the way we look at technologies that try to model it, or at least use similar approaches - i.e. we're primed to treat them as more "magic" than they really are, or else to invent some reason why we think ourselves to be different.

cmrdporcupine 4y ago

This is a great essay. I'm going to read it a few times. One thing I have thought about over the years more and more, in this general vein is that it is sometimes to think about "computing technology" as being an evolution less as a specific thing itself and more of a broader "automation technology" which emerges most dramatically with the arrival of the industrial revolution (though there were aspects of it before that.) Thinking with a kind of materialist analysis, we can see computing but even (more broadly) higher level mathematics, logic, etc. as an intellectual effort to chip away at labour costs & the percentage of human effort involved in production/work/business. It's a deeply obscured profit motive. The ideological blindness described in this article I think fits neatly into that: we are all so enamoured with this computing thing, this process, the puzzle and the end results that we often fail to see that it is not some magical force operating under its own motive, with its own natural inevitable destination and we often do not see that there's likely a deeper force underlying all of this... human efforts, labour, driven by profit and hunger. Ultimately computing technology is capitalist automation technology. It reflects the needs of the capitalist market system, and the drive to efficiency that the market economy requires. The 'magic' we seem to pepper overtop is our own flourish. (Maybe when I'm finally done with this software engineering career that has eaten my life, I will go back and finish my philosophy degree I walked away from in the 90s... and then I will write more coherently about this)

CatWChainsaw 4y ago
We need more people talking about this seriously now, because we are on a precipice of some sort.
- cmrdporcupine 4y ago
  I feel like people have never been not talking about it. I probably should have tossed in a reference to Heidegger & his focus on 'instrumentality' etc, too. But the thing is.. how do they say it? "Money talks, bullshit walks?" People follow what fills their stomach. And mechanization/automation/computation, it does that in spades. I mean... I dropped out of my philosophy major decades ago to go do the More Fun To Compute thing... My bank account shows it. In any case... the first major uses of computers were for war. The next, for automation in commerce and finance. That's what computers are really for.

lxgr 4y ago

In my computer science undergrad program, we watched and discussed a movie about Joseph Weizenbaum and Ray Kurzweil. I still vividly remember that discussion. At the time, it seemed obvious to me that Kurzweil got it right and Weizenbaum got it wrong; that clearly, ELIZA was a red herring; that people wouldn't be fooled that easily; that the first AI to truly pass the Turing test would be the real thing (and we'd definitely see one, and with that the Singularity, in my lifetime). I think about Weizenbaum a lot these days.

throwanem 4y ago
Was it Plug & Pray? If so, I'll add it to my list.
- lxgr 4y ago
  Not 100% sure, but that sounds like it.

grantcas 4y ago

[flagged]

throwanem 4y ago
This is wildly off topic for a thread from an article that touches nowhere near the question of whether a machine can be made conscious.
- FreakLegion 4y ago
  If you look at their comment history you'll see this account exclusively regurgitates this same text, over and over. It's the entirety of their comment history.

ccvannorman 4y ago

> “On the one hand the computer makes it possible in principle to live in a world of plenty for everyone, on the other hand we are well on our way to using it to create a world of suffering and chaos. Paradoxical, no?” Um, what? I use a computer and it 100% is responsbile for my "world of plenty." If you're using a computer and your life sucks, you're using it wrong. That's not the computer's fault, and one would do well to abstain from making blanket judgements about the human-computer relationship as if the idea of a computer is to blame.

8bitsrule 4y ago

"And if you are being fooled, it is worth considering who it is that is trying to fool you…" Well worth reading. You'd think we'd be more cautious after learning what the oil companies didn't share with us. And how much we've invested in that we now have to discard. Does GPT provide easily-read citations for each of its assertions? What steps does it take to ensure it's balanced, ethical? Doing that stuff well takes time. Instant gratification? Danger Will Robinson.