5 ms·

> what largely transformed Weizenbaum into an outspoken critic of AI ... was his revelation that even once the processes were explained many people still bought

> what largely transformed Weizenbaum into an outspoken critic of AI ... was his revelation that even once the processes were explained many people still bought into the “illusion.” ... that even many people who understood the inner workings of computers quite well could still get swept away as well. Weizenbaum observed that ... “if nothing else, how easy it is to create and maintain the illusion of understanding, hence perhaps of judgement deserving of credibility... A certain danger lurks there.” This speaks to my view exactly, and I haven't seen it put better. The latent (imv) schizophrenic impulse to impart consciousness to any-old-object in the world is here being exploited by peddlers (often charlatans) of technology. This makes the "confirmation bias" here severe and dangerous. People don't just want to believe, their animal psychology primes them for (over-)imparting consciousness. This is a specific cognitive vulnerability that is highly pernicious. Even, trivially, consider "deresponsibilization", that process by which people take whatever-the-machine-says to be "The Answer". Why? Well, the machine "must know". Less trivially, moral bias: "presumably" the machine is "objective" and so is morally superior. How often do we hear this? Too much.

rhn_mk1 4y ago
I'm not so quick to attach a judgement to anthropomorphization. Well okay, I am, but not because I think it's wrong. After all, panpsychism has not been defeated as a philosophical standpoint [0]. Perhaps any-old-object in the world is actually conscius, just not all of us can detect it. That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. [0]https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/
- mjburgess 4y ago
  I think part of the scam is the move from "practical definitions" of words to (pseudo-) philosophical ones. There ought be a name for this type of psychological manipulation, but I don't know it. It is universal in a certain type of "motivational scam". Crypto Scammer: Isn't it bad that finance is so centralised? Crypto Scammer: Let me sell you a decentralized database! Sceptic: But that isn't the same sense of decentarlization? Scammer: Ah! But doesnt power flow from centralization of infrastructure? Sceptic:... maybe, maybe not. Hey, why do you have your hand in my pocket? Here, whether "consciousness" is on a continuum across all physical objects, or whether only a subset of objects have it, is kinda irrelevant: The machine doesnt care about you, has no intentions, has nothing it wishes to say, and doesnt know anything about your or anything else. Its text generation is only a reflection of you, and the average of the superficial data of everyone else around you. In sum: Don't take philosophy classes from sales people.
  - rhn_mk1 4y ago
    > whether only a subset of objects have it, is kinda irrelevant: I'm with you. The scammer doesn't care. > The machine doesnt care about you Uh, no. It's irrelevant to the scam. But now you're guessing at the internal feelings of an alien consciousness. How do I know if I am not just a reflection of my environment, and my behaviour an average of superficial data of everyone around me? Prove me conscious, and I'll start taking you seriously.
    - mjburgess 4y ago
      >Prove me conscious, and I'll start taking you seriously. You realise, this is the same fallacy? You've shifted the burden of proof on to some bizarre philosophical ground so that you dont have to defend your trick magic box. A priori, by common sense, you're conscious. A priori, by common sense, a machine is not. There lie the default burdens of proof. From this the sales person must proceed to convince me, not vice versa.
      - rhn_mk1 4y ago
        I don't know what's in the box. You don't know what's in the box. Yet you say it can't possibly be this - so prove it. It's really simple. I don't have to defend the box. I defend the right to say that we don't know in the absence of facts. Common sense isn't facts, nor is it a priori.
- dbspin 4y ago
  > That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. It has quite a bit of bearing, given that the reason we have discussions of consciousness at all in our own perception (real or imagined) of our own awareness being conscious. This arises from the very same brain / self that is innately driven to pareidolia. Literally the mechanisms that let us perceive consciousness arising (or project consciousness onto) other humans are the same ones responsible for pareidolia.
adamsmith143 4y ago
>The latent (imv) schizophrenic impulse to impart consciousness to any-old-object in the world is here being exploited by peddlers (often charlatans) of technology. This makes the "confirmation bias" here severe and dangerous. Have any examples of charlatans peddling "conscious" AIs?
- haswell 4y ago
  Even when no direct claims about consciousness are being made, that is not stopping people from behaving as if they are conscious programs. I think we're seeing this playing out in discussions about generative AI with people justifying certain behaviors regarding the use of artist's data in training models by comparing the program to a human going through the process of "learning" and "gaining inspiration" from the work of others. Some people seem fully convinced that what these programs are doing is equivalent to human behavior to a degree that would qualify the program to receive human-like considerations, i.e. treating the software like an entity instead of just a program/tool to be wielded by the end-user. I think this is one of the most pernicious issues, and people seem to have a hard time recognizing the trap they've fallen into. I think this is also an outcome of this software being sufficiently complex to be too hard to understand without investing time and effort in understanding it. I think this results in the mis-attribution of intelligence when instead it's just a really clever piece of software. But the casual commenter cannot estimate the gap between "pretty clever" and "so complex that it's actual intelligence".
  - pixl97 4y ago
    But I think in many ways you're falling into the same trap... So, let me set the first trap for you. Give me a logically/mathematically defined definition of intelligence and one of consciousness. Unfortunately after decades of back and forth on this we typically end up with answers that fall out of science and into the "I'll know it when I see it territory". If you are going to describe machine intelligence/consciousness your definition must be able to cover intelligent behaviors we see from single cell life to the massive complexity we see in humans. Attempting to handwave and say "it's not as complex as humans therefore it's not intelligent/conscious" is a complete and total failure from my point of view.
eointierney 4y ago
Anthropomorphisation is a curse, eh? We've seen this in deities for as long as we've been around, royalties then took the crown, later subsumed by governments, then the daft notion of non-human personalities called companies. Now, aproximate intelligence systems. We humans are credulous
adamisom 4y ago
I'm only addressing a fraction of your many excellent points, but: this is actually why uncontrolled AGI is so spooky. It's the prospect of a future where space gets colonized-or whatever crazy goal(s)-but there's "nobody home", no qualia, no experiencer.
- BlueTemplar 4y ago
  This assumes though that : - AGI is possible. - AGI is possible without recreating qualia / an experiencer in some form.
  - jamiek88 4y ago
    Have you ever read Blindsight? It goes into the idea of intelligence without consciousness very well. Why should there be an experiencer? Is that necessary or even optimal for intelligence?
    - BlueTemplar 4y ago
      We really don't know ! (At least yet...)
  - adamisom 4y ago
    Well, I hope it’s not possible !
  - dbspin 4y ago
    The original point stands even if AGI isn't possible in the strong sense. Self replicating embodied AI robots (in the contemporary machine learning / deep learning limited sense) may well be possible, and dangerous while being cognitively limited / affectively absent.
api 4y ago
Don’t all these processes also play a role in cult behaviors and why people follow empty charismatic leaders? Even when the Great Leader is unmasked as a fool or a deliberate con, people often still continue to believe. Sometimes they respond to the revelation by doubling down.
JoshCole 4y ago
Including the dataset of lives lived under slavery or cut short by genocide makes me come to radically different conclusions with regard to how prone to over-imparting consciousness humans tend to be. Adjusting exploited entities by probability of exploitation and adjusting advantaged entities by probability of advantage also leads me to radically different conclusions. For example, Ray Dalio is one advocate for using AI to supplement decisions. He is also exceedingly rich and a large part of his fortune was made with the help of investment decisions predicated on AI reasoning combined with human reasoning. Even though on average people who try to benefit from AI might not, the average benefit can still be massively positive, because the outcome distribution isn't uniform. Losing means losing a small fraction of a small amount, but winning means gaining an extreme amount to such an extent that it pays for the bad bets. This appears to often happen in practice - see the venture industries economics for evidence of this - and I don't see why it shouldn't apply to AI, which is something present in most unicorns that people often point to when discussing this concept.
- tuatoru 4y ago
  > how prone to over-imparting consciousness humans tend to be gp was talking about humans' attribution of consciousness to inanimate objects, not other humans. Nearly all religions apart from the more abstract big three would be counterexamples to your example. Dryads and naiads, all kinds of shrines and oracles and talismans...
  - cutemonster 4y ago
    Also, two wrongs (slavery etc, and imparting consciousness on inanimate objects and machines) don't make a right
    - JoshCole 4y ago
      Okay. How confident are you that I'm wrong? Would you be willing to construct a wager with me? I ask, for the purpose of the wager, that you generalize the claim that overestimation of intelligence is exploitable. The bet will be as follows: $10,000, 1:1 odds. You will play, without the aid of a computer, against StockFish. My contention is that you will (1) lose the game and (2) lose it despite the fact that StockFish assumes you make better moves than you actually do. Your contention is that you will (1) win the game and (2) do this because StockFish predicted moves which overestimated your actual abilities, which makes it exploitable. I don't think you will be confident enough to put disagreement with my epistemics where your money is. However, to encourage you to do so, I will allow you to instead opt to donate the money to a charity of your choosing rather than giving it to me.
tpoacher 4y ago
I would highly recommend The Talos Principle to you if you haven't played it. I think you would enjoy it. It certainly influenced how I think about this topic.

rhn_mk1 4y ago

I'm not so quick to attach a judgement to anthropomorphization. Well okay, I am, but not because I think it's wrong. After all, panpsychism has not been defeated as a philosophical standpoint [0]. Perhaps any-old-object in the world is actually conscius, just not all of us can detect it. That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. [0]https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/

mjburgess 4y ago
I think part of the scam is the move from "practical definitions" of words to (pseudo-) philosophical ones. There ought be a name for this type of psychological manipulation, but I don't know it. It is universal in a certain type of "motivational scam". Crypto Scammer: Isn't it bad that finance is so centralised? Crypto Scammer: Let me sell you a decentralized database! Sceptic: But that isn't the same sense of decentarlization? Scammer: Ah! But doesnt power flow from centralization of infrastructure? Sceptic:... maybe, maybe not. Hey, why do you have your hand in my pocket? Here, whether "consciousness" is on a continuum across all physical objects, or whether only a subset of objects have it, is kinda irrelevant: The machine doesnt care about you, has no intentions, has nothing it wishes to say, and doesnt know anything about your or anything else. Its text generation is only a reflection of you, and the average of the superficial data of everyone else around you. In sum: Don't take philosophy classes from sales people.
- rhn_mk1 4y ago
  > whether only a subset of objects have it, is kinda irrelevant: I'm with you. The scammer doesn't care. > The machine doesnt care about you Uh, no. It's irrelevant to the scam. But now you're guessing at the internal feelings of an alien consciousness. How do I know if I am not just a reflection of my environment, and my behaviour an average of superficial data of everyone around me? Prove me conscious, and I'll start taking you seriously.
  - mjburgess 4y ago
    >Prove me conscious, and I'll start taking you seriously. You realise, this is the same fallacy? You've shifted the burden of proof on to some bizarre philosophical ground so that you dont have to defend your trick magic box. A priori, by common sense, you're conscious. A priori, by common sense, a machine is not. There lie the default burdens of proof. From this the sales person must proceed to convince me, not vice versa.
    - rhn_mk1 4y ago
      I don't know what's in the box. You don't know what's in the box. Yet you say it can't possibly be this - so prove it. It's really simple. I don't have to defend the box. I defend the right to say that we don't know in the absence of facts. Common sense isn't facts, nor is it a priori.
dbspin 4y ago
> That the impulse to see consciousness in a thing is being used by charlatan is unfortunate, but has no bearing on the actual truth of whether the thing actually is conscious. It has quite a bit of bearing, given that the reason we have discussions of consciousness at all in our own perception (real or imagined) of our own awareness being conscious. This arises from the very same brain / self that is innately driven to pareidolia. Literally the mechanisms that let us perceive consciousness arising (or project consciousness onto) other humans are the same ones responsible for pareidolia.

adamsmith143 4y ago

>The latent (imv) schizophrenic impulse to impart consciousness to any-old-object in the world is here being exploited by peddlers (often charlatans) of technology. This makes the "confirmation bias" here severe and dangerous. Have any examples of charlatans peddling "conscious" AIs?

haswell 4y ago
Even when no direct claims about consciousness are being made, that is not stopping people from behaving as if they are conscious programs. I think we're seeing this playing out in discussions about generative AI with people justifying certain behaviors regarding the use of artist's data in training models by comparing the program to a human going through the process of "learning" and "gaining inspiration" from the work of others. Some people seem fully convinced that what these programs are doing is equivalent to human behavior to a degree that would qualify the program to receive human-like considerations, i.e. treating the software like an entity instead of just a program/tool to be wielded by the end-user. I think this is one of the most pernicious issues, and people seem to have a hard time recognizing the trap they've fallen into. I think this is also an outcome of this software being sufficiently complex to be too hard to understand without investing time and effort in understanding it. I think this results in the mis-attribution of intelligence when instead it's just a really clever piece of software. But the casual commenter cannot estimate the gap between "pretty clever" and "so complex that it's actual intelligence".
- pixl97 4y ago
  But I think in many ways you're falling into the same trap... So, let me set the first trap for you. Give me a logically/mathematically defined definition of intelligence and one of consciousness. Unfortunately after decades of back and forth on this we typically end up with answers that fall out of science and into the "I'll know it when I see it territory". If you are going to describe machine intelligence/consciousness your definition must be able to cover intelligent behaviors we see from single cell life to the massive complexity we see in humans. Attempting to handwave and say "it's not as complex as humans therefore it's not intelligent/conscious" is a complete and total failure from my point of view.

eointierney 4y ago

Anthropomorphisation is a curse, eh? We've seen this in deities for as long as we've been around, royalties then took the crown, later subsumed by governments, then the daft notion of non-human personalities called companies. Now, aproximate intelligence systems. We humans are credulous

adamisom 4y ago

I'm only addressing a fraction of your many excellent points, but: this is actually why uncontrolled AGI is so spooky. It's the prospect of a future where space gets colonized-or whatever crazy goal(s)-but there's "nobody home", no qualia, no experiencer.

BlueTemplar 4y ago
This assumes though that : - AGI is possible. - AGI is possible without recreating qualia / an experiencer in some form.
- jamiek88 4y ago
  Have you ever read Blindsight? It goes into the idea of intelligence without consciousness very well. Why should there be an experiencer? Is that necessary or even optimal for intelligence?
  - BlueTemplar 4y ago
    We really don't know ! (At least yet...)
- adamisom 4y ago
  Well, I hope it’s not possible !
- dbspin 4y ago
  The original point stands even if AGI isn't possible in the strong sense. Self replicating embodied AI robots (in the contemporary machine learning / deep learning limited sense) may well be possible, and dangerous while being cognitively limited / affectively absent.

api 4y ago

Don’t all these processes also play a role in cult behaviors and why people follow empty charismatic leaders? Even when the Great Leader is unmasked as a fool or a deliberate con, people often still continue to believe. Sometimes they respond to the revelation by doubling down.

JoshCole 4y ago

Including the dataset of lives lived under slavery or cut short by genocide makes me come to radically different conclusions with regard to how prone to over-imparting consciousness humans tend to be. Adjusting exploited entities by probability of exploitation and adjusting advantaged entities by probability of advantage also leads me to radically different conclusions. For example, Ray Dalio is one advocate for using AI to supplement decisions. He is also exceedingly rich and a large part of his fortune was made with the help of investment decisions predicated on AI reasoning combined with human reasoning. Even though on average people who try to benefit from AI might not, the average benefit can still be massively positive, because the outcome distribution isn't uniform. Losing means losing a small fraction of a small amount, but winning means gaining an extreme amount to such an extent that it pays for the bad bets. This appears to often happen in practice - see the venture industries economics for evidence of this - and I don't see why it shouldn't apply to AI, which is something present in most unicorns that people often point to when discussing this concept.

tuatoru 4y ago
> how prone to over-imparting consciousness humans tend to be gp was talking about humans' attribution of consciousness to inanimate objects, not other humans. Nearly all religions apart from the more abstract big three would be counterexamples to your example. Dryads and naiads, all kinds of shrines and oracles and talismans...
- cutemonster 4y ago
  Also, two wrongs (slavery etc, and imparting consciousness on inanimate objects and machines) don't make a right
  - JoshCole 4y ago
    Okay. How confident are you that I'm wrong? Would you be willing to construct a wager with me? I ask, for the purpose of the wager, that you generalize the claim that overestimation of intelligence is exploitable. The bet will be as follows: $10,000, 1:1 odds. You will play, without the aid of a computer, against StockFish. My contention is that you will (1) lose the game and (2) lose it despite the fact that StockFish assumes you make better moves than you actually do. Your contention is that you will (1) win the game and (2) do this because StockFish predicted moves which overestimated your actual abilities, which makes it exploitable. I don't think you will be confident enough to put disagreement with my epistemics where your money is. However, to encourage you to do so, I will allow you to instead opt to donate the money to a charity of your choosing rather than giving it to me.

tpoacher 4y ago

I would highly recommend The Talos Principle to you if you haven't played it. I think you would enjoy it. It certainly influenced how I think about this topic.