4 ms·

Isn’t our brain a black box too? Why exactly is it worrisome?

by ramraj07 4y ago

tobr 4y ago
Because human brains have been known to intentionally lie and manipulate, and secretly plan horrible deeds.
- eternalban 4y ago
  We'll need psych tests for our AI. Malicious AI will note this and adapt. An arms races ensues, where one adversary finances and invests in upgrading the other. Guess who wins?
shadowofneptune 4y ago
The mind itself, perhaps. The person in possession of it though usually does have a conscious strategy to play a game once they are proficient, and can happily explain it to you.
- pyinstallwoes 4y ago
  How many people can actually explain their state of mind and rational or seemingly rational thought process?
  - space_fountain 4y ago
    And we have all sorts of social science these days implying the explanations when people do try to explain themselves they’re justifying decisions that came out of black box rather than actual describing how the decision happened
    - nequo 4y ago
      And also neuroscience examining whether what we think we did out of choice was really done out of our choice: https://en.wikipedia.org/wiki/Neuroscience_of_free_will https://en.wikipedia.org/wiki/Neuroscience_of_free_will
      - taneq 4y ago
        What do you (or they) think “our choice” is if not the result of these processes? It’s not like “you” are just the part of your brain that is consciously aware of making decisions.
  - Tepix 4y ago
    I play Othello. In most situations i can give you detailed reasons why i choose to make a particular move.
    - taneq 4y ago
      Are they actually the reasons for your move, or are they post-hoc justifications, and how would you know?
    - Isinlor 4y ago
      You are not explaining your internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused you to arrive at given moves. You will only provide proofs that your moves are good.
    - bobbruno 4y ago
      You can justify post facto the move you made. Can you write down the logical process (probably a combination of strategy, move generation, analysis and selection) that would replicate the moves you picked from the beginning of the game? I don’t even need the brain chemistry (the implementation), but I don’t think even writing down the logic that generates those moves (in contrast with a logic to justify each move after it was generated) is something humans can do in most cases.
  - bsenftner 4y ago
    Asking people to explain themselves often triggers various sorts of youth punishment memory, resulting in them being both uncomfortable probing their own reasoning, and then they produce what is essentially a comforting deception for their reasoning simple for the sake of ending the agony of self probing. Asking people their internal states requires their internal guard and internal (potentially filled with self deception) voice what they are up to is very similar to asking children why they are misbehaving, and all manners of mental protections activate.
- gpm 4y ago
  I strongly disagree with this statement. A good player can usually walk you through the portion of the game tree that they expanded in their head, but the moves they had to think about (expand their mental game tree to include the resulting position) versus the moves they instinctively ruled out is basically inexplicable, and where a lot of the skill lies. (I'm thinking more chess than othello, since I'm much better at the former, but I assume the thought process is similar)
  - sour-taste 4y ago
    Strongly agree. I'm NOT good at chess, but I still like watching it on YouTube. The explanations that top players give for their moves sound like magic a lot of the time. If they sat down with me for an hour could they explain every step of how they got to their conclusion? I think they could, but I don't think that they'd explain the exact same process they actually went through, which is based on inexplicable heuristic as parent mentioned. Often times at complex decision points in chess players will just say that a move feels better. That feeling can't be explained.
    - shadowofneptune 4y ago
      I have to admit you're right there, now thinking about how I would explain games I play, approaches to tasks, etc.
    - jacquesm 4y ago
      That's not correct. It doesn't matter whether they can explain their moves to you. What matters is if they can explain their moves to another grand master and they always can unless they moved a random piece to break up a position to make a boring game more interesting (this rarely happens, but it is an interesting way to play, but not to win contests).
      - Isinlor 4y ago
        They are not explaining their internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused them to arrive at given moves. They just provide proofs that their moves were good.
squokko 4y ago
A person with a million loyal adherents is worrisome even if those adherents are only sitting on computers. For similar reasons, a person with a trillion unflinchingly loyal AI adherents is worrisome.
- xwolfi 4y ago
  What, why? Unless they vote or are convincing to humans, it's not that worrisome. I mean end of the day, I care about eating well, raising my kids in peace, being useful to others: if an AI tries to convince me to act, it'd have to factor these constraints, in which case it's a useful fellow human able to understand my need and proposing an action for debate. Wars and disasters are usually pursued by minority groups in charge of means of violence, and manipulation is a way for them to trap you into letting a crack open long enough for them to invade it, but if we cant resist an AI, they deserve the Earth more than us :)
stackbutterflow 4y ago
Because the world has had thousand years to evolve, adapt and settle around the capabilities and limitations of the human brain. It is somewhat stable and predictable. Now if a new species of humans suddenly emerged with a brain not only has capable as our but faster, more adaptable and extensible, it would disrupt the world.
- belugacat 4y ago
  Ecosystems (what I infer you mean by "the world", since things that are part of the world such as gravity, plate tectonics, etc don't follow the laws of evolution) haven't had thousands of years to evolve, adapt, and settle around the capabilities that we've had since the industrial revolution started. AI is merely an aspect and consequence of the logical progression of that continuity. (In terms of tangible effect on the world today and what seems on track for the few decades to come, AI is still far behind coal - to pick one - when it comes to concrete negative externalities tho)
  - gjm11 4y ago
    I think something is wrong with the logic of your first paragraph. Suppose (by way of analogy; this is not meant to be directly addressing AI) that in the next year some ingenious physicists finally get quantum gravity figured out, and that soon after the papers are published someone notices that the theory implies that one can use common household materials to make a device capable of destroying a planet. (Seems unlikely, but I don't think anyone would have predicted that the last revolutionary theory of gravity we worked out would imply that one can use not-so-common materials to make a reasonably small device capable of destroying a city, but it turns out it does.) It seems fairly likely that (1) the knowledge of how to do this could not be suppressed for ever, and that (2) if making the device turns out not to need all the fancy apparatus and hard-to-get materials that e.g. hydrogen bombs require, it wouldn't be that long before some idiot actually does it and destroys the earth. And yet, these new discoveries would be "merely an aspect and consequence of the logical progression of that continuity", as you put it. So, whatever the actual situation is with AI, I think it is demonstrably not the case that we can be confident AI won't somehow kill us all or destroy the world merely because AI is a thing we made and the world has had thousands of years to adapt to us. Maybe everything will be fine, maybe not, but figuring out which will require looking in more detail rather than handwaving about how ecosystems always adapt.
mihaic 4y ago
We are forced to work with the imperfections of the human mind, and a great deal of effort in our world is put into working with them. Why would we want more of this?
- taylorius 4y ago
  You make it sound like the human brain is a shoddy piece of work. In any case, even if it is imperfect, it isn't imperfect because it is a black box.
  - mihaic 4y ago
    We sometimes make life-changing decisions based on where Jupiter was when we were born. I mean, we've been struggling for milenia against our biases. I'd let some time pass until we can decide on what these ML models can get subtly wrong.
bigbluedots 4y ago
So that if it starts killing people at some point, you can debug it or maybe figure out why it has done that.
meindnoch 4y ago
We're all driving one, so at least we can extrapolate from our own experience.
dweinus 4y ago
Because, as a human, I can introspect my world model. I can explain it to myself and others. I can explain how it guides my decisions. I can directly update it without needing "training data". I don't have a map of my neurons, but I have self awareness. For LLMs, there is the opposite: transparent structure but opaque understanding.
d1sxeyes 4y ago
Computers have historically been 'correct' about things. Obviously, humans who program computers make mistakes, and occasionally, things like cosmic bit flips can occur, leading a computer to produce an incorrect answer. An LLM upends that paradigm, and worse, the closer it comes to appearing to reason and discuss, the more it implies that the human brain is nothing more than a biological computer, which is worrying for a lot of people. If humans are nothing more than biological computers, do (or can) silicon computers have souls? If a collection of diodes can have a soul, what makes us special?

Isn’t our brain a black box too? Why exactly is it worrisome?

tobr 4y ago

Because human brains have been known to intentionally lie and manipulate, and secretly plan horrible deeds.

eternalban 4y ago
We'll need psych tests for our AI. Malicious AI will note this and adapt. An arms races ensues, where one adversary finances and invests in upgrading the other. Guess who wins?

shadowofneptune 4y ago

The mind itself, perhaps. The person in possession of it though usually does have a conscious strategy to play a game once they are proficient, and can happily explain it to you.

pyinstallwoes 4y ago
How many people can actually explain their state of mind and rational or seemingly rational thought process?
- space_fountain 4y ago
  And we have all sorts of social science these days implying the explanations when people do try to explain themselves they’re justifying decisions that came out of black box rather than actual describing how the decision happened
  - nequo 4y ago
    And also neuroscience examining whether what we think we did out of choice was really done out of our choice: https://en.wikipedia.org/wiki/Neuroscience_of_free_will https://en.wikipedia.org/wiki/Neuroscience_of_free_will
    - taneq 4y ago
      What do you (or they) think “our choice” is if not the result of these processes? It’s not like “you” are just the part of your brain that is consciously aware of making decisions.
- Tepix 4y ago
  I play Othello. In most situations i can give you detailed reasons why i choose to make a particular move.
  - taneq 4y ago
    Are they actually the reasons for your move, or are they post-hoc justifications, and how would you know?
  - Isinlor 4y ago
    You are not explaining your internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused you to arrive at given moves. You will only provide proofs that your moves are good.
  - bobbruno 4y ago
    You can justify post facto the move you made. Can you write down the logical process (probably a combination of strategy, move generation, analysis and selection) that would replicate the moves you picked from the beginning of the game? I don’t even need the brain chemistry (the implementation), but I don’t think even writing down the logic that generates those moves (in contrast with a logic to justify each move after it was generated) is something humans can do in most cases.
- bsenftner 4y ago
  Asking people to explain themselves often triggers various sorts of youth punishment memory, resulting in them being both uncomfortable probing their own reasoning, and then they produce what is essentially a comforting deception for their reasoning simple for the sake of ending the agony of self probing. Asking people their internal states requires their internal guard and internal (potentially filled with self deception) voice what they are up to is very similar to asking children why they are misbehaving, and all manners of mental protections activate.
gpm 4y ago
I strongly disagree with this statement. A good player can usually walk you through the portion of the game tree that they expanded in their head, but the moves they had to think about (expand their mental game tree to include the resulting position) versus the moves they instinctively ruled out is basically inexplicable, and where a lot of the skill lies. (I'm thinking more chess than othello, since I'm much better at the former, but I assume the thought process is similar)
- sour-taste 4y ago
  Strongly agree. I'm NOT good at chess, but I still like watching it on YouTube. The explanations that top players give for their moves sound like magic a lot of the time. If they sat down with me for an hour could they explain every step of how they got to their conclusion? I think they could, but I don't think that they'd explain the exact same process they actually went through, which is based on inexplicable heuristic as parent mentioned. Often times at complex decision points in chess players will just say that a move feels better. That feeling can't be explained.
  - shadowofneptune 4y ago
    I have to admit you're right there, now thinking about how I would explain games I play, approaches to tasks, etc.
  - jacquesm 4y ago
    That's not correct. It doesn't matter whether they can explain their moves to you. What matters is if they can explain their moves to another grand master and they always can unless they moved a random piece to break up a position to make a boring game more interesting (this rarely happens, but it is an interesting way to play, but not to win contests).
    - Isinlor 4y ago
      They are not explaining their internal processes (neurobiology, brain chemistry, neuron firing patterns etc.) that caused them to arrive at given moves. They just provide proofs that their moves were good.

squokko 4y ago

A person with a million loyal adherents is worrisome even if those adherents are only sitting on computers. For similar reasons, a person with a trillion unflinchingly loyal AI adherents is worrisome.

xwolfi 4y ago
What, why? Unless they vote or are convincing to humans, it's not that worrisome. I mean end of the day, I care about eating well, raising my kids in peace, being useful to others: if an AI tries to convince me to act, it'd have to factor these constraints, in which case it's a useful fellow human able to understand my need and proposing an action for debate. Wars and disasters are usually pursued by minority groups in charge of means of violence, and manipulation is a way for them to trap you into letting a crack open long enough for them to invade it, but if we cant resist an AI, they deserve the Earth more than us :)

stackbutterflow 4y ago

Because the world has had thousand years to evolve, adapt and settle around the capabilities and limitations of the human brain. It is somewhat stable and predictable. Now if a new species of humans suddenly emerged with a brain not only has capable as our but faster, more adaptable and extensible, it would disrupt the world.

belugacat 4y ago
Ecosystems (what I infer you mean by "the world", since things that are part of the world such as gravity, plate tectonics, etc don't follow the laws of evolution) haven't had thousands of years to evolve, adapt, and settle around the capabilities that we've had since the industrial revolution started. AI is merely an aspect and consequence of the logical progression of that continuity. (In terms of tangible effect on the world today and what seems on track for the few decades to come, AI is still far behind coal - to pick one - when it comes to concrete negative externalities tho)
- gjm11 4y ago
  I think something is wrong with the logic of your first paragraph. Suppose (by way of analogy; this is not meant to be directly addressing AI) that in the next year some ingenious physicists finally get quantum gravity figured out, and that soon after the papers are published someone notices that the theory implies that one can use common household materials to make a device capable of destroying a planet. (Seems unlikely, but I don't think anyone would have predicted that the last revolutionary theory of gravity we worked out would imply that one can use not-so-common materials to make a reasonably small device capable of destroying a city, but it turns out it does.) It seems fairly likely that (1) the knowledge of how to do this could not be suppressed for ever, and that (2) if making the device turns out not to need all the fancy apparatus and hard-to-get materials that e.g. hydrogen bombs require, it wouldn't be that long before some idiot actually does it and destroys the earth. And yet, these new discoveries would be "merely an aspect and consequence of the logical progression of that continuity", as you put it. So, whatever the actual situation is with AI, I think it is demonstrably not the case that we can be confident AI won't somehow kill us all or destroy the world merely because AI is a thing we made and the world has had thousands of years to adapt to us. Maybe everything will be fine, maybe not, but figuring out which will require looking in more detail rather than handwaving about how ecosystems always adapt.

mihaic 4y ago

We are forced to work with the imperfections of the human mind, and a great deal of effort in our world is put into working with them. Why would we want more of this?

taylorius 4y ago
You make it sound like the human brain is a shoddy piece of work. In any case, even if it is imperfect, it isn't imperfect because it is a black box.
- mihaic 4y ago
  We sometimes make life-changing decisions based on where Jupiter was when we were born. I mean, we've been struggling for milenia against our biases. I'd let some time pass until we can decide on what these ML models can get subtly wrong.

bigbluedots 4y ago

So that if it starts killing people at some point, you can debug it or maybe figure out why it has done that.

meindnoch 4y ago

We're all driving one, so at least we can extrapolate from our own experience.

dweinus 4y ago

Because, as a human, I can introspect my world model. I can explain it to myself and others. I can explain how it guides my decisions. I can directly update it without needing "training data". I don't have a map of my neurons, but I have self awareness. For LLMs, there is the opposite: transparent structure but opaque understanding.

d1sxeyes 4y ago

Computers have historically been 'correct' about things. Obviously, humans who program computers make mistakes, and occasionally, things like cosmic bit flips can occur, leading a computer to produce an incorrect answer. An LLM upends that paradigm, and worse, the closer it comes to appearing to reason and discuss, the more it implies that the human brain is nothing more than a biological computer, which is worrying for a lot of people. If humans are nothing more than biological computers, do (or can) silicon computers have souls? If a collection of diodes can have a soul, what makes us special?