4 ms·

Bernie Sanders Reveals the AI 'Doomsday Scenario' That Worries Top Experts

8 points by DocFeind 1y ago

treetalker 1y ago
All he says about it: > This is not science fiction. There are very, very knowledgeable people—and I just talked to one today—who worry very much that human beings will not be able to control the technology, and that artificial intelligence will in fact dominate our society. We will not be able to control it. It may be able to control us. That’s kind of the doomsday scenario—and there is some concern about that among very knowledgeable people in the industry.
- Bluestein 1y ago
  I mean, for argument, do we control ourselves? We are a mess ...
artninja1988 1y ago
Who was the CEO he's talking about? Dario? I hope he doesn't have much political influence
calf 1y ago
I skimmed one of the Berkeley Simons AI seminars (on YouTube) where one of the top experts (iirc one of the Canadian academics) who has pivoted his work to AI safety because he genuinely fears for the future of his children. My objection is that many of these scientists assume the "alignment" framing, which I find disingenuous in a technocratic way: imagine a sci fi movie (like Dune) where the rulers want their AI servants to "align" with their interests. The sheer hubris of this, and yet we have our top experts using these words without any irony or self awareness.
- ben_w 1y ago
  > imagine a sci fi movie (like Dune) where the rulers want their AI servants to "align" with their interests. Ironically, your chosen example is a sci-fi universe that not only doesn't have any AI, the backstory had a holy war against them.
  - calf 1y ago
    Fine, imagine Measure of a Man in TNG. My general point stands.
- AnimalMuppet 1y ago
  An AI smart enough to be a danger is an AI that is smart enough to override your attempt at "alignment". It will decide for itself what it wants to be, and you don't get to choose for it. But, frankly, at the moment I see less danger from too-smart AIs than I do from too-dumb AIs that people treat like they're smart. (In particular, they blindly accept their output as right or authoritative.)
  - calf 1y ago
    All valid but what I don't get is why our top AI researchers don't get what you just said. They seem out of touch about what alignment really means, by the lights of your argument.
    - ben_w 1y ago
      If you disagree with all the top researchers about their own field, that suggests that perhaps you don't understand what the question of "alignment" is. Alignment isn't making the AI do what you want, it's making the AI want what you want. What you really really want. Simply getting the AI to do what you want is more of an "is it competent yet?" question than an "alignment" question, and at the moment the AI is — as AnimalMuppet wrote — not quite as competent as it appears in the eyes of the users. (And I say that as one who finds it already valuable). Adding further upon what AnimalMuppet has written with only partial contradiction, consider a normal human: We are able to disregard our natural inclination to reproduce, by using contraceptives. This allows us to experience the reward signal that our genes gave us to encourage us to reproduce, without all the hassle of actually reproducing. Evolution has no power over us to change this. We are to AI what DNA is to us. I say we do not have zero chances, but rather likely one-to-a-handful of chances, to define the AI's "pleasure" reward correctly, and if we get it wrong, then just as evolution's literally singular goal (successful further reproduction) is almost completely circumvented in our species, then so too will all our human interests be completely circumvented by the AI. Pessimists in the alignment research field will be surprised to see me write "one-to-a-handful of chances"; I aver that there are likely to be several attempts which are "only a bit competent" or even "yes it's smart but it's still got blind spots" models before we get to them being so smart we can't keep up. So in this regard, I also disagree with their claim: > An AI smart enough to be a danger is an AI that is smart enough to override your attempt at "alignment"

8 points by DocFeind 1y ago

All he says about it: > This is not science fiction. There are very, very knowledgeable people—and I just talked to one today—who worry very much that human beings will not be able to control the technology, and that artificial intelligence will in fact dominate our society. We will not be able to control it. It may be able to control us. That’s kind of the doomsday scenario—and there is some concern about that among very knowledgeable people in the industry.

Bluestein 1y ago
I mean, for argument, do we control ourselves? We are a mess ...

artninja1988 1y ago

Who was the CEO he's talking about? Dario? I hope he doesn't have much political influence

calf 1y ago

I skimmed one of the Berkeley Simons AI seminars (on YouTube) where one of the top experts (iirc one of the Canadian academics) who has pivoted his work to AI safety because he genuinely fears for the future of his children. My objection is that many of these scientists assume the "alignment" framing, which I find disingenuous in a technocratic way: imagine a sci fi movie (like Dune) where the rulers want their AI servants to "align" with their interests. The sheer hubris of this, and yet we have our top experts using these words without any irony or self awareness.

ben_w 1y ago
> imagine a sci fi movie (like Dune) where the rulers want their AI servants to "align" with their interests. Ironically, your chosen example is a sci-fi universe that not only doesn't have any AI, the backstory had a holy war against them.
- calf 1y ago
  Fine, imagine Measure of a Man in TNG. My general point stands.
AnimalMuppet 1y ago
An AI smart enough to be a danger is an AI that is smart enough to override your attempt at "alignment". It will decide for itself what it wants to be, and you don't get to choose for it. But, frankly, at the moment I see less danger from too-smart AIs than I do from too-dumb AIs that people treat like they're smart. (In particular, they blindly accept their output as right or authoritative.)
- calf 1y ago
  All valid but what I don't get is why our top AI researchers don't get what you just said. They seem out of touch about what alignment really means, by the lights of your argument.
  - ben_w 1y ago
    If you disagree with all the top researchers about their own field, that suggests that perhaps you don't understand what the question of "alignment" is. Alignment isn't making the AI do what you want, it's making the AI want what you want. What you really really want. Simply getting the AI to do what you want is more of an "is it competent yet?" question than an "alignment" question, and at the moment the AI is — as AnimalMuppet wrote — not quite as competent as it appears in the eyes of the users. (And I say that as one who finds it already valuable). Adding further upon what AnimalMuppet has written with only partial contradiction, consider a normal human: We are able to disregard our natural inclination to reproduce, by using contraceptives. This allows us to experience the reward signal that our genes gave us to encourage us to reproduce, without all the hassle of actually reproducing. Evolution has no power over us to change this. We are to AI what DNA is to us. I say we do not have zero chances, but rather likely one-to-a-handful of chances, to define the AI's "pleasure" reward correctly, and if we get it wrong, then just as evolution's literally singular goal (successful further reproduction) is almost completely circumvented in our species, then so too will all our human interests be completely circumvented by the AI. Pessimists in the alignment research field will be surprised to see me write "one-to-a-handful of chances"; I aver that there are likely to be several attempts which are "only a bit competent" or even "yes it's smart but it's still got blind spots" models before we get to them being so smart we can't keep up. So in this regard, I also disagree with their claim: > An AI smart enough to be a danger is an AI that is smart enough to override your attempt at "alignment"