6 ms·

I wonder if he got the idea from Philip K. Dick's 1955 story "Autofac"? https://en.wikipedia.org/wiki/Autofac https://en.wikipedia.org/wiki/Autofac https://ww

I wonder if he got the idea from Philip K. Dick's 1955 story "Autofac"? https://en.wikipedia.org/wiki/Autofac https://en.wikipedia.org/wiki/Autofac https://www.vulture.com/2018/01/electric-dreams-recap-season-1-episode-2-autofac.html https://www.vulture.com/2018/01/electric-dreams-recap-season...

actionfromafar 3y ago
PKD keeps surprising me.
- dylan604 3y ago
  I like how the sci-fi authors have spent time thinking about what an advanced AI could do, yet those building the AI have not taken a moment's pause to consider what they are doing
  - d1str0 3y ago
    Do you honestly believe that the only people who have thought about the ramifications of AI are Sci Fi authors? I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. This isn’t Jurassic Park.
    - chess_buster 3y ago
      They have – and decided to do it anyway.
    - actionfromafar 3y ago
      Haha, but it is! :-D LLMs are pretty basic stuff but we are all struggling with what to use them for! OpenAI is manually playing whac-a-mole with ChatGPT saying the darndest things!
    - otachack 3y ago
      Depends what we're looking at. Ride share disruption was very Jurassic Park and we've been dealing with ramifications ever since.
    - AlotOfReading 3y ago
      If you accept the implied premise that there are irresponsible deployments of AI out there, the alternative explanation is that they did consider the ramifications and simply don't care. That's even worse. Calling them ignorant is actually giving them the benefit of the doubt.
      - goatlover 3y ago
        Or the researchers don't think existential threats are realistic, and paper maximizing thought experiments are silly. Maybe they're wrong, but maybe not. It's easy to imagine AI takeover scenarios by giving them unlimited powers, it's hard to show the actual path to such abilities. It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. So while alignment remains an issue, the existential alignment threats are too ill-specified. AGIs would understand we don't want to paperclip the world. Fun game though.
        AlotOfReading 3y ago
        That would be not accepting the premise that deployments are irresponsible. I guess there could be a situation where every researcher thinks everyone else's deployment is irresponsible and theirs is fine, but I don't think that's what you're saying.
        cubefox 3y ago
        > It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. Terminal goals can't be futile, since they do not serve to achieve other (instrumental) goals. Compare: Humans like to have protected sex, watch movies, eat ice cream, even though these activities might be called "futile" or "useless" (by someone who doesn't have those goals) as they don't serve any further purpose. But criticizing terminal goals for not being instrumentally useful is a category error. For a paperclipper, us having sex would seem just as futile as creating paperclips seems to us. Increased intelligence won't let you abandon any of your terminal goals, since they do not depend on your intelligence, unlike instrumental goals.
    - throwaway290 3y ago
      > I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. Super easy to not think about something if your job depends on it. And even if you do, things don't go as you think (see bombings of civilian Hiroshima and Nagasaki despite objections of nuclear physicists).
    - mitthrowaway2 3y ago
      Some of us considered it and even decided to go into different fields as a result. Some others entered the field, made progress, and apparently regretted it. Others are willing to put their concerns aside for money. Salaries get very high in that field.
    - killerstorm 3y ago
      That's not how it works. People publish papers demonstrating improvements without thinking about "ramifications".
  - JRandomHacker42 3y ago
    https://twitter.com/AlexBlechman/status/1457842724128833538 https://twitter.com/AlexBlechman/status/1457842724128833538 Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
  - willcipriano 3y ago
    The people building it know it isn't AI, the people selling it call it that.
    - Karunamon 3y ago
      To say that something like GPT-4 does not count as "AI" requires a gargantuan shifting of goalposts from where they were at this time last year.
      - actionfromafar 3y ago
        So it was ever since computers started playing chess.
  - feoren 3y ago
    That's because the people who are building the AI actually know how it works, understand how fundamentally simple it all is, and know that there's no room for consciousness to magically emerge. The current state of AI is not so much a story of any kind of "intelligence" being amazing, but rather the sum total of humanity's data being amazing. The amazing feats LLMs perform come from the words we all wrote, not the code they wrote. The code just unlocks the previously latent power of all that data. It is nothing close to being an actual intelligence, regardless of how much we anthropomorphize it. We also anthropomorphize stick figures, stuffed animals, and weighted companion cubes.
    - bananapub 3y ago
      that's indeed not a sensible worry, but the actual consequences on society of such things are extremely real and already happening, and are something the people involved seem either uninterested in worrying about or actively encouraging.
    - flangola7 3y ago
      You've written a lot of words yet I don't see a compelling reason to believe a single one of them.
    - actionfromafar 3y ago
      Good thing no new technology ever caused anything bad thanks to us not anthropomorphising it.
      - deleted 3y ago
        [deleted]
abecedarius 3y ago
He got it from Eliezer Yudkowsky's somewhat different paperclip maximizer in a mailing-list post. (My memory of a Twitter thread a while back which included Yudkowsky, saying he'd told Bostrom not to worry about attributing ideas like that to him.)
- deleted 3y ago
  [deleted]

actionfromafar 3y ago

PKD keeps surprising me.

dylan604 3y ago
I like how the sci-fi authors have spent time thinking about what an advanced AI could do, yet those building the AI have not taken a moment's pause to consider what they are doing
- d1str0 3y ago
  Do you honestly believe that the only people who have thought about the ramifications of AI are Sci Fi authors? I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. This isn’t Jurassic Park.
  - chess_buster 3y ago
    They have – and decided to do it anyway.
  - actionfromafar 3y ago
    Haha, but it is! :-D LLMs are pretty basic stuff but we are all struggling with what to use them for! OpenAI is manually playing whac-a-mole with ChatGPT saying the darndest things!
  - otachack 3y ago
    Depends what we're looking at. Ride share disruption was very Jurassic Park and we've been dealing with ramifications ever since.
  - AlotOfReading 3y ago
    If you accept the implied premise that there are irresponsible deployments of AI out there, the alternative explanation is that they did consider the ramifications and simply don't care. That's even worse. Calling them ignorant is actually giving them the benefit of the doubt.
    - goatlover 3y ago
      Or the researchers don't think existential threats are realistic, and paper maximizing thought experiments are silly. Maybe they're wrong, but maybe not. It's easy to imagine AI takeover scenarios by giving them unlimited powers, it's hard to show the actual path to such abilities. It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. So while alignment remains an issue, the existential alignment threats are too ill-specified. AGIs would understand we don't want to paperclip the world. Fun game though.
      - AlotOfReading 3y ago
        That would be not accepting the premise that deployments are irresponsible. I guess there could be a situation where every researcher thinks everyone else's deployment is irresponsible and theirs is fine, but I don't think that's what you're saying.
      - cubefox 3y ago
        > It's also hard to understand why an AI smart enough to paperclip the world wouldn't also be smart enough to realize the futility in doing so. Terminal goals can't be futile, since they do not serve to achieve other (instrumental) goals. Compare: Humans like to have protected sex, watch movies, eat ice cream, even though these activities might be called "futile" or "useless" (by someone who doesn't have those goals) as they don't serve any further purpose. But criticizing terminal goals for not being instrumentally useful is a category error. For a paperclipper, us having sex would seem just as futile as creating paperclips seems to us. Increased intelligence won't let you abandon any of your terminal goals, since they do not depend on your intelligence, unlike instrumental goals.
  - throwaway290 3y ago
    > I can guarantee people who spend years researching and building advanced language models have thought about the ramifications of their work. Super easy to not think about something if your job depends on it. And even if you do, things don't go as you think (see bombings of civilian Hiroshima and Nagasaki despite objections of nuclear physicists).
  - mitthrowaway2 3y ago
    Some of us considered it and even decided to go into different fields as a result. Some others entered the field, made progress, and apparently regretted it. Others are willing to put their concerns aside for money. Salaries get very high in that field.
  - killerstorm 3y ago
    That's not how it works. People publish papers demonstrating improvements without thinking about "ramifications".
- JRandomHacker42 3y ago
  https://twitter.com/AlexBlechman/status/1457842724128833538 https://twitter.com/AlexBlechman/status/1457842724128833538 Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
- willcipriano 3y ago
  The people building it know it isn't AI, the people selling it call it that.
  - Karunamon 3y ago
    To say that something like GPT-4 does not count as "AI" requires a gargantuan shifting of goalposts from where they were at this time last year.
    - actionfromafar 3y ago
      So it was ever since computers started playing chess.
- feoren 3y ago
  That's because the people who are building the AI actually know how it works, understand how fundamentally simple it all is, and know that there's no room for consciousness to magically emerge. The current state of AI is not so much a story of any kind of "intelligence" being amazing, but rather the sum total of humanity's data being amazing. The amazing feats LLMs perform come from the words we all wrote, not the code they wrote. The code just unlocks the previously latent power of all that data. It is nothing close to being an actual intelligence, regardless of how much we anthropomorphize it. We also anthropomorphize stick figures, stuffed animals, and weighted companion cubes.
  - bananapub 3y ago
    that's indeed not a sensible worry, but the actual consequences on society of such things are extremely real and already happening, and are something the people involved seem either uninterested in worrying about or actively encouraging.
  - flangola7 3y ago
    You've written a lot of words yet I don't see a compelling reason to believe a single one of them.
  - actionfromafar 3y ago
    Good thing no new technology ever caused anything bad thanks to us not anthropomorphising it.
    - deleted 3y ago
      [deleted]

abecedarius 3y ago

He got it from Eliezer Yudkowsky's somewhat different paperclip maximizer in a mailing-list post. (My memory of a Twitter thread a while back which included Yudkowsky, saying he'd told Bostrom not to worry about attributing ideas like that to him.)

deleted 3y ago
[deleted]