3 ms·

Zvi's an AI-doomer, so he sees the endgame as AI becoming smart enough that they don't need us anymore and then killing us to take our stuff, but your scenario

Zvi's an AI-doomer, so he sees the endgame as AI becoming smart enough that they don't need us anymore and then killing us to take our stuff, but your scenario is also pretty bad.

Terr_ 1y ago
Speaking in generalities, some source of "doom" are really covert marketing, where Thing X is so amazing and magical that nobody can risk not investing in it, paying attention to it, or constantly talking with their friends about it.
- oceanofsolaris 1y ago
  I don’t think you can really accuse the lesswrong crowd of being in it to hype up OpenAI. They don’t profit from it (well at least the current doom crowd), they have been saying that this is super-risky since “forever” (15 years), they strongly argue against e.g. OpenAI going private … I really don’t understand where this whole strand of thinking of hidden ulteriour motives for AI doomers comes from, AFAIK there was never a single clear case where this happened (and the arc of people invested in AI is they they become less of a doomer the more financially entangled they become with AI, see Elon or Altman).
  - reducesuffering 1y ago
    It's so common for people to throw out those accusations, despite zero evidence from the lesswrong crowd, because they need explanations that are way more comforting than facing down the scary implication that lesswrong "doomers" are right. One thought path leads to business-as-usual comfort, happily motivated by trillions of $ in market cap and VC. The other is dread. It makes sense why people take the easy road. We're still battling climate change denialism after all.
  - jononor 1y ago
    They could be be playing into such a narrative, even though they have different motivations. Certainly Altman has had a few moments where he is saying "this stuff is getting so good it is society-scale dangerous". His motivations are to create FOMO with investors and users, and to try to form the regulatory landscape in their favor.
    - SpicyLemonZest 1y ago
      I just don't understand the line of reasoning at all. It sounds to me like postulating that oil executives have started admitting climate change is real in order to create FOMO. Investors and users live in society, why would they fear missing out on destroying it?
      - Terr_ 1y ago
        > like postulating that oil executives have started admitting climate change is real The subtext isn't "our product has bad side-effects and can be avoided with known alternatives", but more like: "Our product's core benefit is too awesome at what it does; If we don't pursue it then someone less-enlightened will do it anyway; Whether you love it or hate it the only way to influence the outcome is to support us." So the oil-executive version would be something like "worrying" that petrochemical success will quickly bankrupt every other power-generation or energy-storage system overnight, causing economic disruption, and that eventually it will make mankind too satisfied and too comfortable so that it falls into existential torpor... But at least DinoCo™ has a What If It's Too Awesome working group to study the issue while there's still time before our inevitable corporate win of All The Things.
        SpicyLemonZest 1y ago
        Why would the oil executive benefit from saying that if it's not true? Wouldn't it be better for his stock price to say that disruption will be limited, and the DinoCo takeover will usher in a golden age like Amazon has for shopping? If anything, that's my criticism of the guy; as OpenAI revenue has risen, he's become much more vocal about the benefits of AI and much more confident that the risks and costs are solvable.
        jononor 1y ago
        If enough people believe it is true and act on this belief, it will become true (to a larger degree, at least). There is a massive shift of the direction of money happening right now, away from workers into "AI" (LLM based systems). The more of that shift can be produced now, the stronger the AI companies will be in the future. Because they need massive capital investments in compute, engineering investments in efficiency, and to reach economies of scale, to be able to really compete with human-based workforce.
  - Terr_ 1y ago
    I'm hoping that's intended as: "That group exists, but for clarity I want to say the lesswrong crowd aren't part of it", as opposed to my initial reading of: "Your 'generality' is a false covert accusation, you cad."

Zvi's an AI-doomer, so he sees the endgame as AI becoming smart enough that they don't need us anymore and then killing us to take our stuff, but your scenario is also pretty bad.

Terr_ 1y ago

Speaking in generalities, some source of "doom" are really covert marketing, where Thing X is so amazing and magical that nobody can risk not investing in it, paying attention to it, or constantly talking with their friends about it.

oceanofsolaris 1y ago
I don’t think you can really accuse the lesswrong crowd of being in it to hype up OpenAI. They don’t profit from it (well at least the current doom crowd), they have been saying that this is super-risky since “forever” (15 years), they strongly argue against e.g. OpenAI going private … I really don’t understand where this whole strand of thinking of hidden ulteriour motives for AI doomers comes from, AFAIK there was never a single clear case where this happened (and the arc of people invested in AI is they they become less of a doomer the more financially entangled they become with AI, see Elon or Altman).
- reducesuffering 1y ago
  It's so common for people to throw out those accusations, despite zero evidence from the lesswrong crowd, because they need explanations that are way more comforting than facing down the scary implication that lesswrong "doomers" are right. One thought path leads to business-as-usual comfort, happily motivated by trillions of $ in market cap and VC. The other is dread. It makes sense why people take the easy road. We're still battling climate change denialism after all.
- jononor 1y ago
  They could be be playing into such a narrative, even though they have different motivations. Certainly Altman has had a few moments where he is saying "this stuff is getting so good it is society-scale dangerous". His motivations are to create FOMO with investors and users, and to try to form the regulatory landscape in their favor.
  - SpicyLemonZest 1y ago
    I just don't understand the line of reasoning at all. It sounds to me like postulating that oil executives have started admitting climate change is real in order to create FOMO. Investors and users live in society, why would they fear missing out on destroying it?
    - Terr_ 1y ago
      > like postulating that oil executives have started admitting climate change is real The subtext isn't "our product has bad side-effects and can be avoided with known alternatives", but more like: "Our product's core benefit is too awesome at what it does; If we don't pursue it then someone less-enlightened will do it anyway; Whether you love it or hate it the only way to influence the outcome is to support us." So the oil-executive version would be something like "worrying" that petrochemical success will quickly bankrupt every other power-generation or energy-storage system overnight, causing economic disruption, and that eventually it will make mankind too satisfied and too comfortable so that it falls into existential torpor... But at least DinoCo™ has a What If It's Too Awesome working group to study the issue while there's still time before our inevitable corporate win of All The Things.
      - SpicyLemonZest 1y ago
        Why would the oil executive benefit from saying that if it's not true? Wouldn't it be better for his stock price to say that disruption will be limited, and the DinoCo takeover will usher in a golden age like Amazon has for shopping? If anything, that's my criticism of the guy; as OpenAI revenue has risen, he's become much more vocal about the benefits of AI and much more confident that the risks and costs are solvable.
        jononor 1y ago
        If enough people believe it is true and act on this belief, it will become true (to a larger degree, at least). There is a massive shift of the direction of money happening right now, away from workers into "AI" (LLM based systems). The more of that shift can be produced now, the stronger the AI companies will be in the future. Because they need massive capital investments in compute, engineering investments in efficiency, and to reach economies of scale, to be able to really compete with human-based workforce.
- Terr_ 1y ago
  I'm hoping that's intended as: "That group exists, but for clarity I want to say the lesswrong crowd aren't part of it", as opposed to my initial reading of: "Your 'generality' is a false covert accusation, you cad."