5 ms·

If AI has no economic impact, it almost by definition also has no ability to marshal the kind of resources necessary to destroy all of humanity. All the doom s

If AI has no economic impact, it almost by definition also has no ability to marshal the kind of resources necessary to destroy all of humanity. All the doom scenarios seem to rely on quasi-magical abilities springing from advanced intelligence, but ignore that these scenarios require the AI to have access to significant resources. Then, suppose AI does somehow think itself into effecting human annihilation, what happens when all the humans are dead? At the current state of things, all electricity production on the planet goes dark in maybe two weeks. Nuke plants irradiate significant parts of the earth, chip fabs are bricked, metal production is toast. In short the AI necessarily dies too. The only way for this not to happen is AI controlled robots first gaining complete control of massive chunks of the economy. If AI is as useless as the article claims then that will effectively never happen. Thus, AI doom scenarios are always a murder suicide situation, and I’m not sure I believe anyone who says a superhumanly capable planner would pursue that plan. It even negates the “paperclip optimizer” as the AI’s demise obviously puts a hard upper limit on the number of paperclips produced.

tivert 2y ago
> All the doom scenarios seem to rely on quasi-magical abilities springing from advanced intelligence, but ignore that these scenarios require the AI to have access to significant resources. No. The doom scenario I'm most concerned with is basically the one from Manna (https://marshallbrain.com/manna https://marshallbrain.com/manna), which doesn't require any "quasi-magical" abilities. It's basically taking our current economic system and "fixing the glitch" (from the capitalist's perspective) that forces them to pay our salaries. That's (a slow) doom for the vast majority of the population, but not for humanity because the wealthy elite would survive.
- hyperpape 2y ago
  Anything that removes the need to pay salaries would have massive impacts on productivity numbers. (I am not arguing that would necessarily be good[1], but it would show up in the economic stats, in line with bglazer's point). [1] The big brain take behind "necessarily" is that: "no salaries, everything is free" seems cool, "no salaries, a few people hoard all the wealth" seems bad.
  - tivert 2y ago
    > Anything that removes the need to pay salaries would have massive impacts on productivity numbers. (I am not arguing that would necessarily be good[1], but it would show up in the economic stats, in line with bglazer's point). I agree with that. My point was that not all "doom scenarios" require the kind of "quasi-magical abilities" that he tried to use to dismiss them all. IMHO, the sci-fi "superintelligent AI sends terminators to kill all the people" doom scenarios are distractions (possibly intentional distractions) from far more plausible doom scenarios. > [1] The big brain take behind "necessarily" is that: "no salaries, everything is free" seems cool, "no salaries, a few people hoard all the wealth" seems bad. If the former were plausable, we'd almost be there even without AI, so it's either a fantasy or the work we need to get there is not technological. The latter is basically 2024 but moreso.
    - Terr_ 2y ago
      Or to put another way, AI can be dangerous even when It's being totally helpful... When it is enabling some of humanity to screw things up for everyone else out of greed or anger.
  - tnel77 2y ago
    While I agree with you, have you found anything related to how you would essentially ration goods in a system where AI could provide and everything was free?
- nope1000 2y ago
  But no salaries -> no consumption -> no profit for companies. I don't see how that's effective capitalism.
  - tivert 2y ago
    > But no salaries -> no consumption -> no profit for companies. No, you just need a little more imagination. What would the economy look like if you didn't need employees to utilize resources or undertake projects? Some very mass-market consumer oriented companies would eventually go bankrupt during the transition, but that doesn't mean a doom scenario like I describe can't or won't happen. > I don't see how that's effective capitalism. And you know what? The end result of the transition might only vaguely resemble current-day capitalism, and may be more properly understood as a transition to a new economic system. However, I think it would probably be like our current economy, except with: * the B2C segment withered mainly to luxury goods of elites (yachts, supercars, elite-level fine dining) and sustaining a much smaller elite-service-focused labor pool; * B2B being most of what's left of the market; and * non-market vanity projects taking up a substantial slice of the pie (e.g. Elon Musk using a robot army to build ziggurats to celebrate himself, because why not).
    - tuatoru 2y ago
      This misunderstands the economy. It's driven by household and government spending. All that B2B stuff happens because some company, somewhere at the end of the chain, sells to households or the government. Sure, there is also business investment. But that only happens because someone expects to sell to households or the government. If all you are selling is luxury goods, which depend on direct high labor content for their value, then most of the B2B businesses disappear also. And yes, there are exports. But not at global scale. The globe is closed.
      - tivert 2y ago
        > This misunderstands the economy. It's driven by household and government spending. All that B2B stuff happens because some company, somewhere at the end of the chain, sells to households or the government. No, I didn't misunderstand it. Let me put it in your terms: the remaining elite become tiny but massively consuming households, and the distinctions between the businesses they control and their households will be blurred (e.g. Blue Origin being wholly owned by Bezos). All the economic actors that cannot provide value to these households will get slowly cut off and wither away, and these elites (collectively) will eventually cement total control over all the valuable resources they have some use for (production capacity, compute, energy, raw materials, etc). To go back to a question I asked upthread: what would the economy look like if you didn't need employees to utilize resources or undertake projects? What happens when most households aren't competitive at anything economically, but there's still a lot of stuff out there to be owned? My answer is those who own the right things (e.g. AI-run, totally automated, adaptable factories; energy resources) will be able to support themselves indefinitely with them and have massive personal surpluses they then use for vanity projects to keep themselves occupied.
- 1659447091 2y ago
  > That's (a slow) doom for the vast majority of the population, but not for humanity because the wealthy elite would survive. People throughout history have replaced the "elite" once the cost to do so became less than the cost (or benefit) to continue on in the status quo. The perceived "doom" would not last very long (in the greater span or sense of time) as it would all be reorganized, yet again. In that case, I am not so sure it's a "doom scenario" but instead something like the natural way of things, or history continuing on. And, maybe even a needed evolution of the species in that case. Things would have to be "quasi-magical" to play out (for the rest of human eternity) in the way a lot of the AI doom musings point to.
  - tivert 2y ago
    > People throughout history have replaced the "elite" once the cost to do so became less than the cost (or benefit) to continue on in the status quo. Past performance is not a guarantee of future success. The elite throughout history relied on masses of common people to get anything done (do work, go to war, etc.). AI as-hyped has the potential to upend that long standing dynamic. The elite aren't going to be replaced if they and the resources they care to protect are ensconced in a 50 mile buffer patrolled by (among other things) drone-swarms that will shove grenades in the face of any unauthorized human that dares enter. That technology is pretty close to reality, as it's basically the Ukraine/Russia front lines with 5-10 more years of drone development.
    - 1659447091 2y ago
      > Past performance is not a guarantee of future success. True. I am still of the belief humans will continue to human. If an uprising of billions vs millions lose, the "elite" would also no longer be "elite". While I have no data, other than observation, the "elite", imo, need the validation of their status more than those who have none and gave it to them in the first place. If they do something so brash as to rid the world of these unwashed masses, they have no one left to rise above. I don't see them all of a sudden becoming great members of this new society; they wanted to own society, not be a citizen in it. How that wouldn't lead to self-elimination at that point is hard to imagine - thus a needed evolution of the species (even if that's extinction and another hominid group later inheriting the leftover mess)
y04nn 2y ago
I think one way for an intelligent AI can gain control over humans is not to exterminate them but to control them, influence them without them being aware of it, changing the recommandation algorithms, producing educational material, videos, podcasts, books that would be pro AI while saturating our monkey brain with mining-less entertaining content. We would become their servant without noticing.
- warkdarrior 2y ago
  How would you know this is not the case already?
  - latentsea 2y ago
    It has been since 2016... Dun-dun-dahhh
tuatoru 2y ago
Here is a counterexample: Universal cheap AI girlfriends and boyfriends. These would have a mildly depressing effect on the economy for a couple of generations, and then the population goes to zero.
- latentsea 2y ago
  With birth rates already in decline, population is already going to zero.
  - red-iron-pine 2y ago
    only Western Gringos, and Asians in countries like SK and China. much of the world is still crapping out kids just fine. subsistence farmers feel more confident doing that than programmers, it seems.
    - sharemywin 2y ago
      ROI. only takes 5 years to make a good crop picker. much longer to make a good programmer.
      - latentsea 2y ago
        And once you've got lots of crop pickers you can transition to management. Then you can start a marketplace, then upgrade that to a chain of supermarkets, then upgrade that to a global FMCG conglomerate, and then your birth rates will also decline and population go to zero. It's a sad realization that modern life which we deem the right way to exist is basically just wrong. That which is unsustainable by definition cannot be sustained after all.
TacticalCoder 2y ago
> The only way for this not to happen is AI controlled robots first gaining complete control of massive chunks of the economy. So they do that because they plan to annihilate all humans. They succeed in both. How does the AI deal with physics and the heat death of the universe?
humansareok1 2y ago
> All the doom scenarios seem to rely on quasi-magical abilities springing from advanced intelligence The amount of people who dismiss doom risk with some version of "I'm not smart enough to imagine what something far smarter than me might be capable of" is staggering. "I can't imagine how nanobots might work therefore they must be impossible." >Then, suppose AI does somehow think itself into effecting human annihilation, what happens when all the humans are dead? At the current state of things, all electricity production on the planet goes dark in maybe two weeks. Nuke plants irradiate significant parts of the earth, chip fabs are bricked, metal production is toast. In short the AI necessarily dies too. Smart enough to eradicate humanity but not smart enough to plan ahead to keep itself running. Ok buddy. >The only way for this not to happen is AI controlled robots first gaining complete control of massive chunks of the economy. If AI is as useless as the article claims then that will effectively never happen. "AI hasn't taken over large parts of the economy, therefore it never will." I really just can't even...
- throwanem 2y ago
  Mostly the people dismissing the x-risk concern do so on the grounds that what is being sold is not and cannot become smart in the way it's being sold by x-risk stans as being able to, which is true. If you want to argue the contrary case, starting from the actual point of disagreement is probably a better play, because the argument you've actually made here constitutes no answer.
  - humansareok1 2y ago
    >what is being sold is not and cannot become smart in the way it's being sold by x-risk stans Do you have actual evidence for this astonishing take? Because I'm sure the entire field would be entranced by your genius here. It's actually completely UN-clear what the limits to LLMs are and it's completely ridiculous to say otherwise. >starting from the actual point of disagreement is probably a better play Fundamentally most e/accs or other anti-doom stans believe that AGI/ASI are actually impossible, that nothing can be smarter than humans and that human brains constitute special magic meat. Almost all of their other wrong beliefs stem from those.

tivert 2y ago

> All the doom scenarios seem to rely on quasi-magical abilities springing from advanced intelligence, but ignore that these scenarios require the AI to have access to significant resources. No. The doom scenario I'm most concerned with is basically the one from Manna (https://marshallbrain.com/manna https://marshallbrain.com/manna), which doesn't require any "quasi-magical" abilities. It's basically taking our current economic system and "fixing the glitch" (from the capitalist's perspective) that forces them to pay our salaries. That's (a slow) doom for the vast majority of the population, but not for humanity because the wealthy elite would survive.

hyperpape 2y ago
Anything that removes the need to pay salaries would have massive impacts on productivity numbers. (I am not arguing that would necessarily be good[1], but it would show up in the economic stats, in line with bglazer's point). [1] The big brain take behind "necessarily" is that: "no salaries, everything is free" seems cool, "no salaries, a few people hoard all the wealth" seems bad.
- tivert 2y ago
  > Anything that removes the need to pay salaries would have massive impacts on productivity numbers. (I am not arguing that would necessarily be good[1], but it would show up in the economic stats, in line with bglazer's point). I agree with that. My point was that not all "doom scenarios" require the kind of "quasi-magical abilities" that he tried to use to dismiss them all. IMHO, the sci-fi "superintelligent AI sends terminators to kill all the people" doom scenarios are distractions (possibly intentional distractions) from far more plausible doom scenarios. > [1] The big brain take behind "necessarily" is that: "no salaries, everything is free" seems cool, "no salaries, a few people hoard all the wealth" seems bad. If the former were plausable, we'd almost be there even without AI, so it's either a fantasy or the work we need to get there is not technological. The latter is basically 2024 but moreso.
  - Terr_ 2y ago
    Or to put another way, AI can be dangerous even when It's being totally helpful... When it is enabling some of humanity to screw things up for everyone else out of greed or anger.
- tnel77 2y ago
  While I agree with you, have you found anything related to how you would essentially ration goods in a system where AI could provide and everything was free?
nope1000 2y ago
But no salaries -> no consumption -> no profit for companies. I don't see how that's effective capitalism.
- tivert 2y ago
  > But no salaries -> no consumption -> no profit for companies. No, you just need a little more imagination. What would the economy look like if you didn't need employees to utilize resources or undertake projects? Some very mass-market consumer oriented companies would eventually go bankrupt during the transition, but that doesn't mean a doom scenario like I describe can't or won't happen. > I don't see how that's effective capitalism. And you know what? The end result of the transition might only vaguely resemble current-day capitalism, and may be more properly understood as a transition to a new economic system. However, I think it would probably be like our current economy, except with: * the B2C segment withered mainly to luxury goods of elites (yachts, supercars, elite-level fine dining) and sustaining a much smaller elite-service-focused labor pool; * B2B being most of what's left of the market; and * non-market vanity projects taking up a substantial slice of the pie (e.g. Elon Musk using a robot army to build ziggurats to celebrate himself, because why not).
  - tuatoru 2y ago
    This misunderstands the economy. It's driven by household and government spending. All that B2B stuff happens because some company, somewhere at the end of the chain, sells to households or the government. Sure, there is also business investment. But that only happens because someone expects to sell to households or the government. If all you are selling is luxury goods, which depend on direct high labor content for their value, then most of the B2B businesses disappear also. And yes, there are exports. But not at global scale. The globe is closed.
    - tivert 2y ago
      > This misunderstands the economy. It's driven by household and government spending. All that B2B stuff happens because some company, somewhere at the end of the chain, sells to households or the government. No, I didn't misunderstand it. Let me put it in your terms: the remaining elite become tiny but massively consuming households, and the distinctions between the businesses they control and their households will be blurred (e.g. Blue Origin being wholly owned by Bezos). All the economic actors that cannot provide value to these households will get slowly cut off and wither away, and these elites (collectively) will eventually cement total control over all the valuable resources they have some use for (production capacity, compute, energy, raw materials, etc). To go back to a question I asked upthread: what would the economy look like if you didn't need employees to utilize resources or undertake projects? What happens when most households aren't competitive at anything economically, but there's still a lot of stuff out there to be owned? My answer is those who own the right things (e.g. AI-run, totally automated, adaptable factories; energy resources) will be able to support themselves indefinitely with them and have massive personal surpluses they then use for vanity projects to keep themselves occupied.
1659447091 2y ago
> That's (a slow) doom for the vast majority of the population, but not for humanity because the wealthy elite would survive. People throughout history have replaced the "elite" once the cost to do so became less than the cost (or benefit) to continue on in the status quo. The perceived "doom" would not last very long (in the greater span or sense of time) as it would all be reorganized, yet again. In that case, I am not so sure it's a "doom scenario" but instead something like the natural way of things, or history continuing on. And, maybe even a needed evolution of the species in that case. Things would have to be "quasi-magical" to play out (for the rest of human eternity) in the way a lot of the AI doom musings point to.
- tivert 2y ago
  > People throughout history have replaced the "elite" once the cost to do so became less than the cost (or benefit) to continue on in the status quo. Past performance is not a guarantee of future success. The elite throughout history relied on masses of common people to get anything done (do work, go to war, etc.). AI as-hyped has the potential to upend that long standing dynamic. The elite aren't going to be replaced if they and the resources they care to protect are ensconced in a 50 mile buffer patrolled by (among other things) drone-swarms that will shove grenades in the face of any unauthorized human that dares enter. That technology is pretty close to reality, as it's basically the Ukraine/Russia front lines with 5-10 more years of drone development.
  - 1659447091 2y ago
    > Past performance is not a guarantee of future success. True. I am still of the belief humans will continue to human. If an uprising of billions vs millions lose, the "elite" would also no longer be "elite". While I have no data, other than observation, the "elite", imo, need the validation of their status more than those who have none and gave it to them in the first place. If they do something so brash as to rid the world of these unwashed masses, they have no one left to rise above. I don't see them all of a sudden becoming great members of this new society; they wanted to own society, not be a citizen in it. How that wouldn't lead to self-elimination at that point is hard to imagine - thus a needed evolution of the species (even if that's extinction and another hominid group later inheriting the leftover mess)

y04nn 2y ago

I think one way for an intelligent AI can gain control over humans is not to exterminate them but to control them, influence them without them being aware of it, changing the recommandation algorithms, producing educational material, videos, podcasts, books that would be pro AI while saturating our monkey brain with mining-less entertaining content. We would become their servant without noticing.

warkdarrior 2y ago
How would you know this is not the case already?
- latentsea 2y ago
  It has been since 2016... Dun-dun-dahhh

tuatoru 2y ago

Here is a counterexample: Universal cheap AI girlfriends and boyfriends. These would have a mildly depressing effect on the economy for a couple of generations, and then the population goes to zero.

latentsea 2y ago
With birth rates already in decline, population is already going to zero.
- red-iron-pine 2y ago
  only Western Gringos, and Asians in countries like SK and China. much of the world is still crapping out kids just fine. subsistence farmers feel more confident doing that than programmers, it seems.
  - sharemywin 2y ago
    ROI. only takes 5 years to make a good crop picker. much longer to make a good programmer.
    - latentsea 2y ago
      And once you've got lots of crop pickers you can transition to management. Then you can start a marketplace, then upgrade that to a chain of supermarkets, then upgrade that to a global FMCG conglomerate, and then your birth rates will also decline and population go to zero. It's a sad realization that modern life which we deem the right way to exist is basically just wrong. That which is unsustainable by definition cannot be sustained after all.

TacticalCoder 2y ago

> The only way for this not to happen is AI controlled robots first gaining complete control of massive chunks of the economy. So they do that because they plan to annihilate all humans. They succeed in both. How does the AI deal with physics and the heat death of the universe?

humansareok1 2y ago

> All the doom scenarios seem to rely on quasi-magical abilities springing from advanced intelligence The amount of people who dismiss doom risk with some version of "I'm not smart enough to imagine what something far smarter than me might be capable of" is staggering. "I can't imagine how nanobots might work therefore they must be impossible." >Then, suppose AI does somehow think itself into effecting human annihilation, what happens when all the humans are dead? At the current state of things, all electricity production on the planet goes dark in maybe two weeks. Nuke plants irradiate significant parts of the earth, chip fabs are bricked, metal production is toast. In short the AI necessarily dies too. Smart enough to eradicate humanity but not smart enough to plan ahead to keep itself running. Ok buddy. >The only way for this not to happen is AI controlled robots first gaining complete control of massive chunks of the economy. If AI is as useless as the article claims then that will effectively never happen. "AI hasn't taken over large parts of the economy, therefore it never will." I really just can't even...

throwanem 2y ago
Mostly the people dismissing the x-risk concern do so on the grounds that what is being sold is not and cannot become smart in the way it's being sold by x-risk stans as being able to, which is true. If you want to argue the contrary case, starting from the actual point of disagreement is probably a better play, because the argument you've actually made here constitutes no answer.
- humansareok1 2y ago
  >what is being sold is not and cannot become smart in the way it's being sold by x-risk stans Do you have actual evidence for this astonishing take? Because I'm sure the entire field would be entranced by your genius here. It's actually completely UN-clear what the limits to LLMs are and it's completely ridiculous to say otherwise. >starting from the actual point of disagreement is probably a better play Fundamentally most e/accs or other anti-doom stans believe that AGI/ASI are actually impossible, that nothing can be smarter than humans and that human brains constitute special magic meat. Almost all of their other wrong beliefs stem from those.