4 ms·

This is a straw man argument. Cowen does not argue against the idea of existential risk, he argues that specifically nobody should believe they can anticipate

This is a straw man argument. Cowen does not argue against the idea of existential risk, he argues that specifically nobody should believe they can anticipate the consequences of technological change. You can agree with him and still be existentially concerned about a specific asteroid, or climate change. And you could use a little bit of energy monitoring the unintended consequences of AI, the way we monitor asteroids. But to wish to halt AI advancement requires an unhealthy mix of pessimism and overconfidence in your predictive powers.

blagie 4y ago
The number of potentially world-ending technologies is growing exponentially: Chemical, biological, nuclear, etc. I don't think any of them individually are high-risk, but put together, I increasingly believe we need to fundamentally change how we govern humanity to mitigate existential risks. That risk keeps growing as you extend timeline. I don't know whether we can or should halt AI advancement, but I do believe it's not something which should be market-driven. If we set up free markets, market forces become indomitable. You can't stop them.
- evrydayhustling 4y ago
  > we need to fundamentally change how we govern humanity to mitigate existential risks In both cases, which "we" can you possibly mean? Like, today's voters? worldwide? Today's HN users? The CCP? The idea that any "we" may or must choose how to govern humanity is a category error. There are lots of forces other than markets to appeal to - nations, industry groups, religions. But any of them will be in a power struggle, except to the degree that they can show a legitimacy to control. Who has the most legitimacy on this subject?
  - blagie 4y ago
    > There are lots of forces other than markets to appeal to - nations, industry groups, religions. Most of which suffer from an analogue of market forces. For example, elections are won by playing the "getting elected" game near-perfectly. That leads to behavior which no one might want, but people are forced into. > The idea that any "we" may or must choose how to govern humanity is a category error. The category error is that not making a choice isn't a choice. Right now, we're governed by complex dynamics, which are at least as restrictive as choices we might engineer. The book "Dictator's Handbook" gives a nice overview of the dynamics which govern us. It's a game-theoretic (but non-mathematical) analysis of political systems, including corporate and democratic governance.
- BurningFrog 4y ago
  > The number of potentially world-ending technologies is growing exponentially This is an argument for increasing the number of worlds.
  - RajT88 4y ago
    I can imagine now the argument, "bit who will pay for it?". Ironically the periods of time in which we did not care about cost and really did start leaving Earth, it was driven by (drumroll) the existential threat of a particular technology.
  - gremlinsinc 4y ago
    Ironically, the only way we ever do that - probably, is by using Super AGI to solve many physics problems, and get us warp drives or wormholes.
    - Pxtl 4y ago
      AGI also provides a way to have a payload intelligence that's capable of surviving interstellar travel without warp - it can live for a thousand year trip in a shoebox. Which is why i think our culture and stories might make it to the stars, but I'm pretty sure we won't.
  - jeremyjh 4y ago
    That isn't a bad idea but it really isn't an answer that helps us in the next several hundred years. A technological civilization that is fully self-sufficient would need millions of people in every profession like we have on earth. Vernor Vinge estimated 100 million people are required to run a civilization that could refit a starship.
    - AstralStorm 4y ago
      Sufficiently automated and focused, I'd drop one order of magnitude from it. (So 10 million, based on manufacturing, refining and mining estimates, assuming the materials are available etc.) Think of it like applying the city of Shenzhen to the task, if they wanted they could do it. Heck, build one from scratch if materials are there. Having a (reasonably) sustainable 10 million sized habitat is quite hard.
- keithalewis 4y ago
  Paging the Unabomber. Ted to the white manifesto phone.
  - dang 4y ago
    Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. We've asked you more than once to stop, and eventually we have to ban such accounts. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
raviparikh 4y ago
Yes, Cowen argues against the idea we can anticipate consequences of technological change, and the specific consequence he focuses on is the idea of existential risk stemming from AI. He says because technology is unpredictable, we shouldn’t try to predict the type of risk imposed by AI, and we should mostly just accept that this change will happen and cope with it afterward. This stance is what I was arguing against in the post. > But to wish to halt AI advancement requires an unhealthy mix of pessimism and overconfidence in your predictive powers. I did not argue for this in my article!

blagie 4y ago

The number of potentially world-ending technologies is growing exponentially: Chemical, biological, nuclear, etc. I don't think any of them individually are high-risk, but put together, I increasingly believe we need to fundamentally change how we govern humanity to mitigate existential risks. That risk keeps growing as you extend timeline. I don't know whether we can or should halt AI advancement, but I do believe it's not something which should be market-driven. If we set up free markets, market forces become indomitable. You can't stop them.

evrydayhustling 4y ago
> we need to fundamentally change how we govern humanity to mitigate existential risks In both cases, which "we" can you possibly mean? Like, today's voters? worldwide? Today's HN users? The CCP? The idea that any "we" may or must choose how to govern humanity is a category error. There are lots of forces other than markets to appeal to - nations, industry groups, religions. But any of them will be in a power struggle, except to the degree that they can show a legitimacy to control. Who has the most legitimacy on this subject?
- blagie 4y ago
  > There are lots of forces other than markets to appeal to - nations, industry groups, religions. Most of which suffer from an analogue of market forces. For example, elections are won by playing the "getting elected" game near-perfectly. That leads to behavior which no one might want, but people are forced into. > The idea that any "we" may or must choose how to govern humanity is a category error. The category error is that not making a choice isn't a choice. Right now, we're governed by complex dynamics, which are at least as restrictive as choices we might engineer. The book "Dictator's Handbook" gives a nice overview of the dynamics which govern us. It's a game-theoretic (but non-mathematical) analysis of political systems, including corporate and democratic governance.
BurningFrog 4y ago
> The number of potentially world-ending technologies is growing exponentially This is an argument for increasing the number of worlds.
- RajT88 4y ago
  I can imagine now the argument, "bit who will pay for it?". Ironically the periods of time in which we did not care about cost and really did start leaving Earth, it was driven by (drumroll) the existential threat of a particular technology.
- gremlinsinc 4y ago
  Ironically, the only way we ever do that - probably, is by using Super AGI to solve many physics problems, and get us warp drives or wormholes.
  - Pxtl 4y ago
    AGI also provides a way to have a payload intelligence that's capable of surviving interstellar travel without warp - it can live for a thousand year trip in a shoebox. Which is why i think our culture and stories might make it to the stars, but I'm pretty sure we won't.
- jeremyjh 4y ago
  That isn't a bad idea but it really isn't an answer that helps us in the next several hundred years. A technological civilization that is fully self-sufficient would need millions of people in every profession like we have on earth. Vernor Vinge estimated 100 million people are required to run a civilization that could refit a starship.
  - AstralStorm 4y ago
    Sufficiently automated and focused, I'd drop one order of magnitude from it. (So 10 million, based on manufacturing, refining and mining estimates, assuming the materials are available etc.) Think of it like applying the city of Shenzhen to the task, if they wanted they could do it. Heck, build one from scratch if materials are there. Having a (reasonably) sustainable 10 million sized habitat is quite hard.
keithalewis 4y ago
Paging the Unabomber. Ted to the white manifesto phone.
- dang 4y ago
  Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. We've asked you more than once to stop, and eventually we have to ban such accounts. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

raviparikh 4y ago

Yes, Cowen argues against the idea we can anticipate consequences of technological change, and the specific consequence he focuses on is the idea of existential risk stemming from AI. He says because technology is unpredictable, we shouldn’t try to predict the type of risk imposed by AI, and we should mostly just accept that this change will happen and cope with it afterward. This stance is what I was arguing against in the post. > But to wish to halt AI advancement requires an unhealthy mix of pessimism and overconfidence in your predictive powers. I did not argue for this in my article!