6 ms·

> it's worth trying to stop it OP's point has nothing to do with this, OP's point is that you can't stop it. The methods and materials are too diffuse and the

> it's worth trying to stop it OP's point has nothing to do with this, OP's point is that you can't stop it. The methods and materials are too diffuse and the biggest players (nation states) have a strong incentive to be first. Do you really expect China to coordinate with the West on this?

hollerith 2y ago
I don't expect China to coordinate with the West, but I think there is a good chance that the only reason Beijing is interested in AI beyond the AI tech they need to keep internal potential revolutionaries under surveillance is to prevent a repeat of the Century of Humiliation (which was caused by the West's technological superiority) so that if the Western governments banned AI, Beijing would be glad to ban it inside China, too.
- philomath_mn 2y ago
  That is a massive bet based on the supposed psychology of a world super power. There are many other less-superficial reasons why Beijing may be interested in AI, plus China may not trust that we actually banned our own AI development. I wouldn't take that bet in a million years.
  - hollerith 2y ago
    You seem to think that if we refuse this bet, you are somehow safe to live out the rest of your life. (If you are old, replace "you" with "your children".) The discussion started when someone argued that even if this AI juggernaut were in fact very dangerous, there is no way to stop it. When I pushed back on the second part of that, you reject my push-back. On what basis? I hope it is not, "I just want things to keep on going the way they are," as if ignoring the AI danger somehow makes the AI danger go away.
    - philomath_mn 2y ago
      No, I do not expect things to just work out. I just think our best chance is for the US to be a leader in AI development and hope that we're able to develop it safely. I don't have a lot of confidence that this will be the case, but I think the US continuing to develop AI is the decision with the best distribution of possible outcomes.
    - philomath_mn 2y ago
      Also, to be clear: I reject your pushback based on my understanding of the incentives/goals/interests of nation states like China. This is completely separate from my personal preferences or hopes about the future of AI.
- hcurtiss 2y ago
  I find it exceedingly unlikely that if the US got rid of all its nukes, that China would too. I also find the inverse unlikely. This is not how state power (or even humans) have ever worked. Ever.
  - hackinthebochs 2y ago
    Nukes are in control of the ruling class in perpetuity. AGI has the potential to overturn the current political order and remake it into something entirely unpredictable. Why the hell would an authoritarian regime want that? I strongly suspect China would take a way out of the AGI race if a legitimate one was offered.
    - hollerith 2y ago
      I agree. Westerners, particularly Americans and Brits, are comfortable or at least reconciled with drastic societal change. China and Russia have seen too many invasions, revolutions, peasant rebellions and ethnic-autonomy rebellions (each of which taking millions of lives) to have anything like the same comfort level that Westerners have.
      - hcurtiss 2y ago
        Oh, I agree that neither power wants the peasants to have them. But make no mistake -- both governments want them, and desperately. There is no universe where there is a multi-lateral agreement to actually eliminate these tools. With loitering munitions and drone swarms, they are ALREADY key components of nation-state force projection.
        hollerith 2y ago
        I'm old enough to remember the public debate about human cloning and human germ-line engineering. In the 1970s some argued like you are arguing here, but those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field and no one is willing to fund the work because of reputational, legal and criminal-prosecution risk.
JoshTriplett 2y ago
> OP's point has nothing to do with this, OP's point is that you can't stop it. So what is your solution? Give up and die? It's worth trying. If it buys us a few years that's a few more years to figure out alignment. > The methods and materials are too diffuse and the biggest players (nation states) have a strong incentive to be first. So there's a strong incentive to convince them "stop racing towards death". > Do you really expect China to coordinate with the West on this? Yes, there have been concrete examples of willingness towards doing so.
- philomath_mn 2y ago
  I think it is extremely unlikely we are going to be able to convince every interested party that they should give up the golden goose for the sake of possible calamity. I think there are risks here, not trying to minimize that, but the coordination problem becomes untenable when the risks/benefits are so large. It is essentially the same problem as the atom bomb: it would have been better if we all agreed not to do it, but thats just not possible. Why should China trust the US or vice versa? Who wants to live in a world where your competitors have world-changing technology but you don't? But here we have a technology with immense militaristic and economic value, so the everyone-wants-it problem is even more pronounced. I don't _like_ this, I just don't see how we can achieve an AI moratorium outside of bombing the data centers (which I also don't think is a good idea). We need to choose the policy with the best distribution of possible outcomes: - The US leads an effort to stop AI development: too much risk that other parties do it anyway - The US continues to lead AI development: hope that P(takeoff) is low and that the good intentions of some US labs are able to achieve safe development I prefer the latter -- this is far from the best hypothetical outcome, but I think it is the best we can do when constrained by reality.
  - JoshTriplett 2y ago
    (The thread at https://news.ycombinator.com/item?id=43029149 https://news.ycombinator.com/item?id=43029149 is also responsive to this; linking it to avoid duplication.)

hollerith 2y ago

I don't expect China to coordinate with the West, but I think there is a good chance that the only reason Beijing is interested in AI beyond the AI tech they need to keep internal potential revolutionaries under surveillance is to prevent a repeat of the Century of Humiliation (which was caused by the West's technological superiority) so that if the Western governments banned AI, Beijing would be glad to ban it inside China, too.

philomath_mn 2y ago
That is a massive bet based on the supposed psychology of a world super power. There are many other less-superficial reasons why Beijing may be interested in AI, plus China may not trust that we actually banned our own AI development. I wouldn't take that bet in a million years.
- hollerith 2y ago
  You seem to think that if we refuse this bet, you are somehow safe to live out the rest of your life. (If you are old, replace "you" with "your children".) The discussion started when someone argued that even if this AI juggernaut were in fact very dangerous, there is no way to stop it. When I pushed back on the second part of that, you reject my push-back. On what basis? I hope it is not, "I just want things to keep on going the way they are," as if ignoring the AI danger somehow makes the AI danger go away.
  - philomath_mn 2y ago
    No, I do not expect things to just work out. I just think our best chance is for the US to be a leader in AI development and hope that we're able to develop it safely. I don't have a lot of confidence that this will be the case, but I think the US continuing to develop AI is the decision with the best distribution of possible outcomes.
  - philomath_mn 2y ago
    Also, to be clear: I reject your pushback based on my understanding of the incentives/goals/interests of nation states like China. This is completely separate from my personal preferences or hopes about the future of AI.
hcurtiss 2y ago
I find it exceedingly unlikely that if the US got rid of all its nukes, that China would too. I also find the inverse unlikely. This is not how state power (or even humans) have ever worked. Ever.
- hackinthebochs 2y ago
  Nukes are in control of the ruling class in perpetuity. AGI has the potential to overturn the current political order and remake it into something entirely unpredictable. Why the hell would an authoritarian regime want that? I strongly suspect China would take a way out of the AGI race if a legitimate one was offered.
  - hollerith 2y ago
    I agree. Westerners, particularly Americans and Brits, are comfortable or at least reconciled with drastic societal change. China and Russia have seen too many invasions, revolutions, peasant rebellions and ethnic-autonomy rebellions (each of which taking millions of lives) to have anything like the same comfort level that Westerners have.
    - hcurtiss 2y ago
      Oh, I agree that neither power wants the peasants to have them. But make no mistake -- both governments want them, and desperately. There is no universe where there is a multi-lateral agreement to actually eliminate these tools. With loitering munitions and drone swarms, they are ALREADY key components of nation-state force projection.
      - hollerith 2y ago
        I'm old enough to remember the public debate about human cloning and human germ-line engineering. In the 1970s some argued like you are arguing here, but those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field and no one is willing to fund the work because of reputational, legal and criminal-prosecution risk.

JoshTriplett 2y ago

> OP's point has nothing to do with this, OP's point is that you can't stop it. So what is your solution? Give up and die? It's worth trying. If it buys us a few years that's a few more years to figure out alignment. > The methods and materials are too diffuse and the biggest players (nation states) have a strong incentive to be first. So there's a strong incentive to convince them "stop racing towards death". > Do you really expect China to coordinate with the West on this? Yes, there have been concrete examples of willingness towards doing so.

philomath_mn 2y ago
I think it is extremely unlikely we are going to be able to convince every interested party that they should give up the golden goose for the sake of possible calamity. I think there are risks here, not trying to minimize that, but the coordination problem becomes untenable when the risks/benefits are so large. It is essentially the same problem as the atom bomb: it would have been better if we all agreed not to do it, but thats just not possible. Why should China trust the US or vice versa? Who wants to live in a world where your competitors have world-changing technology but you don't? But here we have a technology with immense militaristic and economic value, so the everyone-wants-it problem is even more pronounced. I don't _like_ this, I just don't see how we can achieve an AI moratorium outside of bombing the data centers (which I also don't think is a good idea). We need to choose the policy with the best distribution of possible outcomes: - The US leads an effort to stop AI development: too much risk that other parties do it anyway - The US continues to lead AI development: hope that P(takeoff) is low and that the good intentions of some US labs are able to achieve safe development I prefer the latter -- this is far from the best hypothetical outcome, but I think it is the best we can do when constrained by reality.
- JoshTriplett 2y ago
  (The thread at https://news.ycombinator.com/item?id=43029149 https://news.ycombinator.com/item?id=43029149 is also responsive to this; linking it to avoid duplication.)