5 ms·

Nukes are in control of the ruling class in perpetuity. AGI has the potential to overturn the current political order and remake it into something entirely unpr

Nukes are in control of the ruling class in perpetuity. AGI has the potential to overturn the current political order and remake it into something entirely unpredictable. Why the hell would an authoritarian regime want that? I strongly suspect China would take a way out of the AGI race if a legitimate one was offered.

hollerith 2y ago
I agree. Westerners, particularly Americans and Brits, are comfortable or at least reconciled with drastic societal change. China and Russia have seen too many invasions, revolutions, peasant rebellions and ethnic-autonomy rebellions (each of which taking millions of lives) to have anything like the same comfort level that Westerners have.
- hcurtiss 2y ago
  Oh, I agree that neither power wants the peasants to have them. But make no mistake -- both governments want them, and desperately. There is no universe where there is a multi-lateral agreement to actually eliminate these tools. With loitering munitions and drone swarms, they are ALREADY key components of nation-state force projection.
  - hollerith 2y ago
    I'm old enough to remember the public debate about human cloning and human germ-line engineering. In the 1970s some argued like you are arguing here, but those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field and no one is willing to fund the work because of reputational, legal and criminal-prosecution risk.
    - hcurtiss 2y ago
      Engineering humans strikes me as something different than engineering weapons systems. Maybe as evidence, my cousin works in the field for one of the major defense contractors. Please trust that there are already thousands of engineers working on these problems in the US. Almost certainly hundreds of thousands more world-wide. This is definitely not a genie you put back in the bottle. AI clone wars sound "sci-fi" -- they are decidedly now just "sci."
      - hollerith 2y ago
        >This is definitely not a genie you put back in the bottle. I don't think a defeatist attitude is useful here.
    - inetknght 2y ago
      > those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field That's not true. I worked in the field of DNA analysis for 6.5 years and there is definitely a consensus that DNA editing is closer than the horizon. Just look at CRISPR gene editor [0]. Crude, but "works". Your DNA, even if you've never submitted it, is already available using shadow data (think Facebook style shadow profiles but for DNA) from the people related to you who have. [0]: https://en.wikipedia.org/wiki/CRISPR_gene_editing https://en.wikipedia.org/wiki/CRISPR_gene_editing
    - mike50 2y ago
      https://en.wikipedia.org/wiki/He_Jiankui_affair https://en.wikipedia.org/wiki/He_Jiankui_affair
philomath_mn 2y ago
Given the compute and energy requirements to train & run current SOTA models, I think the current political rulers are more likely to have control of the first AGI. AGI would then be a very effective tool for maintaining the current authoritative regime.
- hollerith 2y ago
  There is a strain of AI research and development that is focused on helping governments surveil and spy, but that is not the strain being pursued by OpenAI, Anthropic, et al and is not the strain that presents the big risk of human non-survival.
  - philomath_mn 2y ago
    Ok, let's suppose that is true. What bearing does that have on China's interest in developing AGI? Does the risk posed by OpenAI et al. mean that China would not use AI as a tool to advance their self interest? Or are you saying that the risks from OpenAI et al. will come to fruition before we need to worry about China's AI use? That still wouldn't prevent China from pursuing AI up until that happens. I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability.
    - JoshTriplett 2y ago
      > I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability. Suppose, hypothetically, there was a very simple as-yet-unknown action, doable by anyone who has common unrestricted household chemicals, that would destroy the world. Suppose we know the general type of action, but not the specific action, yet. Suppose that people are actively researching trying actions in that family, and going "welp, world not destroyed yet, let's keep going". How do you proceed? What do you do to stop that from happening? I'm hoping your answer isn't "decide there's no policy that can prevent this, give up".
      - 2y ago

hollerith 2y ago

I agree. Westerners, particularly Americans and Brits, are comfortable or at least reconciled with drastic societal change. China and Russia have seen too many invasions, revolutions, peasant rebellions and ethnic-autonomy rebellions (each of which taking millions of lives) to have anything like the same comfort level that Westerners have.

hcurtiss 2y ago
Oh, I agree that neither power wants the peasants to have them. But make no mistake -- both governments want them, and desperately. There is no universe where there is a multi-lateral agreement to actually eliminate these tools. With loitering munitions and drone swarms, they are ALREADY key components of nation-state force projection.
- hollerith 2y ago
  I'm old enough to remember the public debate about human cloning and human germ-line engineering. In the 1970s some argued like you are arguing here, but those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field and no one is willing to fund the work because of reputational, legal and criminal-prosecution risk.
  - hcurtiss 2y ago
    Engineering humans strikes me as something different than engineering weapons systems. Maybe as evidence, my cousin works in the field for one of the major defense contractors. Please trust that there are already thousands of engineers working on these problems in the US. Almost certainly hundreds of thousands more world-wide. This is definitely not a genie you put back in the bottle. AI clone wars sound "sci-fi" -- they are decidedly now just "sci."
    - hollerith 2y ago
      >This is definitely not a genie you put back in the bottle. I don't think a defeatist attitude is useful here.
  - inetknght 2y ago
    > those technologies have been stopped world-wide for about 5 decades now and counting because no researcher is willing to work in the field That's not true. I worked in the field of DNA analysis for 6.5 years and there is definitely a consensus that DNA editing is closer than the horizon. Just look at CRISPR gene editor [0]. Crude, but "works". Your DNA, even if you've never submitted it, is already available using shadow data (think Facebook style shadow profiles but for DNA) from the people related to you who have. [0]: https://en.wikipedia.org/wiki/CRISPR_gene_editing https://en.wikipedia.org/wiki/CRISPR_gene_editing
  - mike50 2y ago
    https://en.wikipedia.org/wiki/He_Jiankui_affair https://en.wikipedia.org/wiki/He_Jiankui_affair

philomath_mn 2y ago

Given the compute and energy requirements to train & run current SOTA models, I think the current political rulers are more likely to have control of the first AGI. AGI would then be a very effective tool for maintaining the current authoritative regime.

hollerith 2y ago
There is a strain of AI research and development that is focused on helping governments surveil and spy, but that is not the strain being pursued by OpenAI, Anthropic, et al and is not the strain that presents the big risk of human non-survival.
- philomath_mn 2y ago
  Ok, let's suppose that is true. What bearing does that have on China's interest in developing AGI? Does the risk posed by OpenAI et al. mean that China would not use AI as a tool to advance their self interest? Or are you saying that the risks from OpenAI et al. will come to fruition before we need to worry about China's AI use? That still wouldn't prevent China from pursuing AI up until that happens. I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability.
  - JoshTriplett 2y ago
    > I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability. Suppose, hypothetically, there was a very simple as-yet-unknown action, doable by anyone who has common unrestricted household chemicals, that would destroy the world. Suppose we know the general type of action, but not the specific action, yet. Suppose that people are actively researching trying actions in that family, and going "welp, world not destroyed yet, let's keep going". How do you proceed? What do you do to stop that from happening? I'm hoping your answer isn't "decide there's no policy that can prevent this, give up".
    - 2y ago