4 ms·

Given the compute and energy requirements to train & run current SOTA models, I think the current political rulers are more likely to have control of the first

Given the compute and energy requirements to train & run current SOTA models, I think the current political rulers are more likely to have control of the first AGI. AGI would then be a very effective tool for maintaining the current authoritative regime.

hollerith 2y ago
There is a strain of AI research and development that is focused on helping governments surveil and spy, but that is not the strain being pursued by OpenAI, Anthropic, et al and is not the strain that presents the big risk of human non-survival.
- philomath_mn 2y ago
  Ok, let's suppose that is true. What bearing does that have on China's interest in developing AGI? Does the risk posed by OpenAI et al. mean that China would not use AI as a tool to advance their self interest? Or are you saying that the risks from OpenAI et al. will come to fruition before we need to worry about China's AI use? That still wouldn't prevent China from pursuing AI up until that happens. I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability.
  - JoshTriplett 2y ago
    > I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability. Suppose, hypothetically, there was a very simple as-yet-unknown action, doable by anyone who has common unrestricted household chemicals, that would destroy the world. Suppose we know the general type of action, but not the specific action, yet. Suppose that people are actively researching trying actions in that family, and going "welp, world not destroyed yet, let's keep going". How do you proceed? What do you do to stop that from happening? I'm hoping your answer isn't "decide there's no policy that can prevent this, give up".
    - philomath_mn 2y ago
      Not a great analogy. If - there were a range of expert opinions that P(destroy-the-world) < 100 AND - the chemical could turn lead into gold AND - the chemical would give you a militaristic advantage over your adversaries AND - the US were in the race and could use the chemical to keep other people from making / using the the chemical Then I think we'd be in the same situation as we are with AI: stopping it isn't really a choice, we need to do the best we can with the hand we've been dealt.
      - JoshTriplett 2y ago
        > there were a range of expert opinions that P(destroy-the-world) < 100 I would hope that it would not suffice to say "not a 100% chance of destroying the world". Because there's a wide range of expert opinions saying values in the 1-99% range (see https://en.wikipedia.org/wiki/P(doom) https://en.wikipedia.org/wiki/P(doom) for sample values), and none of those values are even slightly acceptable. But sure, by all means stipulate all the things you said; they're roughly accurate, and comparably discouraging. I think it's completely, deadly wrong to think that "race to find it" is safer than "stop everyone from finding it". Right now, at least, the hardware necessary to do training runs is very expensive and produced in very few places. And the amount of power needed is large on an industrial-data-center scale. Let's start there. We're not yet at the point where someone in their basement can train a new frontier model. (They can run one, but not train one.)
        philomath_mn 2y ago
        > Let's start there Ok, I can imagine a domestic policy like you describe. Through the might and force of the US government, I can see this happening in the US (after considerable effort). But how do you enforce something like that globally? When I say "not really possible" I am leaving out "except by excessive force, up to and including outright war". For the reasons I've mentioned above, lots of people around the world will want this technology. I haven't seen an argument for how we can guarantee that everyone will agree with your level of "acceptable" P(doom). So all we are left with is "bombing the datacenters", which, if your P(doom) is high enough, is internally consistent. I guess what it comes down to is: my P(doom) for AI developed by the US is less than my P(doom) from the war we'd need to stop AI development globally.
- int_19h 2y ago
  That's just not true, though. LLMs are the perfect spies and censors, and any totalitarian state worth its salt is going to want them just for this reason alone.

hollerith 2y ago

There is a strain of AI research and development that is focused on helping governments surveil and spy, but that is not the strain being pursued by OpenAI, Anthropic, et al and is not the strain that presents the big risk of human non-survival.

philomath_mn 2y ago
Ok, let's suppose that is true. What bearing does that have on China's interest in developing AGI? Does the risk posed by OpenAI et al. mean that China would not use AI as a tool to advance their self interest? Or are you saying that the risks from OpenAI et al. will come to fruition before we need to worry about China's AI use? That still wouldn't prevent China from pursuing AI up until that happens. I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability.
- JoshTriplett 2y ago
  > I am still not convinced that there is a policy which can prevent AI from developing outside of the US with high probability. Suppose, hypothetically, there was a very simple as-yet-unknown action, doable by anyone who has common unrestricted household chemicals, that would destroy the world. Suppose we know the general type of action, but not the specific action, yet. Suppose that people are actively researching trying actions in that family, and going "welp, world not destroyed yet, let's keep going". How do you proceed? What do you do to stop that from happening? I'm hoping your answer isn't "decide there's no policy that can prevent this, give up".
  - philomath_mn 2y ago
    Not a great analogy. If - there were a range of expert opinions that P(destroy-the-world) < 100 AND - the chemical could turn lead into gold AND - the chemical would give you a militaristic advantage over your adversaries AND - the US were in the race and could use the chemical to keep other people from making / using the the chemical Then I think we'd be in the same situation as we are with AI: stopping it isn't really a choice, we need to do the best we can with the hand we've been dealt.
    - JoshTriplett 2y ago
      > there were a range of expert opinions that P(destroy-the-world) < 100 I would hope that it would not suffice to say "not a 100% chance of destroying the world". Because there's a wide range of expert opinions saying values in the 1-99% range (see https://en.wikipedia.org/wiki/P(doom) https://en.wikipedia.org/wiki/P(doom) for sample values), and none of those values are even slightly acceptable. But sure, by all means stipulate all the things you said; they're roughly accurate, and comparably discouraging. I think it's completely, deadly wrong to think that "race to find it" is safer than "stop everyone from finding it". Right now, at least, the hardware necessary to do training runs is very expensive and produced in very few places. And the amount of power needed is large on an industrial-data-center scale. Let's start there. We're not yet at the point where someone in their basement can train a new frontier model. (They can run one, but not train one.)
      - philomath_mn 2y ago
        > Let's start there Ok, I can imagine a domestic policy like you describe. Through the might and force of the US government, I can see this happening in the US (after considerable effort). But how do you enforce something like that globally? When I say "not really possible" I am leaving out "except by excessive force, up to and including outright war". For the reasons I've mentioned above, lots of people around the world will want this technology. I haven't seen an argument for how we can guarantee that everyone will agree with your level of "acceptable" P(doom). So all we are left with is "bombing the datacenters", which, if your P(doom) is high enough, is internally consistent. I guess what it comes down to is: my P(doom) for AI developed by the US is less than my P(doom) from the war we'd need to stop AI development globally.
int_19h 2y ago
That's just not true, though. LLMs are the perfect spies and censors, and any totalitarian state worth its salt is going to want them just for this reason alone.