3 ms·

It's not just America. The main secret is out of the bag. If it wasn't Anthropic, it would be another company/nation state. Sure they could obtain, and with not

It's not just America. The main secret is out of the bag. If it wasn't Anthropic, it would be another company/nation state. Sure they could obtain, and with not.money or leverage, complain about data centers at local rallies, or they can be in the game, and hopefully steer it. It's going to happen with or without any one company or country. The secret it out, and it's unstoppable without complete societal breakdown..So either you advocate for the end of civilization, or you hope that you can help steer the emergence of super intelligence into something not wholly terrible. Personally, I don't see much hope, even if there wasn't such a thing as AI. The power to destroy is always easier than the power to create, and as our power grows, the differential grows, until at some point, containment is no longer possible

caconym_ 4mo ago
> It's not just America. I'll mention again the nuclear analogy. It is, believe it or not, possible for great powers, and even adversary great powers, to agree to limit the development and proliferation of dangerous technologies. > The main secret is out of the bag. This is not something you can do in a shed with a handful of GPUs just because you know "the main secret". To build something like Mythos you need tens of billions of dollars, massive amounts of power, enormous buildings filled with specialized bleeding edge computer chips that are made by (optimistically) a handful of companies with deep government ties. You need free access to all the intellectual property that humans have created and posted openly on the internet. You need all of this at each step, and to take each next step you (or somebody) needs to have taken the previous one. For now, there are a million ways for a government to pump the brakes on this cycle.
- estearum 4mo ago
  > It is, believe it or not, possible for great powers, and even adversary great powers, to agree to limit the development and proliferation of dangerous technologies. You were literally just criticizing Anthropic as disingenuous for begging for this. Or is your position that people other than those near the front of the race can agree to limit development? And if so: provide evidence. Note also a key ingredient that makes nuclear non-proliferation possible is that they're pretty much useless weapons. There is no smaller order nuke that's dramatically more useful than a large conventional weapon. That's not true of AI models, which appear to be monotonically useful as they become more powerful.
- SubiculumCode 4mo ago
  Yeah, I think the nuclear analogy fails, honestly. The bomb does one thing: Destroy. AI can build, selectively infect, selectively manipulate. It is a vastly useful tool.
  - caconym_ 4mo ago
    This is a very imprecise way to think about it. What is the difference? It's easier to make money with the AI you get at each incremental step toward potentially destroying human civilization (though, of course, it's debatable whether these companies really are making money as such). So what? You are implicitly arguing that human civilization will be unable to resist engaging in a large-scale, coordinated effort to destroy itself, just to make a few bucks along the way. Is this true? I don't know. The point of the nuclear analogy is that we have previously shown that we can, under certain conditions, put the eschaton back on the shelf for some period of time, despite very real pressure to take more incremental steps toward doom. "But AI can write code" is not a refutation of the possibility that we could take a more measured approach to AI development.
    - SubiculumCode 4mo ago
      I never claimed to be precise, ha ha. But let's not lose sight that each of the great powers agreed to limits on the number of nukes only after making enough to collectively destroy each other several times over. They stopped exactly when building more did not gain them any more security or advantage over their adversaries. Not a moment before that. There may or may not be such a point with AI: A point at which ever smarter machines provides no marginal benefit to security. If that happens, I do expect agreements, should any one of us still exist.

caconym_ 4mo ago

> It's not just America. I'll mention again the nuclear analogy. It is, believe it or not, possible for great powers, and even adversary great powers, to agree to limit the development and proliferation of dangerous technologies. > The main secret is out of the bag. This is not something you can do in a shed with a handful of GPUs just because you know "the main secret". To build something like Mythos you need tens of billions of dollars, massive amounts of power, enormous buildings filled with specialized bleeding edge computer chips that are made by (optimistically) a handful of companies with deep government ties. You need free access to all the intellectual property that humans have created and posted openly on the internet. You need all of this at each step, and to take each next step you (or somebody) needs to have taken the previous one. For now, there are a million ways for a government to pump the brakes on this cycle.

estearum 4mo ago
> It is, believe it or not, possible for great powers, and even adversary great powers, to agree to limit the development and proliferation of dangerous technologies. You were literally just criticizing Anthropic as disingenuous for begging for this. Or is your position that people other than those near the front of the race can agree to limit development? And if so: provide evidence. Note also a key ingredient that makes nuclear non-proliferation possible is that they're pretty much useless weapons. There is no smaller order nuke that's dramatically more useful than a large conventional weapon. That's not true of AI models, which appear to be monotonically useful as they become more powerful.
SubiculumCode 4mo ago
Yeah, I think the nuclear analogy fails, honestly. The bomb does one thing: Destroy. AI can build, selectively infect, selectively manipulate. It is a vastly useful tool.
- caconym_ 4mo ago
  This is a very imprecise way to think about it. What is the difference? It's easier to make money with the AI you get at each incremental step toward potentially destroying human civilization (though, of course, it's debatable whether these companies really are making money as such). So what? You are implicitly arguing that human civilization will be unable to resist engaging in a large-scale, coordinated effort to destroy itself, just to make a few bucks along the way. Is this true? I don't know. The point of the nuclear analogy is that we have previously shown that we can, under certain conditions, put the eschaton back on the shelf for some period of time, despite very real pressure to take more incremental steps toward doom. "But AI can write code" is not a refutation of the possibility that we could take a more measured approach to AI development.
  - SubiculumCode 4mo ago
    I never claimed to be precise, ha ha. But let's not lose sight that each of the great powers agreed to limits on the number of nukes only after making enough to collectively destroy each other several times over. They stopped exactly when building more did not gain them any more security or advantage over their adversaries. Not a moment before that. There may or may not be such a point with AI: A point at which ever smarter machines provides no marginal benefit to security. If that happens, I do expect agreements, should any one of us still exist.