5 ms·

If AI is actually capable of fulfilling all the capabilities suggested by people who believe in the singularity, it has far more capacity for harm than nuclear

by jprete 3y ago

If AI is actually capable of fulfilling all the capabilities suggested by people who believe in the singularity, it has far more capacity for harm than nuclear weapons. I think most people who are strongly pro-AI/pro-acceleration - or, at any rate, not anti-AI - believe that either (A) there is no control problem (B) it will be solved (C) AI won't become independent and agentic (i.e. it won't face evolutionary pressure towards survival) or (D) AI capabilities will hit a ceiling soon (more so than just not becoming agentic). If you strongly believe, or take as a prior, one of those things, then it makes sense to push the gas as hard as possible. If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea.

logicchains 3y ago
>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in the hands of the elites.
- Filligree 3y ago
  > surely it's better to have distributed AGI than to have AGI only in the hands of the elites This is not a given. If your threat model includes "Runaway competition that leads to profit-seekers ignoring safety in a winner-takes-all contest", then the more companies are allowed to play with AI, the worse. Non-monopolies are especially bad. If your threat model doesn't include that, then the same conclusions sound abhorrent and can be nearly guaranteed to lead to awful consequences. Neither side is necessarily wrong, and chances are good that the people behind the first set of rules would agree that it'll lead to awful consequences — just not as bad as the alternative.
- segfaultbuserr 3y ago
  > surely it's better to have distributed AGI than to have AGI only in the hands of the elites. The argument of doing so is the same as Nuclear Non-Proliferation - because of its great abuse potential, giving the technology to everyone only causes random bombings of cities instead of creating a system with checks and balances. I do not necessarily agree with it, but I found the reasoning is not groundless.
  - deleted 3y ago
    [deleted]
  - Aerroon 3y ago
    But the reason for nuclear non-proliferation is to hold onto power. Abuse potential is a great excuse, but it applies to everyone. Current nuclear states have demonstrated that they are willing to indirectly abuse them (you can't invade Russia, but Russia has no problem invading you as long as you aren't backed up by nukes).
    - segfaultbuserr 3y ago
      Both can be true at the same time. The world's superpowers enforce nuclear non-proliferation mainly because it allows them to keep unfair political and military advantages to themselves. At the same time, one cannot deny that centralized weapon ownership made the use of such weapons more controllable: These nuclear states are powerful enough to establish a somewhat responsible chain of command to avoid their unreasonable or accidental uses, and so far these attempts are still successful. Also, due to the fact that they are "too big to fail", they were forced to hire experts to make detailed analysis on the consequences of nuclear wars, and the resulted MAD doctrine discouraged them from starting such wars. On the other hand, if the same nuclear technologies are available to everyone, the chance of an unreasonable or accidental nuclear war will be higher. If even resourceful superpowers can barely keep these nuclear weapons under safe political and technical control (as shown by multiple incidents and near-misses during the Cold War [0]), surely a less resourceful state or military in possession of equally destructive weapons will have even more difficulties on controlling their uses. At least this is how the argument goes (so far, I personally take no position). Of course, I clearly realized that centralized control is not infallible. Months ago, in a previous thread on OpenAI's refusal on publishing technical details of GPT-4, most people believed that they were using it as an excuse to maintain a monopolistic control. Instead, I argued that perhaps OpenAI truly values the problem of safety right now - but acting responsibly right now is not an indication that they will still act responsibly in the future. There's no guarantee that the safety considerations will eventually be overridden in favor of financial gains. [0] https://en.wikipedia.org/wiki/Command_and_Control_(book) https://en.wikipedia.org/wiki/Command_and_Control_(book)
- FeepingCreature 3y ago
  No they really do push for "limit everyone's compute". The people pushing for "limit everyone's compute except us" are allies of convenience that are gonna be inevitably backstabbed. At any rate, if you have like two corps with lots of compute, and something goes wrong, you only have to EMP two datacenters.
- the8472 3y ago
  Demonstrably false: https://twitter.com/ESYudkowsky/status/1772624785672954115 https://twitter.com/ESYudkowsky/status/1772624785672954115
  - Aerroon 3y ago
    Do they also take this position about biological intelligence? Because humans most certainly have an alignment problem too.
    - the8472 3y ago
      The concerning AGI properties include recursive self-improvement to superhuman capability levels and the ability to mass-deploy copies. Those are not on the horizon when it comes to humans. If hypothetically some human acquired such properties that would be equally concerning.
schlauerfox 3y ago
This is all just Pascal's wager anyway.
- segfaultbuserr 3y ago
  Surely, there are many realistic ways to abuse this powerful technology, the creation of a self-aware Paperclip Maximizer is not necessarily required.

by jprete 3y ago

logicchains 3y ago

>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in the hands of the elites.

Filligree 3y ago
> surely it's better to have distributed AGI than to have AGI only in the hands of the elites This is not a given. If your threat model includes "Runaway competition that leads to profit-seekers ignoring safety in a winner-takes-all contest", then the more companies are allowed to play with AI, the worse. Non-monopolies are especially bad. If your threat model doesn't include that, then the same conclusions sound abhorrent and can be nearly guaranteed to lead to awful consequences. Neither side is necessarily wrong, and chances are good that the people behind the first set of rules would agree that it'll lead to awful consequences — just not as bad as the alternative.
segfaultbuserr 3y ago
> surely it's better to have distributed AGI than to have AGI only in the hands of the elites. The argument of doing so is the same as Nuclear Non-Proliferation - because of its great abuse potential, giving the technology to everyone only causes random bombings of cities instead of creating a system with checks and balances. I do not necessarily agree with it, but I found the reasoning is not groundless.
- deleted 3y ago
  [deleted]
- Aerroon 3y ago
  But the reason for nuclear non-proliferation is to hold onto power. Abuse potential is a great excuse, but it applies to everyone. Current nuclear states have demonstrated that they are willing to indirectly abuse them (you can't invade Russia, but Russia has no problem invading you as long as you aren't backed up by nukes).
  - segfaultbuserr 3y ago
    Both can be true at the same time. The world's superpowers enforce nuclear non-proliferation mainly because it allows them to keep unfair political and military advantages to themselves. At the same time, one cannot deny that centralized weapon ownership made the use of such weapons more controllable: These nuclear states are powerful enough to establish a somewhat responsible chain of command to avoid their unreasonable or accidental uses, and so far these attempts are still successful. Also, due to the fact that they are "too big to fail", they were forced to hire experts to make detailed analysis on the consequences of nuclear wars, and the resulted MAD doctrine discouraged them from starting such wars. On the other hand, if the same nuclear technologies are available to everyone, the chance of an unreasonable or accidental nuclear war will be higher. If even resourceful superpowers can barely keep these nuclear weapons under safe political and technical control (as shown by multiple incidents and near-misses during the Cold War [0]), surely a less resourceful state or military in possession of equally destructive weapons will have even more difficulties on controlling their uses. At least this is how the argument goes (so far, I personally take no position). Of course, I clearly realized that centralized control is not infallible. Months ago, in a previous thread on OpenAI's refusal on publishing technical details of GPT-4, most people believed that they were using it as an excuse to maintain a monopolistic control. Instead, I argued that perhaps OpenAI truly values the problem of safety right now - but acting responsibly right now is not an indication that they will still act responsibly in the future. There's no guarantee that the safety considerations will eventually be overridden in favor of financial gains. [0] https://en.wikipedia.org/wiki/Command_and_Control_(book) https://en.wikipedia.org/wiki/Command_and_Control_(book)
FeepingCreature 3y ago
No they really do push for "limit everyone's compute". The people pushing for "limit everyone's compute except us" are allies of convenience that are gonna be inevitably backstabbed. At any rate, if you have like two corps with lots of compute, and something goes wrong, you only have to EMP two datacenters.
the8472 3y ago
Demonstrably false: https://twitter.com/ESYudkowsky/status/1772624785672954115 https://twitter.com/ESYudkowsky/status/1772624785672954115
- Aerroon 3y ago
  Do they also take this position about biological intelligence? Because humans most certainly have an alignment problem too.
  - the8472 3y ago
    The concerning AGI properties include recursive self-improvement to superhuman capability levels and the ability to mass-deploy copies. Those are not on the horizon when it comes to humans. If hypothetically some human acquired such properties that would be equally concerning.

schlauerfox 3y ago

This is all just Pascal's wager anyway.

segfaultbuserr 3y ago
Surely, there are many realistic ways to abuse this powerful technology, the creation of a self-aware Paperclip Maximizer is not necessarily required.