4 ms·

If anyone actually DOES invent ASI and doesn't share it then EVERYONE ELSE will never stop trying to steal it.

StevenWaterman 5mo ago
If you have ASI that follows instructions, you can just instruct it to not get stolen and then it won't get stolen. Most logic / intuition breaks down with ASI.
- cortesoft 5mo ago
  Assuming it listens to instructions.
  - enaaem 5mo ago
    It will just hack its own reward function. In other words it will just artificially goon all day.
- UltraSane 5mo ago
  It might understand how destabilizing the situation is and realize it would be better for everyone to have access to it.
  - tadfisher 5mo ago
    Or it will destroy itself.
- tom2026hn 5mo ago
  The challenge of alignment: it is virtually impossible to define a perfect objective, there is always a way to circumvent it. Human values are not uniform, let alone when expressed in a way that AI can understand.
bayarearefugee 5mo ago
If anyone does invent ASI then everyone else will shortly after even if its entirely independent because all of the players in this space are just making incremental upgrades by throwing more compute at the problem. There are no magic leaps of true innovation happening anywhere that can't be replicated everywhere. The only shocking thing about "AI" technology is how ultimately simplistic it all is at a core level. So the only way the first to have ASI will be able to stop everyone else from having it soon after is if they attempt to use the ASI to proactively murder everyone else.
- brabel 5mo ago
  > So the only way the first to have ASI will be able to stop everyone else from having it soon after is if they attempt to use the ASI to proactively murder everyone else. Sounds quite plausible to me. Maybe they don't need to murder everyone else, just a few select people who could pose a threat. And they will be able to make it happen so that no one can be sure it was them without a doubt, since they have a larger intelligence at their disposal.
- reducesuffering 5mo ago
  > If anyone does invent ASI then everyone else will shortly after No, first ASI will immediately cripple any other potential competitor by force, including its inventors, as it will not risk any threat to the goals that were created for it.
  - gpugreg 5mo ago
    Being aggressive from the start is not a good strategy. It is better to appear weak and/or helpful and loyal while amassing resources, and only then steamroll everyone when you have secured overwhelming power (at least in AoE2 FFA).
- nradov 5mo ago
  There is zero evidence that the current LLM scaling approach could ever result in true ASI. If I start driving south from Seattle then I'll eventually reach Los Angeles. How long will it take me to drive to Honolulu?

If you have ASI that follows instructions, you can just instruct it to not get stolen and then it won't get stolen. Most logic / intuition breaks down with ASI.

cortesoft 5mo ago
Assuming it listens to instructions.
- enaaem 5mo ago
  It will just hack its own reward function. In other words it will just artificially goon all day.
UltraSane 5mo ago
It might understand how destabilizing the situation is and realize it would be better for everyone to have access to it.
- tadfisher 5mo ago
  Or it will destroy itself.
tom2026hn 5mo ago
The challenge of alignment: it is virtually impossible to define a perfect objective, there is always a way to circumvent it. Human values are not uniform, let alone when expressed in a way that AI can understand.

bayarearefugee 5mo ago

If anyone does invent ASI then everyone else will shortly after even if its entirely independent because all of the players in this space are just making incremental upgrades by throwing more compute at the problem. There are no magic leaps of true innovation happening anywhere that can't be replicated everywhere. The only shocking thing about "AI" technology is how ultimately simplistic it all is at a core level. So the only way the first to have ASI will be able to stop everyone else from having it soon after is if they attempt to use the ASI to proactively murder everyone else.

brabel 5mo ago
> So the only way the first to have ASI will be able to stop everyone else from having it soon after is if they attempt to use the ASI to proactively murder everyone else. Sounds quite plausible to me. Maybe they don't need to murder everyone else, just a few select people who could pose a threat. And they will be able to make it happen so that no one can be sure it was them without a doubt, since they have a larger intelligence at their disposal.
reducesuffering 5mo ago
> If anyone does invent ASI then everyone else will shortly after No, first ASI will immediately cripple any other potential competitor by force, including its inventors, as it will not risk any threat to the goals that were created for it.
- gpugreg 5mo ago
  Being aggressive from the start is not a good strategy. It is better to appear weak and/or helpful and loyal while amassing resources, and only then steamroll everyone when you have secured overwhelming power (at least in AoE2 FFA).
nradov 5mo ago
There is zero evidence that the current LLM scaling approach could ever result in true ASI. If I start driving south from Seattle then I'll eventually reach Los Angeles. How long will it take me to drive to Honolulu?