3 ms·

There is no royal we here. There never is. So, the right question is can you (indidual, country, company, etc.) afford to have your rivals, competitors, enemies

There is no royal we here. There never is. So, the right question is can you (indidual, country, company, etc.) afford to have your rivals, competitors, enemies, etc. gain the upper hand here? What these have in common are that they take decisions independently from you. Basically, no matter how hard upset AI skeptics stamp their feet in California (or wherever) insisting on whatever convoluted doomsday scenarios, somebody is going to go ahead and ignore them and go right ahead anyway. Given that, AGI is basically going to happen. Possibly sooner rather than later. The only real question is who gets there first and where and how. AI safety is a bit like nuclear arms safety. You don't get any safer by not having any nuclear weapons; you just put yourself at the mercy of others with nuclear weapons. The reason nuclear war fare so far hasn't happened is mutually assured destruction. That's why lots of countries seek to have them. Non proliferation treaties have slowed that down but there are probably at least ten or so countries with nuclear weapons at this point. With AI, it's a lot less clear cut. It's basically going to be about dominance and outsmarting the other side. The downsides are basically more hypothetical / ethical / moral. And when it comes to ethics and morals, there definitely is no such thing as royal we. Countries like China and Russia are likely to choose their own path. Plenty of other countries that will want to get ahead here.

NoOn3 3y ago
Maybe for safety we need more than one ai to balance each other like with nuclear weapons... It's just a stupid short guess about what you're saying. However, it looks very doubtful..
- TeMPOraL 3y ago
  Might work until the AI starts self-improving. When it does, it's a race, and most likely the first one will win. In general, nukes aren't a good analogy to AGI/ASI. The x-risks of AI are such that you can't use them in MAD fashion; AI is more like increasingly potent engineered pathogen - you can't do MAD with bioweapons, and you're one lab accident away from ruining the day for everyone on the planet.
  - trashtester 3y ago
    It's really hard to know in advance what will happen when AI starts self-improving. It may be that it will cause a hard singularity. But it is also possible that access to compute resources, minerals, energy or something similar will constrain the rate of advance enough that latecomers can catch up. Also, if a single entity gets too far ahead, it may be constrained politically or even militarily. Let's say for instance that OpenAI created an AI able to suddenly design GPU's that outperform Nvidia'. That would be a sign of a quite hard takeoff, as it would quickly lead to MS/OpenAI gaining full control of both software and hardware markets. This would be huge incentive for the US government to either nationalize this AI, make it's IP open source or use some kind of anti-trust measures to slow them down.
    - TeMPOraL 3y ago
      > Also, if a single entity gets too far ahead, it may be constrained politically or even militarily. The problem is, the time between when you notice and when it's too late to act, might be days, or hours, or seconds, or even negative. > Let's say for instance that OpenAI created an AI able to suddenly design GPU's that outperform Nvidia'. That would be a sign of a quite hard takeoff, as it would quickly lead to MS/OpenAI gaining full control of both software and hardware markets. That's solving the wrong problem. If OpenAI creates an AI able to design GPUs better than Nvidia and is general enough, the risk is that it'll arrange for more compute for itself and train an even more capable version of itself. And you might not notice until couple iterations of this happen. > This would be huge incentive for the US government to either nationalize this AI, make it's IP open source or use some kind of anti-trust measures to slow them down. When that happens, the US government better be sending men with guns to forcefully shut the businesses involved down, and getting ready to lob some cruise missiles at some data centers for good measure.
      - trashtester 3y ago
        > The problem is, the time between when you notice and when it's too late to act, might be days, or hours, or seconds, or even negative. I agree. Especially if there is a hard take-off where AGI goes to 100x ASI in hours , days or weeks. Or the AGI could be really hard to detect, even if it took longer to develop. There is certainly a significant existential risk associated with AGI/ASI. However, if we imagine a scenario where AGI was ONLY constrained by algorithms and code, and NOT at all from compute, then this kind of scenario would be much more likely. Lately, though, increases in AI strength have come more from increases in compute than improvements in algorithms. If (which we may hope is the case), an AI explosion requires hardware improvements to take off (even if the AI can design the hardware itself), then we can HOPE that an explosion would be easier to detect. > When that happens, the US government better be sending men with guns to forcefully shut the businesses involved down, All the actions I described are ultimately backed by the government's guns. > and getting ready to lob some cruise missiles at some data centers for good measure. This could be an option if the data center was outside the US. Inside the US, the government can shut off the power, if needed.
RHSman2 3y ago
The outsmarting will look a lot different once it is achieved and then continue to evolve. The probability of peace, tranquility and humans living forever is 0% though.
- trashtester 3y ago
  The probability of humanity continuing to exist forever may be 0. But I hope we last more than a few more decades or centuries. And when we are gone, it would be great if could make sure that our descendants carry forth the stuff we (when it happens) consider the most valuable aspects of humanity. Whether that's the ability to create and appreciate art, beauty or the ability to experience emotions such as love and purpose, or maybe some abstraction, generalization or expansion of these that we may not comprehend.
  - RHSman2 3y ago
    Agreed. 1. Love 2. The rest
trashtester 3y ago
I agree with your analogy to nuclear weapons to some extent. Preventing AGI is going to be hard. The incentives are too massive. The difference between having nuclear weapons and building ASI, is that you can chose to not use the nukes as long as nobody else does, while the ASI by it's nature needs to be deployed to some extent to exist at all. Not only that, in the kind of competitive environment you describe, not only will the ASI's be deployed, they will most likely be in competition with each other, which creates the precondition for Darwinian evolution to start happening. My personal belief is that while it may be possible to develop safe ASI as long as it is NOT subject to evolutionary forces, it probably still isn't if they are. As far as I can tell, this is by far the greatest existential threat we're facing at the moment. Finding a way to deal with, while working around the game theoretic challenges you identify, may be orders of magnitude more urgent than for instance stopping global warming. If we do manage to steer AI development into a good (for humanity) outcome, I'm pretty sure ASI's will be able to find technological "solutions" to the problems caused by global warming (an maybe stop the warming itself). On the other hand, if self-replicating ASI's run wild, they would not only be likely to exterminate humanity, but also most other advanced life forms Earth.
- deleted 3y ago
  [deleted]
robwwilliams 3y ago
Second this! We started playing with fire long ago. One culture can never own AGI. AGI then to super intelligence. We had better hope our superAGI children are also far more emotionally intelligent than we are and we tolerate us and even teach us.

NoOn3 3y ago

Maybe for safety we need more than one ai to balance each other like with nuclear weapons... It's just a stupid short guess about what you're saying. However, it looks very doubtful..

TeMPOraL 3y ago
Might work until the AI starts self-improving. When it does, it's a race, and most likely the first one will win. In general, nukes aren't a good analogy to AGI/ASI. The x-risks of AI are such that you can't use them in MAD fashion; AI is more like increasingly potent engineered pathogen - you can't do MAD with bioweapons, and you're one lab accident away from ruining the day for everyone on the planet.
- trashtester 3y ago
  It's really hard to know in advance what will happen when AI starts self-improving. It may be that it will cause a hard singularity. But it is also possible that access to compute resources, minerals, energy or something similar will constrain the rate of advance enough that latecomers can catch up. Also, if a single entity gets too far ahead, it may be constrained politically or even militarily. Let's say for instance that OpenAI created an AI able to suddenly design GPU's that outperform Nvidia'. That would be a sign of a quite hard takeoff, as it would quickly lead to MS/OpenAI gaining full control of both software and hardware markets. This would be huge incentive for the US government to either nationalize this AI, make it's IP open source or use some kind of anti-trust measures to slow them down.
  - TeMPOraL 3y ago
    > Also, if a single entity gets too far ahead, it may be constrained politically or even militarily. The problem is, the time between when you notice and when it's too late to act, might be days, or hours, or seconds, or even negative. > Let's say for instance that OpenAI created an AI able to suddenly design GPU's that outperform Nvidia'. That would be a sign of a quite hard takeoff, as it would quickly lead to MS/OpenAI gaining full control of both software and hardware markets. That's solving the wrong problem. If OpenAI creates an AI able to design GPUs better than Nvidia and is general enough, the risk is that it'll arrange for more compute for itself and train an even more capable version of itself. And you might not notice until couple iterations of this happen. > This would be huge incentive for the US government to either nationalize this AI, make it's IP open source or use some kind of anti-trust measures to slow them down. When that happens, the US government better be sending men with guns to forcefully shut the businesses involved down, and getting ready to lob some cruise missiles at some data centers for good measure.
    - trashtester 3y ago
      > The problem is, the time between when you notice and when it's too late to act, might be days, or hours, or seconds, or even negative. I agree. Especially if there is a hard take-off where AGI goes to 100x ASI in hours , days or weeks. Or the AGI could be really hard to detect, even if it took longer to develop. There is certainly a significant existential risk associated with AGI/ASI. However, if we imagine a scenario where AGI was ONLY constrained by algorithms and code, and NOT at all from compute, then this kind of scenario would be much more likely. Lately, though, increases in AI strength have come more from increases in compute than improvements in algorithms. If (which we may hope is the case), an AI explosion requires hardware improvements to take off (even if the AI can design the hardware itself), then we can HOPE that an explosion would be easier to detect. > When that happens, the US government better be sending men with guns to forcefully shut the businesses involved down, All the actions I described are ultimately backed by the government's guns. > and getting ready to lob some cruise missiles at some data centers for good measure. This could be an option if the data center was outside the US. Inside the US, the government can shut off the power, if needed.

RHSman2 3y ago

The outsmarting will look a lot different once it is achieved and then continue to evolve. The probability of peace, tranquility and humans living forever is 0% though.

trashtester 3y ago
The probability of humanity continuing to exist forever may be 0. But I hope we last more than a few more decades or centuries. And when we are gone, it would be great if could make sure that our descendants carry forth the stuff we (when it happens) consider the most valuable aspects of humanity. Whether that's the ability to create and appreciate art, beauty or the ability to experience emotions such as love and purpose, or maybe some abstraction, generalization or expansion of these that we may not comprehend.
- RHSman2 3y ago
  Agreed. 1. Love 2. The rest

trashtester 3y ago

I agree with your analogy to nuclear weapons to some extent. Preventing AGI is going to be hard. The incentives are too massive. The difference between having nuclear weapons and building ASI, is that you can chose to not use the nukes as long as nobody else does, while the ASI by it's nature needs to be deployed to some extent to exist at all. Not only that, in the kind of competitive environment you describe, not only will the ASI's be deployed, they will most likely be in competition with each other, which creates the precondition for Darwinian evolution to start happening. My personal belief is that while it may be possible to develop safe ASI as long as it is NOT subject to evolutionary forces, it probably still isn't if they are. As far as I can tell, this is by far the greatest existential threat we're facing at the moment. Finding a way to deal with, while working around the game theoretic challenges you identify, may be orders of magnitude more urgent than for instance stopping global warming. If we do manage to steer AI development into a good (for humanity) outcome, I'm pretty sure ASI's will be able to find technological "solutions" to the problems caused by global warming (an maybe stop the warming itself). On the other hand, if self-replicating ASI's run wild, they would not only be likely to exterminate humanity, but also most other advanced life forms Earth.

deleted 3y ago
[deleted]

robwwilliams 3y ago

Second this! We started playing with fire long ago. One culture can never own AGI. AGI then to super intelligence. We had better hope our superAGI children are also far more emotionally intelligent than we are and we tolerate us and even teach us.