4 ms·

Existential threat is a really hard thing to grapple with. You can't wait for real-world evidence of danger to deal with it. If there's a possibility of an as

Existential threat is a really hard thing to grapple with. You can't wait for real-world evidence of danger to deal with it. If there's a possibility of an asteroid coming that will wipe out life on earth, waiting until our telescopes can see it to start taking action may not give us enough time to react. We have to plan for the theoretical possibility. Maybe we'll never be able to create artificial superintelligence. Maybe artificial superintelligence will be much easier to align than we thing. Maybe it'll be hard, but manageable, possibly because we have more time to work on the problem than we think. All of those are possibilities, but the fact that if you ask 100 people about it, you'll get 100 different opinions, means that we have very little idea how things will go. Given that, I don't like the approach of wing it / hope for the best. If we end up stifling AI innovation unnecessarily, that may mean that our techno-utopia is delayed for a few years, but we get there in the end. On the other hand, if we under-react and things turn out really bad, we may not get another try. Scenarios where AGI kills a billion people are really bad, but scenarios where we create it and end up permanently disempowered or dead are worse. I wonder sometimes if people are just really dissatisfied with their lives, that they react with such vitriol to the idea of AI progress being delayed. I know we have a lot of problems, but how many of them are technological vs. having poor human institutions and incentive structures? AI probably won't help much with that.

BeFlatXIII 3y ago
> If there's a possibility of an asteroid coming that will wipe out life on earth, waiting until our telescopes can see it to start taking action may not give us enough time to react. How would we even know what to do until we have a concrete target?
- Vecr 3y ago
  They can test deflection methods on other asteroids, build large spacecraft launchers and have them ready to go, do surveys to make sure the N-body predictions are as accurate as possible, etc. Some of that's being done (NEOWISE, DART) but if you want to deflect a planet killer you need a lot more.
- NumberWangMan 3y ago
  Vecr gives good analogies, but for AGI, I think investing a lot into alignment research, trying to set up incentive structures to actively avoid "arms races", establishing global oversight and agreements, having a "fire alarm" or process in place that a company can trigger if they develop an AI that they think is close to becoming dangerous, or even individuals. Strong whistleblower protections may help. Tracking GPUs so we know where compute is being gathered together. Starting talks with other countries like China, right now, to build a common understanding of how to deal with a potential problem. I think it's true that it may be very difficult to solve alignment theoretically, and solving it when we have real-world systems to look at may make it easier (but still probably not easy). But we need to be able to buy companies time so they can slow down as we approach what seems to be the "edge" of dangerous capability / consciousness / goal directedness, without feeling like they will lose out. And to be very strict about the conditions the models are trained under and what testing they undergo before release. A lot of this is stuff that people are already suggesting, but not everyone is taking the risk seriously.

BeFlatXIII 3y ago

> If there's a possibility of an asteroid coming that will wipe out life on earth, waiting until our telescopes can see it to start taking action may not give us enough time to react. How would we even know what to do until we have a concrete target?

Vecr 3y ago
They can test deflection methods on other asteroids, build large spacecraft launchers and have them ready to go, do surveys to make sure the N-body predictions are as accurate as possible, etc. Some of that's being done (NEOWISE, DART) but if you want to deflect a planet killer you need a lot more.
NumberWangMan 3y ago
Vecr gives good analogies, but for AGI, I think investing a lot into alignment research, trying to set up incentive structures to actively avoid "arms races", establishing global oversight and agreements, having a "fire alarm" or process in place that a company can trigger if they develop an AI that they think is close to becoming dangerous, or even individuals. Strong whistleblower protections may help. Tracking GPUs so we know where compute is being gathered together. Starting talks with other countries like China, right now, to build a common understanding of how to deal with a potential problem. I think it's true that it may be very difficult to solve alignment theoretically, and solving it when we have real-world systems to look at may make it easier (but still probably not easy). But we need to be able to buy companies time so they can slow down as we approach what seems to be the "edge" of dangerous capability / consciousness / goal directedness, without feeling like they will lose out. And to be very strict about the conditions the models are trained under and what testing they undergo before release. A lot of this is stuff that people are already suggesting, but not everyone is taking the risk seriously.