6 ms·

It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutel

It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.

concordDance 3y ago
That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.
- jeffparsons 3y ago
  It depends on there being _some chance_ that it can be achieved. The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
  - TeMPOraL 3y ago
    Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
    - varjag 3y ago
      Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
      - TeMPOraL 3y ago
        I did, and you're right in both of your points.
johnthewise 3y ago
He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U
- jeffparsons 3y ago
  You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
- hollerith 3y ago
  >He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
  - oceanplexian 3y ago
    I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
TeMPOraL 3y ago
> Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal. From the way you put it just now: - "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone. - "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire. - "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war. > Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply. Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention. Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby. Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact. But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky! > If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. There are fates worse than death. Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident. > At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips. With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone". AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world. -- [0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.

concordDance 3y ago

That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.

jeffparsons 3y ago
It depends on there being _some chance_ that it can be achieved. The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
- TeMPOraL 3y ago
  Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
  - varjag 3y ago
    Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
    - TeMPOraL 3y ago
      I did, and you're right in both of your points.

johnthewise 3y ago

He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U

jeffparsons 3y ago
You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
hollerith 3y ago
>He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
- oceanplexian 3y ago
  I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.

TeMPOraL 3y ago

> Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal. From the way you put it just now: - "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone. - "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire. - "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war. > Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply. Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention. Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby. Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact. But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky! > If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. There are fates worse than death. Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident. > At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips. With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone". AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world. -- [0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.