4 ms·

This is a really cool paper (that I definitely haven't had the time to fully grok) but the jump from the theoretical environment to concerns about superintellig

This is a really cool paper (that I definitely haven't had the time to fully grok) but the jump from the theoretical environment to concerns about superintelligent AI taking over is a bit strained. It misses the same key issues that most similar arguments miss, even while presenting some really amazing results on statistical properties of optimal policies - there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible". I can see how the arguments in the paper lead us up to the possibility of kinetic conflict with intelligent robots, but the dynamics of power at that stage then seem to be more about how we've embodied those robots rather than optimality results of their control policies. Did we give them off switches? Will they run out of energy after a while? How susceptible to physical damage are they? If someone outfits a million Predator drones with a control policy with a will to live, we may have trouble, but we probably were going to have trouble from the group that decided to build a million Predator drones, too. If the number is lower, more realistic, then you're talking about standard armed conflict. I could see this line of work having implications for autonomous weapons systems in a hypothetical future, but my understanding is that at this time they're on a pretty short leash with very narrow AI systems if they're using AI at all. Incentives could push those systems to get smarter as the autonomous arms race heats up, and if it does so in an unrestricted manner that could spell trouble along the lines discussed in this paper, but incentives are definitely aligned for any group to build weapons systems that can't be turned on their handlers, whether by human hackers or their own volition. Theories on optimal policies have no bearing if there's some encrypted remote kill command, or if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.). Ultimately, the dynamics here seem to be an evolution of the problems of human force and violence, rather than a qualitative jump to a different sort of threat entirely.

314 5y ago
Without reading the paper it seems from the tweets that the results are overstated. Academics need to make their results sound more impressive and less trivial and so the choice of wording reflects that. The property that optimal agents seek is not "power", which is quite a general term - it looks more like "agency" i.e. a more specific form of power. But "Optimal agents seek agency" won't win any future grants.
- Turn_Trout 5y ago
  Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).
Turn_Trout 5y ago
First author here. Thanks for your comment! > there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible". Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct? > Theories on optimal policies have no bearing if See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one. > if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.) I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes. My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded. [1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satisficers-tend-to-seek-power-instrumental-convergence-via https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis... [2] https://avoiding-side-effects.github.io/ https://avoiding-side-effects.github.io/

314 5y ago

Without reading the paper it seems from the tweets that the results are overstated. Academics need to make their results sound more impressive and less trivial and so the choice of wording reflects that. The property that optimal agents seek is not "power", which is quite a general term - it looks more like "agency" i.e. a more specific form of power. But "Optimal agents seek agency" won't win any future grants.

Turn_Trout 5y ago
Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).

Turn_Trout 5y ago

First author here. Thanks for your comment! > there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible". Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct? > Theories on optimal policies have no bearing if See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one. > if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.) I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes. My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded. [1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satisficers-tend-to-seek-power-instrumental-convergence-via https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis... [2] https://avoiding-side-effects.github.io/ https://avoiding-side-effects.github.io/