3 ms·
> Unlike particles, humans will actively seek to exploit any control system you've set up. But that’s only possible because the control system doesn’t exactly
by layer8 2y ago
> Unlike particles, humans will actively seek to exploit any control system you've set up.
But that’s only possible because the control system doesn’t exactly (and only) control what we want it to control. The control system is only an imperfect proxy for what we really want, in a very similar way as the measure in Goodhart’s law.
Another variation of that is the law of unintended consequences [0]. There is probably a generalized computational or complex-systems version of it that we haven’t discovered yet.
[0] https://www.sas.upenn.edu/~haroldfs/540/handouts/french/unintconseq.html https://www.sas.upenn.edu/~haroldfs/540/handouts/french/unin...
- deleted 2y ago[deleted]
- etiam 2y agoDisagree. Even if it was striving to regulate exactly the right thing in the first place, most of these issues occur for systems where no single actor could be expected to exert complete control and could well be vulnerable anyway. Start working with a nice, clean, fully relevant system, end up modelling that plus the whole range of adversarial perturbations from agents of pretty high complexity.
- layer8 2y agoI don’t exactly see how this is different from what I was describing.
- deleted 2y ago[deleted]