3 ms·
>This is a failure of the "game" rules, provided the intended outcome is something other than this (as it surely was/is). Making a game with winners and losers
by SilasX 9y ago
>This is a failure of the "game" rules, provided the intended outcome is something other than this (as it surely was/is). Making a game with winners and losers and any kind of objective scoring system out of something resembling a real activity is really hard to do without distorting that activity until it no longer resembles what you wanted it to.
Right. In practice, if you want to engineer the rules to be good, (I think) you need both the rigid, pre-defined, unforgiving logical rules, and some inarticulable "I know it when I see it" rules.
I wonder, are there any theorems on the bounds of how far an optimizer can deviate from another's agent's desired outcome if the agent is limited to specifying the rules in n bits and can specify p arbitrary vetoes?
(Made another reply, but kept this separate for its unrelated point.)
- xamuel 9y agoNot a mathematical theorem, and not quantitative, but Campbell's Law is relevant: https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law
- SilasX 9y agoYeah, I'd be interested if there are rigorous bounds on how badly Goodhart/Campell's Law can screw you in theory (or better, in practice).
- pizza 9y agoMaybe this is worth examining: https://arxiv.org/pdf/math/0410076.pdf https://arxiv.org/pdf/math/0410076.pdf