4 ms·

This is an interesting insight into the dangers of ill-chosen metrics. It could only happen in a system in which the police were expected to produce (and were i

This is an interesting insight into the dangers of ill-chosen metrics. It could only happen in a system in which the police were expected to produce (and were in various ways rewarded for) convictions instead of e.g. a maximally accurate presentation of events that can be used to decide how society should respond. Convictions are far easier to measure (and achieve, of course). How would you go about constructing an efficient, corruption-resistant system that is measured by its accuracy? It seems like a hard problem, but not necessarily an unsolvable one. Debugging support issues sometimes feels like this. If you go into a complex system looking for bugs then you'll find plenty - but not necessarily the one that's causing the current issue. What you're really looking for is to fully understand the reason the system does what it does. And then you can change it. Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Kind of difficult to study that. Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization.

Centigonal 12y ago
A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize toward the metric, and it'll stop correlating. Some reading: http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/ http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/
- jacquesm 12y ago
  See also: the breakdown of the link structure of the web because of the rise of google.
brazzy 12y ago
> Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Both, really. There is certainly a small class of scenarios where it does work, but you cannot know in advance where your case is in that class. And is great at making it look like you're "getting results" if you don't look to closely. Another factor: torture is definitely good for maintaining a "reign of terror" style of regime. Until it isn't. I think there's even a passage in Macchiavelli's "The Prince" (i.e. the opus that basically coined the "end justify means" meme) that says ruling people by making them love you is most effective but impossible to do all the time, so ruling through fear is the next best thing - yet one should carefully avoid being hated. > Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization. As a basic principle, definitely. Though no simple rule can solve all moral dilemmas.

Centigonal 12y ago

A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize toward the metric, and it'll stop correlating. Some reading: http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/ http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/

jacquesm 12y ago
See also: the breakdown of the link structure of the web because of the rise of google.

brazzy 12y ago

> Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Both, really. There is certainly a small class of scenarios where it does work, but you cannot know in advance where your case is in that class. And is great at making it look like you're "getting results" if you don't look to closely. Another factor: torture is definitely good for maintaining a "reign of terror" style of regime. Until it isn't. I think there's even a passage in Macchiavelli's "The Prince" (i.e. the opus that basically coined the "end justify means" meme) that says ruling people by making them love you is most effective but impossible to do all the time, so ruling through fear is the next best thing - yet one should carefully avoid being hated. > Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization. As a basic principle, definitely. Though no simple rule can solve all moral dilemmas.