8 ms·

An example, I'm afraid, of situations that happen all too often around the world. When we look at the justice system as any other abstract system (with positive

An example, I'm afraid, of situations that happen all too often around the world. When we look at the justice system as any other abstract system (with positive/negative feedback loops) we find that the actors (the judges, prosecutors, investigators, etc.) are never rewarded for finding the truth. After all, how can you ever know that you found the truth? They are rewarded for finding a truth. For making the picture fit. And that is why you should never collaborate with (or trust) the system. They aren't interested in what's good for you, they are interested in what's good for themselves: finding someone to convict and thereby pushing their yearly evaluation into the positives. [ed.: typo]

moconnor 12y ago
This is an interesting insight into the dangers of ill-chosen metrics. It could only happen in a system in which the police were expected to produce (and were in various ways rewarded for) convictions instead of e.g. a maximally accurate presentation of events that can be used to decide how society should respond. Convictions are far easier to measure (and achieve, of course). How would you go about constructing an efficient, corruption-resistant system that is measured by its accuracy? It seems like a hard problem, but not necessarily an unsolvable one. Debugging support issues sometimes feels like this. If you go into a complex system looking for bugs then you'll find plenty - but not necessarily the one that's causing the current issue. What you're really looking for is to fully understand the reason the system does what it does. And then you can change it. Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Kind of difficult to study that. Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization.
- Centigonal 12y ago
  A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize toward the metric, and it'll stop correlating. Some reading: http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/ http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/
  - jacquesm 12y ago
    See also: the breakdown of the link structure of the web because of the rise of google.
- brazzy 12y ago
  > Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Both, really. There is certainly a small class of scenarios where it does work, but you cannot know in advance where your case is in that class. And is great at making it look like you're "getting results" if you don't look to closely. Another factor: torture is definitely good for maintaining a "reign of terror" style of regime. Until it isn't. I think there's even a passage in Macchiavelli's "The Prince" (i.e. the opus that basically coined the "end justify means" meme) that says ruling people by making them love you is most effective but impossible to do all the time, so ruling through fear is the next best thing - yet one should carefully avoid being hated. > Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization. As a basic principle, definitely. Though no simple rule can solve all moral dilemmas.
simonh 12y ago
> ...They aren't interested in what's good for you, they are interested in what's good for themselves:... Very true, but then that's why the aversarial justice system is so important. It's ok for prosecutors and police to use every means legally at their disposal to try and prove someone guilty, as long as that person has an independent advocate with sufficient authority to protect the accused and challenge the prosecution case. The problem here is that the way these people were treated was perfectly legal and even morally acceptable at the time. Clearly lengthy unecessary periods of solitary confinement, drugs and agressive interrogation without an advocate present should be completely unacceptable. They even used simulated drowning. The paraelells to Abu Ghraib, Guantanamo, the treatment of Bradley Manning and CIA waterboarding are stark. It's pretty clear that these kind of abuses simply do not work. People subjected to these treatments will say, and even come to believe absolutely anything, so any information that comes out of such processes is worthless. That's completely aside from the deep immorality of using such techniques in the first place. ALl it does is provides moral cover and justification for oppressive regimes to use such techniques completely indiscriminately.
- arethuza 12y ago
  Is the adversarial system really any better or worse than the inquisitorial system (I believe they use the latter in Iceland): http://en.wikipedia.org/wiki/Adversarial_system http://en.wikipedia.org/wiki/Adversarial_system http://en.wikipedia.org/wiki/Inquisitorial_system http://en.wikipedia.org/wiki/Inquisitorial_system
  - walshemj 12y ago
    As the inquisitorial system came out of the inquisition NO is the obvious answer.
- roel_v 12y ago
  <snip> agressive interrogation without an advocate present should be completely unacceptable <snip> LOL, that is the standard in much of Europe today. That is of course not incongruent with what you said - it just nuances your 'at the time'.
- 12y ago

moconnor 12y ago

This is an interesting insight into the dangers of ill-chosen metrics. It could only happen in a system in which the police were expected to produce (and were in various ways rewarded for) convictions instead of e.g. a maximally accurate presentation of events that can be used to decide how society should respond. Convictions are far easier to measure (and achieve, of course). How would you go about constructing an efficient, corruption-resistant system that is measured by its accuracy? It seems like a hard problem, but not necessarily an unsolvable one. Debugging support issues sometimes feels like this. If you go into a complex system looking for bugs then you'll find plenty - but not necessarily the one that's causing the current issue. What you're really looking for is to fully understand the reason the system does what it does. And then you can change it. Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Kind of difficult to study that. Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization.

Centigonal 12y ago
A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize toward the metric, and it'll stop correlating. Some reading: http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/ http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/
- jacquesm 12y ago
  See also: the breakdown of the link structure of the web because of the rise of google.
brazzy 12y ago
> Also: further evidence that information produced under torture cannot be relied upon, yet our states still do this. Perhaps sometimes it works? Or is it just very successful in producing "information" that the system wants and expects to receive, which is then interpreted as an objective success that validates the method? Both, really. There is certainly a small class of scenarios where it does work, but you cannot know in advance where your case is in that class. And is great at making it look like you're "getting results" if you don't look to closely. Another factor: torture is definitely good for maintaining a "reign of terror" style of regime. Until it isn't. I think there's even a passage in Macchiavelli's "The Prince" (i.e. the opus that basically coined the "end justify means" meme) that says ruling people by making them love you is most effective but impossible to do all the time, so ruling through fear is the next best thing - yet one should carefully avoid being hated. > Perhaps an "ends cannot be used to justify means" principle is simply a necessary precondition for benevolent civilization. As a basic principle, definitely. Though no simple rule can solve all moral dilemmas.

simonh 12y ago

> ...They aren't interested in what's good for you, they are interested in what's good for themselves:... Very true, but then that's why the aversarial justice system is so important. It's ok for prosecutors and police to use every means legally at their disposal to try and prove someone guilty, as long as that person has an independent advocate with sufficient authority to protect the accused and challenge the prosecution case. The problem here is that the way these people were treated was perfectly legal and even morally acceptable at the time. Clearly lengthy unecessary periods of solitary confinement, drugs and agressive interrogation without an advocate present should be completely unacceptable. They even used simulated drowning. The paraelells to Abu Ghraib, Guantanamo, the treatment of Bradley Manning and CIA waterboarding are stark. It's pretty clear that these kind of abuses simply do not work. People subjected to these treatments will say, and even come to believe absolutely anything, so any information that comes out of such processes is worthless. That's completely aside from the deep immorality of using such techniques in the first place. ALl it does is provides moral cover and justification for oppressive regimes to use such techniques completely indiscriminately.

arethuza 12y ago
Is the adversarial system really any better or worse than the inquisitorial system (I believe they use the latter in Iceland): http://en.wikipedia.org/wiki/Adversarial_system http://en.wikipedia.org/wiki/Adversarial_system http://en.wikipedia.org/wiki/Inquisitorial_system http://en.wikipedia.org/wiki/Inquisitorial_system
- walshemj 12y ago
  As the inquisitorial system came out of the inquisition NO is the obvious answer.
roel_v 12y ago
<snip> agressive interrogation without an advocate present should be completely unacceptable <snip> LOL, that is the standard in much of Europe today. That is of course not incongruent with what you said - it just nuances your 'at the time'.
12y ago