3 ms·

You should trust the "complicated" algorithm far more than you trust the complicated human brain of the judge, who was also trained on unreliable data. Look it

You should trust the "complicated" algorithm far more than you trust the complicated human brain of the judge, who was also trained on unreliable data. Look it's easy to verify whether parole officers are better at predicting recidivism than an algorithm. If the algorithm is objectively better than it should be used.

kragen 10y ago
Given an unbiased algorithm that does better at predicting recidivism, it would be easy to deliberately construct an algorithm that does almost as much better, but is egregiously biased. For example, if you had been wronged by somebody named Thiel, you could persuade it to never recommend parole for anybody named Thiel. There aren't enough people named Thiel for this to substantially worsen its measured performance. Given that it's easy to construct an example of how you could deliberately do this, and it's so easy to accidentally overfit machine-learning algorithms, we should be very concerned about people accidentally doing this. An easy way would be to try a few thousand different algorithm variants and have a biased group of people eyeball the results to see which ones look good. If those people are racist, for example, they could subconsciously give undue weight to type 1 errors for white prisoners and type 2 errors for black prisoners, or vice versa. The outcome of the process would be an algorithm that is "objectively better" by whatever measure you optimize it for, but still unjustly worsens the situation for some group of people. A potential advantage of algorithms and other rules is that, unlike the brain of the judge, they can be publicly analyzed, and the analyses debated. This is the basis for the idea of "rule of law". Aside from historical precedents, though, the exploitation of the publicly-analyzed DAO algorithms should give us pause on this count. Deeper rule of law may help, but it could easily make the situation worse. We must be skeptical, and we must do better.

kragen 10y ago

Given an unbiased algorithm that does better at predicting recidivism, it would be easy to deliberately construct an algorithm that does almost as much better, but is egregiously biased. For example, if you had been wronged by somebody named Thiel, you could persuade it to never recommend parole for anybody named Thiel. There aren't enough people named Thiel for this to substantially worsen its measured performance. Given that it's easy to construct an example of how you could deliberately do this, and it's so easy to accidentally overfit machine-learning algorithms, we should be very concerned about people accidentally doing this. An easy way would be to try a few thousand different algorithm variants and have a biased group of people eyeball the results to see which ones look good. If those people are racist, for example, they could subconsciously give undue weight to type 1 errors for white prisoners and type 2 errors for black prisoners, or vice versa. The outcome of the process would be an algorithm that is "objectively better" by whatever measure you optimize it for, but still unjustly worsens the situation for some group of people. A potential advantage of algorithms and other rules is that, unlike the brain of the judge, they can be publicly analyzed, and the analyses debated. This is the basis for the idea of "rule of law". Aside from historical precedents, though, the exploitation of the publicly-analyzed DAO algorithms should give us pause on this count. Deeper rule of law may help, but it could easily make the situation worse. We must be skeptical, and we must do better.