4 ms·

You could do automatic differentiation. Take each input, race, sex, age, ... and see how the output changes. If one field has a huge swing it might be a good ba

by crb002 10y ago

You could do automatic differentiation. Take each input, race, sex, age, ... and see how the output changes. If one field has a huge swing it might be a good basis for claiming variable X based discrimination.

a3n 10y ago
And if there's not a small number of perturbations, then it's the proprietary combination of a large number of small, individual perturbations. But either way, since it's proprietary, it's not practically refutable. Not to mention you'd never get access to do the differentiation, as it would rely on access to the proprietary process; the company would have to be compelled to cooperate.
BearGoesChirp 10y ago
Sex would have a major impact. At least in the US our justice system is extremely sexist, far more than it is racist. I once saw sex offender guidelines that automatically put the default sex offender sentence as medium for males and low for females (the default was used before an official evaluation was completed, and I would bet some money that the official evaluations discriminated based on sex). While I don't disagree our legal system is also racist, the racism pales in comparison to the current level of sexism.
- dTal 10y ago
  That's not a bug, it's a feature! User jawns posted some of their marketing copy elsewhere in this thread: "COMPAS offers separate norms for males, females, community and incarcerated populations" And they call that "defensible decisions"! In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird".
  - a3n 10y ago
    > In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird". Isn't that what judges do when they follow precedent?
    - cryptarch 10y ago
      Legal precedents tend not to explicitly single out one gender, race or religion.
    - dragonwriter 10y ago
      No, and judges don't always follow precedent (they theoretically have to when it is binding, but no precedent is binding on the highest court, even it's own.) When they follow precedent, they are not matching outcomes to outcomes to avoid looking wierd, they are applying what was previously established as the formal decision rule. That's a very different thing that attempting to preserve biases that may not even reflect the formally-annoubced decision rules in the cases that produced them, but may represent silent deviations from the formal rules.

by crb002 10y ago

a3n 10y ago

And if there's not a small number of perturbations, then it's the proprietary combination of a large number of small, individual perturbations. But either way, since it's proprietary, it's not practically refutable. Not to mention you'd never get access to do the differentiation, as it would rely on access to the proprietary process; the company would have to be compelled to cooperate.

BearGoesChirp 10y ago

Sex would have a major impact. At least in the US our justice system is extremely sexist, far more than it is racist. I once saw sex offender guidelines that automatically put the default sex offender sentence as medium for males and low for females (the default was used before an official evaluation was completed, and I would bet some money that the official evaluations discriminated based on sex). While I don't disagree our legal system is also racist, the racism pales in comparison to the current level of sexism.

dTal 10y ago
That's not a bug, it's a feature! User jawns posted some of their marketing copy elsewhere in this thread: "COMPAS offers separate norms for males, females, community and incarcerated populations" And they call that "defensible decisions"! In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird".
- a3n 10y ago
  > In other words, they want to make sure the software doesn't rock the boat. If there's a bias, they want to preserve it so it doesn't look like their algorithms are "weird". Isn't that what judges do when they follow precedent?
  - cryptarch 10y ago
    Legal precedents tend not to explicitly single out one gender, race or religion.
  - dragonwriter 10y ago
    No, and judges don't always follow precedent (they theoretically have to when it is binding, but no precedent is binding on the highest court, even it's own.) When they follow precedent, they are not matching outcomes to outcomes to avoid looking wierd, they are applying what was previously established as the formal decision rule. That's a very different thing that attempting to preserve biases that may not even reflect the formally-annoubced decision rules in the cases that produced them, but may represent silent deviations from the formal rules.