3 ms·

"A key takeaway is that our algorithms are not impartial." Of course they are, they just underperform on the data points from the long tail of the empirical dis

"A key takeaway is that our algorithms are not impartial." Of course they are, they just underperform on the data points from the long tail of the empirical distribution, as show basically all real-world examples from the paper. Underfitting on the samples from the long tale (aka blinding the model to ethically bad features) removes bias sensu strictu but increases variance, which leads to more errors, which is perceived as "bias" (did someone say catch 22?).

geofft 5y ago
The classic algorithm to solve the https://en.wikipedia.org/wiki/Stable_marriage_problem https://en.wikipedia.org/wiki/Stable_marriage_problem is a pretty straightforward and frankly introductory-level example of an algorithm which is not impartial (and not just because it ought to be called the stable heterosexual monogamous marriage problem). The algorithm outputs a matching between two sets A and B (e.g., bachelors and bachelorettes) where each member of set A has ranked each member of set B in preference order and vice versa. The output is called stable because there's no situation where any pair would prefer to be with each other than their current partners, e.g., there's never a case where Alice prefers Bob to her current partner and Bob prefers Alice to his current partner and are tempted to run away with each other. But a curious property of this algorithm is that the matching is always best for one group and always worst for the other. Of the possible answers to the problem, this algorithm an answer that is at least as good for set A as any other one, and at least as bad for set B as any other one. You can, of course, flip sets A and B when running the algorithm, and it will them be optimal for set B and pessimal for set A. This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm. And yet it is clearly partial, and you must choose when using it whether to be partial to set A or set B, to the men or to the women, to med school students or to their employers, etc. It would be silly to say that the algorithm is impartial and that it's solely the operator's choice to make it impartial - this particular algorithm forces the operator to make some choice. I think in the end you're agreeing with the author, who is saying that you must make a tradeoff of some sort, and the question is which tradeoff to choose. The author (and I) would describe that as the algorithm being partial, and the operator only being able to make some choice about what to do with that.
- macleginn 5y ago
  "This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm." --- that's why I wouldn't use it as an example to analyse the question under discussion. If we disregard the name of the algorithm, it's just a non-commutative function: there are lots of those in CS and data analysis (e.g., KL divergence), and people have more or less learned how to deal with them. Generally, I do not agree with the author in what they call "bias". Sometimes algorithms are too noisy for particular data subsets; quite often they are over-confident in their predictions, thus exacerbating the differences between small and large sub-populations. These are all _technical_ issues that surely need to be taken into account when making decisions based on systems' outputs and when designing new systems, but the general idea of the ethical AI literature seems to be to recast these technical issues as ethical issues and to ask AI people to strive to eliminate those altogether. I think that this is misguided and very unlikely to work.
- twic 5y ago
  This feels a bit like saying that subtraction is partial because it always treats its first argument as positive and its second as negative. It's important to be aware of the asymmetry in the Gale-Shapley algorithm (and i wasn't - thank you), and to not accidentally (or intentionally!) use it in an unfair way. But if someone does, it is they who have introduced the partiality, not the algorithm. PS It seems there are at least two algorithms for solving the stable marriage problem equitably, one given in this paper, and one it mentions given by Knuth and attributed to Selkow: https://epubs.siam.org/doi/10.1137/0216010 https://epubs.siam.org/doi/10.1137/0216010
Barrin92 5y ago
> Of course they are there are plenty of cases where the 'of course' does not apply. In computer vision for example models generally have a bias towards scale invariance but struggle with rotational invariance. This is not a problem of 'bad data', but of the kind of features that the architecture is prone to extract and represent while it struggles with others. That's why there is an entire zoo of different ML architectures and systems, because there is no magical uniform, algorithm that performs equally well in every domain. CNN's excel at spatial data, while RNNs perform better on sequential data, and so on. Of course one can attempt to reframe this as a 'data problem' and argue that just means you need to input 100x more data of X than of Y, but that actually just shows that algorithm performance is not uniform, and the more productive thing would actually be to understand strengths and weaknesses of architecture.
- deleted 5y ago
  [deleted]
- macleginn 5y ago
  How is this relevant for the notion of bias as discussed in the paper? Of course different approaches have different weak points.

geofft 5y ago

The classic algorithm to solve the https://en.wikipedia.org/wiki/Stable_marriage_problem https://en.wikipedia.org/wiki/Stable_marriage_problem is a pretty straightforward and frankly introductory-level example of an algorithm which is not impartial (and not just because it ought to be called the stable heterosexual monogamous marriage problem). The algorithm outputs a matching between two sets A and B (e.g., bachelors and bachelorettes) where each member of set A has ranked each member of set B in preference order and vice versa. The output is called stable because there's no situation where any pair would prefer to be with each other than their current partners, e.g., there's never a case where Alice prefers Bob to her current partner and Bob prefers Alice to his current partner and are tempted to run away with each other. But a curious property of this algorithm is that the matching is always best for one group and always worst for the other. Of the possible answers to the problem, this algorithm an answer that is at least as good for set A as any other one, and at least as bad for set B as any other one. You can, of course, flip sets A and B when running the algorithm, and it will them be optimal for set B and pessimal for set A. This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm. And yet it is clearly partial, and you must choose when using it whether to be partial to set A or set B, to the men or to the women, to med school students or to their employers, etc. It would be silly to say that the algorithm is impartial and that it's solely the operator's choice to make it impartial - this particular algorithm forces the operator to make some choice. I think in the end you're agreeing with the author, who is saying that you must make a tradeoff of some sort, and the question is which tradeoff to choose. The author (and I) would describe that as the algorithm being partial, and the operator only being able to make some choice about what to do with that.

macleginn 5y ago
"This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm." --- that's why I wouldn't use it as an example to analyse the question under discussion. If we disregard the name of the algorithm, it's just a non-commutative function: there are lots of those in CS and data analysis (e.g., KL divergence), and people have more or less learned how to deal with them. Generally, I do not agree with the author in what they call "bias". Sometimes algorithms are too noisy for particular data subsets; quite often they are over-confident in their predictions, thus exacerbating the differences between small and large sub-populations. These are all _technical_ issues that surely need to be taken into account when making decisions based on systems' outputs and when designing new systems, but the general idea of the ethical AI literature seems to be to recast these technical issues as ethical issues and to ask AI people to strive to eliminate those altogether. I think that this is misguided and very unlikely to work.
twic 5y ago
This feels a bit like saying that subtraction is partial because it always treats its first argument as positive and its second as negative. It's important to be aware of the asymmetry in the Gale-Shapley algorithm (and i wasn't - thank you), and to not accidentally (or intentionally!) use it in an unfair way. But if someone does, it is they who have introduced the partiality, not the algorithm. PS It seems there are at least two algorithms for solving the stable marriage problem equitably, one given in this paper, and one it mentions given by Knuth and attributed to Selkow: https://epubs.siam.org/doi/10.1137/0216010 https://epubs.siam.org/doi/10.1137/0216010

Barrin92 5y ago

> Of course they are there are plenty of cases where the 'of course' does not apply. In computer vision for example models generally have a bias towards scale invariance but struggle with rotational invariance. This is not a problem of 'bad data', but of the kind of features that the architecture is prone to extract and represent while it struggles with others. That's why there is an entire zoo of different ML architectures and systems, because there is no magical uniform, algorithm that performs equally well in every domain. CNN's excel at spatial data, while RNNs perform better on sequential data, and so on. Of course one can attempt to reframe this as a 'data problem' and argue that just means you need to input 100x more data of X than of Y, but that actually just shows that algorithm performance is not uniform, and the more productive thing would actually be to understand strengths and weaknesses of architecture.

deleted 5y ago
[deleted]
macleginn 5y ago
How is this relevant for the notion of bias as discussed in the paper? Of course different approaches have different weak points.