"This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm." --- that's why I wouldn't use it as an examp

"This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm." --- that's why I wouldn't use it as an example to analyse the question under discussion. If we disregard the name of the algorithm, it's just a non-commutative function: there are lots of those in CS and data analysis (e.g., KL divergence), and people have more or less learned how to deal with them. Generally, I do not agree with the author in what they call "bias". Sometimes algorithms are too noisy for particular data subsets; quite often they are over-confident in their predictions, thus exacerbating the differences between small and large sub-populations. These are all _technical_ issues that surely need to be taken into account when making decisions based on systems' outputs and when designing new systems, but the general idea of the ethical AI literature seems to be to recast these technical issues as ethical issues and to ask AI people to strive to eliminate those altogether. I think that this is misguided and very unlikely to work.