3 ms·

It strikes me that longer term, we're just going to end up playing a cat and mouse game with people continually inventing different categories that businesses a

It strikes me that longer term, we're just going to end up playing a cat and mouse game with people continually inventing different categories that businesses aren't permitted to 'discriminate' on. Today it might be gender and race. That makes a lot of sense, because the alternative is to further entrench what are basically inheritances. But aren't we just going to rattle on through and have the algorithms discover (whether we actually realise it or not) that, say, someone diagnosed with X is less creditworthy than someone diagnosed with Y, or that someone bullied in school is less creditworthy, or whatever else? The whole point of ML is to extract this sort of information from a dataset. Is it even possible or meaningful to create an unbiased model? Doesn't a model's profit imply bias, whether we currently consider it morally correct or not? I'd be interested in an argument to convince me otherwise. My view at the moment is basically 'we spend all of this time building models, and then we have to stop using them because they're socially negative/immoral, but for a brief period shareholder value was maximised'?

matthewmacleod 7y ago
That’s not a great argument though - essentially “we can’t prevent an ML model from being biased so we should embrace it.” Western society has mostly accepted the idea that—outside of some specific cases—we should avoid building systems and processes that systematically discriminate against people on the basis of a selection of characteristics which have historically attracted it. The exact application of this concept, the interpretation of it, and the boundaries of discrimination or protected characteristics will continue to be subject to gray areas and refinement. The rules are not perfect. But I do think a better solution to the problem (“we have implemented a whizz-bang new technology which is inherently subject to bias”) is to either fix or discard that technology, rather than discarding the concepts of equalities regulation and civil rights.
- esotericn 7y ago
  I think you've mistaken me. My argument is that if we take the standpoint that we don't just want the metric to be whatever is short-term economically optimal for the designer of the model, we should just stop/ban it now, because we already know that we're going to have to kill it once we actually understand it properly. It's only allowed now because we haven't figured out the bad things that are happening. Inventing more and more categories that businesses are not permitted to discriminate upon is precisely the wrong approach - if we're talking about huge companies and not the bakery down the road, we need something that's more like 'you need a very good reason to exclude someone', rather than 'you can exclude someone for whatever reason they like, unless they're a member of the set of continually extending list of protected categories'.

matthewmacleod 7y ago

That’s not a great argument though - essentially “we can’t prevent an ML model from being biased so we should embrace it.” Western society has mostly accepted the idea that—outside of some specific cases—we should avoid building systems and processes that systematically discriminate against people on the basis of a selection of characteristics which have historically attracted it. The exact application of this concept, the interpretation of it, and the boundaries of discrimination or protected characteristics will continue to be subject to gray areas and refinement. The rules are not perfect. But I do think a better solution to the problem (“we have implemented a whizz-bang new technology which is inherently subject to bias”) is to either fix or discard that technology, rather than discarding the concepts of equalities regulation and civil rights.

esotericn 7y ago
I think you've mistaken me. My argument is that if we take the standpoint that we don't just want the metric to be whatever is short-term economically optimal for the designer of the model, we should just stop/ban it now, because we already know that we're going to have to kill it once we actually understand it properly. It's only allowed now because we haven't figured out the bad things that are happening. Inventing more and more categories that businesses are not permitted to discriminate upon is precisely the wrong approach - if we're talking about huge companies and not the bakery down the road, we need something that's more like 'you need a very good reason to exclude someone', rather than 'you can exclude someone for whatever reason they like, unless they're a member of the set of continually extending list of protected categories'.