3 ms·

> Your list is predominantly racial and I thought that was the point. Ah, no, that wasn't the point. They were just examples of systemic bias. The fact that so

> Your list is predominantly racial and I thought that was the point. Ah, no, that wasn't the point. They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in. > A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous. I honestly don't know how any of the examples posted are deniable, they're all well founded with research and evidence. But you're right that adding more to the list can only help. > Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that. That's not what systemic bias means. "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes." https://en.wikipedia.org/wiki/Systemic_bias https://en.wikipedia.org/wiki/Systemic_bias Also, examples are evidence which support a theory (of systemic bias in this case). Any one data point is rarely intended to be absolute proof of that theory, much less a description of its scope in its entirety. All of these examples do indeed provide evidence toward the existence of systemic bias within systems without fully describing or proving the extent of the entire scope of systemic bias within those systems. > But not serving a sentence for crime is just as bad, if not worse. The article I linked explains that this is not what the algorithm was used for. It was not used to decide whether or not to sentence them, it was used to help decide the severity of their sentence. > If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse. You seem to be agreeing that this was in fact a badly biased algorithm, since it was recommending a stiffer punishment for a kid who attempted to steal a bike than for a repeat felon, previously convicted of armed robbery, who shoplifted. > It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior. If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad; see the description above. Regardless of whether or not you think stealing scooters is a good prediction, the fact here is that it rated the likelihood of future crime as higher than for someone who was already a career criminal. > Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random. If only. Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.

Rule35 6y ago
> They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in. Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues? The medical one made me think about the similar case of differing heart-attack symptoms in men and women. In this case actually because doctors just focused on men. > That's not what systemic bias means. > "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes." That just shifts the definitional question to the scale of the institution, structure, or process. At one end, if everyone follows the same rules it's a system, but what about if one clinic got bad data, do we write that up as a systematic failure? That's okay if it's your definition, but cases at the other end seem a lot more important and a lot more amenable to systematic fixes. > If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population? I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc. Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems. So I was wondering if you'd focused on black issues to make the list. > Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list. So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected? I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror. > You seem to be agreeing that this was in fact a badly biased algorithm No. But not a good one either. More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell. The 'good' case would be if someone built a criminality model and the city was trying to work with police and communities to intervene in a predicted pattern. It's not unreasonable that the societal harm from a non-criminal becoming criminal could be worse than an existing criminal remaining that way. So modeling and discussing this isn't bad, even if the data has a racial component and some of the questions are of bias. But yeah, to recommend sentences, total crap.

Rule35 6y ago

> They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in. Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues? The medical one made me think about the similar case of differing heart-attack symptoms in men and women. In this case actually because doctors just focused on men. > That's not what systemic bias means. > "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes." That just shifts the definitional question to the scale of the institution, structure, or process. At one end, if everyone follows the same rules it's a system, but what about if one clinic got bad data, do we write that up as a systematic failure? That's okay if it's your definition, but cases at the other end seem a lot more important and a lot more amenable to systematic fixes. > If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population? I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc. Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems. So I was wondering if you'd focused on black issues to make the list. > Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list. So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected? I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror. > You seem to be agreeing that this was in fact a badly biased algorithm No. But not a good one either. More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell. The 'good' case would be if someone built a criminality model and the city was trying to work with police and communities to intervene in a predicted pattern. It's not unreasonable that the societal harm from a non-criminal becoming criminal could be worse than an existing criminal remaining that way. So modeling and discussing this isn't bad, even if the data has a racial component and some of the questions are of bias. But yeah, to recommend sentences, total crap.