4 ms·
There's an interesting assumption baked into this entire approach, which is that the human-evaluated system is better. I'll be interested to see if that's an ar
by shadowgovt 7y ago
There's an interesting assumption baked into this entire approach, which is that the human-evaluated system is better. I'll be interested to see if that's an artifact of this summary article or is reflected in the underlying reporting coming up, because it's wise to not leave that assumption unchallenged. FTA:
> Instead of talking to a caseworker who personally assesses your needs, you now are channeled online where predictive analytics will assign you a future risk score and an algorithm decide your fate.
If the caseworker is racist, this isn't a worse scenario... unless such racism has also been baked into the risk assessment algorithm, which is definitely a possibility. But at least from my personal experience, I trust our ability to identify and de-train racism from assessment networks more than I do our ability to consistently identify and hire non-racist caseworkers. Deprogramming racism out of humans is hard (and even if you succeed for one, you can't copy their state-vector into the brains of their peers).
- fwip 7y agoOne benefit is that repeated visits to a government office generally result in you meeting a different person, or the same person but in a good mood this time. Another thing about racism in humans is that it's easier to detect than once it's codified in a technological black box.
- chongli 7y agoI recall reading a story in the book Weapons of Math Destruction [1] about a system used to assess people’s recidivism risk which judges were relying on for sentencing hearings. The problem is that the system showed a clear racial disparity in the risk scores yet at the same time it was more accurate. That means if we try to tune the system to make it less racist, we’ll be making it less accurate. In essence, the system isn’t really racist, it’s a reflection of the racism in society which is leading to these outcomes. Ultimately, the problem is that putting someone in jail increases the likelihood that their relatives will commit crimes. It increases the likelihood that both they and their friends and family will reoffend. It’s a vicious cycle and it doesn’t appear to have any technical solution. [1] https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
- Nasrudith 7y agoIt brings to mind a sarcastic way to get perfect prediction of patient outcomes. Step one is to decapitate the patient. Predictability/accuracy isn't the important part except as means to the end - the outcome.
- vokep 7y agoThat system was the result of asking the system one question "How likely is this person to be arrested?" and taking the answer as an answer to an entirely different question "How likely is this person to commit the crime again?"
- KineticLensman 7y agoCompared with human case workers, algorithms apply any encoded incompetence and / or bias at potentially massive scale
- shadowgovt 7y agoAn army of social workers raised in the same culture will on average do the same, and be much harder to correct once bias is identified.
- Faark 7y ago> An army of social workers raised in the same culture will on average do the same, In a society were it is normalized, sure. But can you imagine what systems such a society would come up with? So far we are only talking about shitty automated systems accidentally producing unwanted outcomes by e.g. reflecting subtle / unwitting / intolerable issues. > and be much harder to correct once bias is identified At the end of the day, I'd expect most social workers wanting to help and do a good job at it. AI hasn't reached that level, yet. So I'd expect a much better baseline from humans, with "fixing" outliers being much harder, yes.