52 ms·

A Discriminatory Algorithm Wrongly Accused Thousands of Families of Fraud

92 points by JeanMarcS 6y ago

34679 6y ago
"Henk and Ingrid pay for Mohammed and Fatima.” I really hope that at some point, the world starts talking about how Mohammed and Fatima ended up there. This is a part of what happens when you spend two decades bombing villages.
- tgv 6y ago
  Mohammed and Fatima represent Morrocans. Muslims, second generation immigrants, but not from the Middle East. Wilders runs an anti-immigration party.
  - Rule35 6y ago
    Most of Europe does now. Governments aren't keeping the locals or other foreigners safe because of blatantly racist policies. But it's wrong to simply say 'anti-immigration', it's for a more screened immigration process for places from where people are seen to have trouble fitting in. Almost everybody (95%+) is totally accepting of people who (intend to, even) speak the language and accept the laws, regardless of what they look like. Moreover, the sentiment is shared by most demographics. They respect their homes and don't want abusers here simply because of some skin-color in common. They love their children and don't want them stolen and prostituted. The UK has actual rape syndicates now. Thousands of people running human trafficking rings. Left almost unmolested (hah!) and undocumented because of the origin of the predators. This is fact. The people on the ground just want their kids to be safe but by reporting the crimes they're accused of being racist. It's really a religion problem, and only the sort that comes from a few of the worst countries, but racist is a better slander word and costs the accuser nothing (ie, burden of proof) in today's climate. Worse, the government that fails to say 'Muslim Pakistani males' is willing to say 'South Asian rape gangs' which slanders a lot more innocent people while totally denying the problem. This lie makes Qatari men (also Muslim) look bad despite generally being as civilized as anyone, as well as non-Muslim Indians and Balinese, etc. Literally 1/3rd of the world, to avoid naming and fixing the problem. And then there's the "criticism of islam" thing. Nobody is willing to accept more of that since the recent attacks in fall 2020. Americans just try to see everything through a racist lens.
    - anoncake 6y ago
      So you think bigotry based on someone's religion shouldn't be considered racism. You may be technically right, but that's a bit of a strange hill to die on. And yes, it is bigotry. Islam may be awful, but so is Christianity. Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
      - Rule35 6y ago
        Right, race and religion are not the same. And no, it's the hill from which to not die. How are you doing over there on rocky Mt Everyone-is-Identical? My discrimination is your bigotry when you're trying to make me the bad guy. But yes, I do reserve the right to judge things for myself. You obviously do to, and have lashed into my views simply because they differ, not because you've shown them to be incorrect. In an area where a vast majority of a class of criminals share a combination of home country and religion it makes perfect sense to want to screen them more thoroughly, because the only other option is every citizen having to be race-conscious when hiring men with access to children. To make this specific situation worse, the governments refusal to acknowledge it means that the criminals aren't getting an appropriate conspiracy charge because that would have to acknowledge the gang nature which would then demonstrate the race (if not religion) and as such, they face minor charges (ex 3y for kidnapping and gang rape) that don't actually stop the problem and then release them back into their communities where they poison others. A gang charge would also prevent them associating with their past conspirators. > Islam may be awful, but so is Christianity. One, your moral relativism is disgusting. There are obvious differences between all religions and trying to one-size-fits-all totally craps on people whose faiths aren't violent or abhorrent, or whose churches have practiced differently for long enough to be recognized. Two, I'm not making the claim that either is bad or good, but that islam and pakistan together are a huge red flag. Whether that means Pakistani islam is different is irrelevant in the context of protecting children. > Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society. Yes, fortunately. But what do we do about those who don't? Close our eyes and pretend they do?
- danaris 6y ago
  I mean, the plight of refugees is certainly a good reason not to mistreat and discriminate against immigrants, but it doesn't matter how they ended up there, you still shouldn't be discriminating against them because of their nationality, their religion, their names. No one deserves to be the target of racism, no matter how bad or good their circumstances are. If someone has done something worthy of derision, then justly revile them for that, but leave their race, ethnicity, religion, sex, gender, orientation, disability, etc out of it.
- oh_sigh 6y ago
  70% of non-native Muslims in the Netherlands are from Morocco and Turkey. When was the last two decade bombing campaign against those countries?
  - 34679 6y ago
    Their numbers will be higher because they've been immigrating since the 60's.[1] In the last decade, Morocco isn't even in the top 10.[2] Most of the asylum seekers are coming from Syria.[3] [1]https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-sources-immigrants-netherlands https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-... [2]https://www.statista.com/statistics/525804/netherlands-largest-groups-of-immigrants-by-nationality/ https://www.statista.com/statistics/525804/netherlands-large... [3]https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-seekers-and-following-relatives https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-see...
- avs733 6y ago
  two decades? Two centuries.
Tarq0n 6y ago
The article is a lot better than the headline lets on. In particular: > Crucially, she said, we should place blame on the human individuals behind the creation and use of the algorithm rather than reify the technology as being the main driver. > “Systems and algorithms are human-made, and do exactly what they’ve been instructed to do,” she said. “They can act as an easy and sort of cowardly way to take the blame off yourself. What’s important to understand is that often algorithms can use historical data without any proper interpretation of the context surrounding that data. With that pattern, you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop that increases social problems that already exist.” As someone in an adjacent institution this affair has been frustrating to watch. In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors. Interestingly though the administration resigned over this affair, it does not look like they'll suffer much over it in the upcoming election this wednesday. The responsible prime minister is likely to get a fifth term if things go as the polls are predicting.
- tchalla 6y ago
  > In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors. We learned from 2000 years of human history - war, disease etc. - to create a transparent system of checks and balances. These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency". I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
  - Rule35 6y ago
    > These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency". Well, the same 'we' that push voting machines and other crap like locking government records up in proprietary software. I push for algorithms specifically because they're (as a written document) transparent. If a human has to judge a historical context you're never going to get repeatable results. But I want the system to be able to diagram, with weights, all the inputs that went into something before I'll trust an automated version.
Glavnokoman 6y ago
"inherently discriminatory because they took variables such as whether someone had a second nationality into account". Which is not necessarily wrong. If people with a second nationality are statistically more prone to cheating the system this is a valid variable to determine the probability that they actually cheat and not made a mistake. What would be wrong though is to blindly take that probability for the fact. But it is really hard to say what actually is going on there cause the article is full of emotions and carries close to zero information.
- zwkrt 6y ago
  Nope, actually it is wrong! Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant. If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge. This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops. Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not! The moral of the story is that stats describing a cohort cannot be projected onto individuals in that cohort. The data flows the other way; individuals comprise an average, no individual /conforms/ to an average. They may happen to coincidentally be average, but no causation has occurred. Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
  - shitpostbot 6y ago
    You're both saying the same thing, statistics are not evidence of guilt
  - Udik 6y ago
    The GP clearly wrote: "What would be wrong though is to blindly take that probability for the fact." And it's right: your algorithms can use all sorts of indicators to flag suspicious cases, but in the end the evaluation needs to be done by a person on a case by case basis. As long as the investigation on the single case is performed "silently" (i.e. without causing any loss of time or money or any annoyance to the person who is investigated) I don't see the issue. Otherwise it's the same as saying you cannot sort rows in an excel sheet according to some criterion and start looking at the data from where it looks more promising.
gambiting 6y ago
This paragraph, holy shit - I have to wonder if whoever wrote it(the algorithm) can look at themselves in the mirror. Or if they just think it's all for the greater good and it's mostly targetting unwashed masses so it's all fine. "In one of the more egregious examples of the lack of humanity in the authorities’ approach, a report from Trouw revealed that the tax office had baselessly applied the mathematical Pareto principle to their punishments, assuming without evidence that 80 percent of the parents investigated for fraud were guilty and 20 percent were innocent. "
- stonecraftwolf 6y ago
  Isnt that also just not how the Pareto principle works? Doesn’t it say that 20% of inputs (people accused of fraud) are responsible for 80% of outputs (cases of fraud)? So this is actually exactly backwards. On top of being frighteningly amoral, I mean.
  - leto_ii 6y ago
    I checked the original article, it doesn't explicitly refer to the Pareto principle, it just calls it the 80/20 rule. Normally by this it's understood that you should be referring to the Pareto principle, but it seems they've invented another 80/20 rule (that makes no sense and is not related to Pareto).
  - kgin 6y ago
    Yeah if that's really what they're doing, they've misunderstood the pareto principle pretty much on every level, from what it is to how someone might attempt to apply it.
- rvanlaar 6y ago
  I'm Dutch and am following this scandal for quite some time. This article strikes me as being odd. Yes, computers were used in the decision process and no I don't believe them to be algorithms. It was specifically coordinated and covered up from the top. As your highlight shows, it was specifically designed this way. Which means it was a business process.
shitpostbot 6y ago
Sounds like what they had was a tool that used correlation to bucket people based on statistics with increased risk of fraud, and for some reason they were using it as proof of fraud. Yes, this is a massive scandal, but why is every article about it focused on spooky *ALGORITHMS* (I have never seen the word used as much as I have in the articles covering this story) instead of the clearly massive fault of the people running this.
urtie 6y ago
Just to get you an idea of how wrongheaded this entire thing was, I have a personal anecdote relating to this shitstorm. Back in autumn 2013 I got a letter stating that I had received too much in child care benefits, with a request to pay it back. On the back of said letter was the text 'If you are still entitled to child care benefits and want to settle this amount against upcoming entitlements, you do not have to do anything'. Given as I was still entitled to benefits, I proceeded to not do anything. A few weeks later I got a reminder, once again stating that if I wanted to settle, I wouldn't have to do a thing. The reminder was followed by a final notice with opportunity to pay, once again with the exact same text on the back of the letter. So, obviously, I once again did nothing. Then we got a 'Dwangbevel in naam des konings', i.e. a writ of execution in the name of the king. Red envelope. No added text. So I had to pay up. Now, as you may well guess, the tax service had already also started settling against the benefits that I was still entitled to! This prompted me to write a strongly worded letter, asking the tax service in no unclear language what I should have done differently, and requesting them to not dock me for that particular money twice. Literally the only reaction to that letter from the tax service was in silently restituting to me the double payment. Now, I'm a reasonably well paid and well educated software developer, who just happens to hold two nationalities. I am perfectly sure that I would not even have had a reminder had I just had the Dutch nationality. I could have paid in the first place and not settled against upcoming benefits; I was just lazy. However, imagine you're scraping by on minimum wage and are put in such a situation. There is no other way you could have acted than I did, but you also would not have been able to pay up to the writ of execution. And there would have been no recourse! Add to that the fact that in many cases, the people hadn't even actually been paid too much, as I had, but merely been tagged as possibly fraudulent and put in to the system for reclaiming paid benefits, so the government could take a stance of being hard against benefits fraud while figuring out if any fraud had taken place in the first place.
JangoSteve 6y ago
Adding this to my running list of algorithms encoding and amplifying systemic bias, like: * A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 * When the dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abstract/2688587 https://jamanetwork.com/journals/jamadermatology/article-abs... * When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/ https://fortune.com/2019/08/16/google-jigsaw-perspective-rac... * When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio... * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though: * When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig https://www.npr.org/2020/06/24/882683463/the-computer-got-it... * When the above is compounded for people of color according to studies which show that facial recognition systems misidentify dark-skinned women 40x more often than for light-skinned men. http://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212 http://news.mit.edu/2018/study-finds-gender-skin-type-bias-a.... Another study showed false positives can be 10x to 100x more frequent for Asian and African American faces compared to Caucasian. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software https://www.nist.gov/news-events/news/2019/12/nist-study-eva... * When an algorithm blocked kidney transplants for Black patients. https://www.wired.com/story/how-algorithm-blocked-kidney-transplants-black-patients/ https://www.wired.com/story/how-algorithm-blocked-kidney-tra... * When clinical algorithms include “corrections” for race which directly raise the bar for the need for interventions in people of color, such that they then receive less clinical screening, less surveillance, less diagnoses, and less treatment for everything, including cancer, organ transplants, birth interventions, urinary and blood, bone, and heart disease. https://www.nejm.org/doi/10.1056/NEJMms2004740 https://www.nejm.org/doi/10.1056/NEJMms2004740
bobdupneu 6y ago
For decades the Netherlands have been helping corporations and the super rich evade taxes at a cost of several hundred billion Euros to the global economy per year. How petty to now attack the poor in such a cowardly manner...
LorenPechtel 6y ago
It sounds to me like the real problem is what appears to be a screening tool being used as evidence of guilt. For other examples: field drug test kits--they really only say "this might be drugs", but the prosecutors use jail to try to get people to plead guilty rather than promptly doing a proper test. Also, the ubiquitous breathalyzer--it's actually coded into law as proof in many places but it's not. There is an inherent biological flaw that makes the readings inaccurate.
FpUser 6y ago
This will only stop when people in the government will be actually punished for the suffering they've caused. Which is mostly never.

92 points by JeanMarcS 6y ago

"Henk and Ingrid pay for Mohammed and Fatima.” I really hope that at some point, the world starts talking about how Mohammed and Fatima ended up there. This is a part of what happens when you spend two decades bombing villages.

tgv 6y ago
Mohammed and Fatima represent Morrocans. Muslims, second generation immigrants, but not from the Middle East. Wilders runs an anti-immigration party.
- Rule35 6y ago
  Most of Europe does now. Governments aren't keeping the locals or other foreigners safe because of blatantly racist policies. But it's wrong to simply say 'anti-immigration', it's for a more screened immigration process for places from where people are seen to have trouble fitting in. Almost everybody (95%+) is totally accepting of people who (intend to, even) speak the language and accept the laws, regardless of what they look like. Moreover, the sentiment is shared by most demographics. They respect their homes and don't want abusers here simply because of some skin-color in common. They love their children and don't want them stolen and prostituted. The UK has actual rape syndicates now. Thousands of people running human trafficking rings. Left almost unmolested (hah!) and undocumented because of the origin of the predators. This is fact. The people on the ground just want their kids to be safe but by reporting the crimes they're accused of being racist. It's really a religion problem, and only the sort that comes from a few of the worst countries, but racist is a better slander word and costs the accuser nothing (ie, burden of proof) in today's climate. Worse, the government that fails to say 'Muslim Pakistani males' is willing to say 'South Asian rape gangs' which slanders a lot more innocent people while totally denying the problem. This lie makes Qatari men (also Muslim) look bad despite generally being as civilized as anyone, as well as non-Muslim Indians and Balinese, etc. Literally 1/3rd of the world, to avoid naming and fixing the problem. And then there's the "criticism of islam" thing. Nobody is willing to accept more of that since the recent attacks in fall 2020. Americans just try to see everything through a racist lens.
  - anoncake 6y ago
    So you think bigotry based on someone's religion shouldn't be considered racism. You may be technically right, but that's a bit of a strange hill to die on. And yes, it is bigotry. Islam may be awful, but so is Christianity. Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
    - Rule35 6y ago
      Right, race and religion are not the same. And no, it's the hill from which to not die. How are you doing over there on rocky Mt Everyone-is-Identical? My discrimination is your bigotry when you're trying to make me the bad guy. But yes, I do reserve the right to judge things for myself. You obviously do to, and have lashed into my views simply because they differ, not because you've shown them to be incorrect. In an area where a vast majority of a class of criminals share a combination of home country and religion it makes perfect sense to want to screen them more thoroughly, because the only other option is every citizen having to be race-conscious when hiring men with access to children. To make this specific situation worse, the governments refusal to acknowledge it means that the criminals aren't getting an appropriate conspiracy charge because that would have to acknowledge the gang nature which would then demonstrate the race (if not religion) and as such, they face minor charges (ex 3y for kidnapping and gang rape) that don't actually stop the problem and then release them back into their communities where they poison others. A gang charge would also prevent them associating with their past conspirators. > Islam may be awful, but so is Christianity. One, your moral relativism is disgusting. There are obvious differences between all religions and trying to one-size-fits-all totally craps on people whose faiths aren't violent or abhorrent, or whose churches have practiced differently for long enough to be recognized. Two, I'm not making the claim that either is bad or good, but that islam and pakistan together are a huge red flag. Whether that means Pakistani islam is different is irrelevant in the context of protecting children. > Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society. Yes, fortunately. But what do we do about those who don't? Close our eyes and pretend they do?
danaris 6y ago
I mean, the plight of refugees is certainly a good reason not to mistreat and discriminate against immigrants, but it doesn't matter how they ended up there, you still shouldn't be discriminating against them because of their nationality, their religion, their names. No one deserves to be the target of racism, no matter how bad or good their circumstances are. If someone has done something worthy of derision, then justly revile them for that, but leave their race, ethnicity, religion, sex, gender, orientation, disability, etc out of it.
oh_sigh 6y ago
70% of non-native Muslims in the Netherlands are from Morocco and Turkey. When was the last two decade bombing campaign against those countries?
- 34679 6y ago
  Their numbers will be higher because they've been immigrating since the 60's.[1] In the last decade, Morocco isn't even in the top 10.[2] Most of the asylum seekers are coming from Syria.[3] [1]https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-sources-immigrants-netherlands https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-... [2]https://www.statista.com/statistics/525804/netherlands-largest-groups-of-immigrants-by-nationality/ https://www.statista.com/statistics/525804/netherlands-large... [3]https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-seekers-and-following-relatives https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-see...
avs733 6y ago
two decades? Two centuries.

Tarq0n 6y ago

The article is a lot better than the headline lets on. In particular: > Crucially, she said, we should place blame on the human individuals behind the creation and use of the algorithm rather than reify the technology as being the main driver. > “Systems and algorithms are human-made, and do exactly what they’ve been instructed to do,” she said. “They can act as an easy and sort of cowardly way to take the blame off yourself. What’s important to understand is that often algorithms can use historical data without any proper interpretation of the context surrounding that data. With that pattern, you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop that increases social problems that already exist.” As someone in an adjacent institution this affair has been frustrating to watch. In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors. Interestingly though the administration resigned over this affair, it does not look like they'll suffer much over it in the upcoming election this wednesday. The responsible prime minister is likely to get a fifth term if things go as the polls are predicting.

tchalla 6y ago
> In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors. We learned from 2000 years of human history - war, disease etc. - to create a transparent system of checks and balances. These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency". I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
- Rule35 6y ago
  > These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency". Well, the same 'we' that push voting machines and other crap like locking government records up in proprietary software. I push for algorithms specifically because they're (as a written document) transparent. If a human has to judge a historical context you're never going to get repeatable results. But I want the system to be able to diagram, with weights, all the inputs that went into something before I'll trust an automated version.

Glavnokoman 6y ago

"inherently discriminatory because they took variables such as whether someone had a second nationality into account". Which is not necessarily wrong. If people with a second nationality are statistically more prone to cheating the system this is a valid variable to determine the probability that they actually cheat and not made a mistake. What would be wrong though is to blindly take that probability for the fact. But it is really hard to say what actually is going on there cause the article is full of emotions and carries close to zero information.

zwkrt 6y ago
Nope, actually it is wrong! Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant. If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge. This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops. Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not! The moral of the story is that stats describing a cohort cannot be projected onto individuals in that cohort. The data flows the other way; individuals comprise an average, no individual /conforms/ to an average. They may happen to coincidentally be average, but no causation has occurred. Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
- shitpostbot 6y ago
  You're both saying the same thing, statistics are not evidence of guilt
- Udik 6y ago
  The GP clearly wrote: "What would be wrong though is to blindly take that probability for the fact." And it's right: your algorithms can use all sorts of indicators to flag suspicious cases, but in the end the evaluation needs to be done by a person on a case by case basis. As long as the investigation on the single case is performed "silently" (i.e. without causing any loss of time or money or any annoyance to the person who is investigated) I don't see the issue. Otherwise it's the same as saying you cannot sort rows in an excel sheet according to some criterion and start looking at the data from where it looks more promising.

gambiting 6y ago

This paragraph, holy shit - I have to wonder if whoever wrote it(the algorithm) can look at themselves in the mirror. Or if they just think it's all for the greater good and it's mostly targetting unwashed masses so it's all fine. "In one of the more egregious examples of the lack of humanity in the authorities’ approach, a report from Trouw revealed that the tax office had baselessly applied the mathematical Pareto principle to their punishments, assuming without evidence that 80 percent of the parents investigated for fraud were guilty and 20 percent were innocent. "

stonecraftwolf 6y ago
Isnt that also just not how the Pareto principle works? Doesn’t it say that 20% of inputs (people accused of fraud) are responsible for 80% of outputs (cases of fraud)? So this is actually exactly backwards. On top of being frighteningly amoral, I mean.
- leto_ii 6y ago
  I checked the original article, it doesn't explicitly refer to the Pareto principle, it just calls it the 80/20 rule. Normally by this it's understood that you should be referring to the Pareto principle, but it seems they've invented another 80/20 rule (that makes no sense and is not related to Pareto).
- kgin 6y ago
  Yeah if that's really what they're doing, they've misunderstood the pareto principle pretty much on every level, from what it is to how someone might attempt to apply it.
rvanlaar 6y ago
I'm Dutch and am following this scandal for quite some time. This article strikes me as being odd. Yes, computers were used in the decision process and no I don't believe them to be algorithms. It was specifically coordinated and covered up from the top. As your highlight shows, it was specifically designed this way. Which means it was a business process.

shitpostbot 6y ago

Sounds like what they had was a tool that used correlation to bucket people based on statistics with increased risk of fraud, and for some reason they were using it as proof of fraud. Yes, this is a massive scandal, but why is every article about it focused on spooky *ALGORITHMS* (I have never seen the word used as much as I have in the articles covering this story) instead of the clearly massive fault of the people running this.

urtie 6y ago

Just to get you an idea of how wrongheaded this entire thing was, I have a personal anecdote relating to this shitstorm. Back in autumn 2013 I got a letter stating that I had received too much in child care benefits, with a request to pay it back. On the back of said letter was the text 'If you are still entitled to child care benefits and want to settle this amount against upcoming entitlements, you do not have to do anything'. Given as I was still entitled to benefits, I proceeded to not do anything. A few weeks later I got a reminder, once again stating that if I wanted to settle, I wouldn't have to do a thing. The reminder was followed by a final notice with opportunity to pay, once again with the exact same text on the back of the letter. So, obviously, I once again did nothing. Then we got a 'Dwangbevel in naam des konings', i.e. a writ of execution in the name of the king. Red envelope. No added text. So I had to pay up. Now, as you may well guess, the tax service had already also started settling against the benefits that I was still entitled to! This prompted me to write a strongly worded letter, asking the tax service in no unclear language what I should have done differently, and requesting them to not dock me for that particular money twice. Literally the only reaction to that letter from the tax service was in silently restituting to me the double payment. Now, I'm a reasonably well paid and well educated software developer, who just happens to hold two nationalities. I am perfectly sure that I would not even have had a reminder had I just had the Dutch nationality. I could have paid in the first place and not settled against upcoming benefits; I was just lazy. However, imagine you're scraping by on minimum wage and are put in such a situation. There is no other way you could have acted than I did, but you also would not have been able to pay up to the writ of execution. And there would have been no recourse! Add to that the fact that in many cases, the people hadn't even actually been paid too much, as I had, but merely been tagged as possibly fraudulent and put in to the system for reclaiming paid benefits, so the government could take a stance of being hard against benefits fraud while figuring out if any fraud had taken place in the first place.

JangoSteve 6y ago

Adding this to my running list of algorithms encoding and amplifying systemic bias, like: * A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 https://www.nejm.org/doi/full/10.1056/NEJMsa1507092 * When the dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abstract/2688587 https://jamanetwork.com/journals/jamadermatology/article-abs... * When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-racial-bias/ https://fortune.com/2019/08/16/google-jigsaw-perspective-rac... * When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G https://www.reuters.com/article/us-amazon-com-jobs-automatio... * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though: * When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it-wrong-how-facial-recognition-led-to-a-false-arrest-in-michig https://www.npr.org/2020/06/24/882683463/the-computer-got-it... * When the above is compounded for people of color according to studies which show that facial recognition systems misidentify dark-skinned women 40x more often than for light-skinned men. http://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212 http://news.mit.edu/2018/study-finds-gender-skin-type-bias-a.... Another study showed false positives can be 10x to 100x more frequent for Asian and African American faces compared to Caucasian. https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software https://www.nist.gov/news-events/news/2019/12/nist-study-eva... * When an algorithm blocked kidney transplants for Black patients. https://www.wired.com/story/how-algorithm-blocked-kidney-transplants-black-patients/ https://www.wired.com/story/how-algorithm-blocked-kidney-tra... * When clinical algorithms include “corrections” for race which directly raise the bar for the need for interventions in people of color, such that they then receive less clinical screening, less surveillance, less diagnoses, and less treatment for everything, including cancer, organ transplants, birth interventions, urinary and blood, bone, and heart disease. https://www.nejm.org/doi/10.1056/NEJMms2004740 https://www.nejm.org/doi/10.1056/NEJMms2004740

bobdupneu 6y ago

For decades the Netherlands have been helping corporations and the super rich evade taxes at a cost of several hundred billion Euros to the global economy per year. How petty to now attack the poor in such a cowardly manner...

LorenPechtel 6y ago

It sounds to me like the real problem is what appears to be a screening tool being used as evidence of guilt. For other examples: field drug test kits--they really only say "this might be drugs", but the prosecutors use jail to try to get people to plead guilty rather than promptly doing a proper test. Also, the ubiquitous breathalyzer--it's actually coded into law as proof in many places but it's not. There is an inherent biological flaw that makes the readings inaccurate.

FpUser 6y ago

This will only stop when people in the government will be actually punished for the suffering they've caused. Which is mostly never.