14 ms·

The Dutch benefits scandal: a cautionary tale for algorithmic enforcement

165 points by sohkamyung 4y ago

defrost 4y ago
Systems that were meant to detect misuse of the benefits scheme, mistakenly labelled over 20,000 parents as fraudsters. Amongst the upheaval, little attention was brought to the fact that the tax authority was making use of algorithms to guide its decision-making. In a report by the Dutch Data Protection Authority, it became clear that a ‘self-learning’ algorithm was used to classify the benefit claims. Whether it's pure AI or simple big spreadsheet chopping off tens of thousands that "match some filter" fraud detection is rarely that simple. When services impact real human beings and false positives have consequences that fall on the least resourced it's important to follow through with human eyeballs and good channels of appeal. In Australia the parallel mistake was the Robodebt Scheme: On 29 May 2020, the Morrison Government announced that it would scrap the debt recovery scheme, with 470,000 wrongly-issued debts to be repaid in full. Initially, the total sum of the repayments was estimated to be A$721 million, however in November 2020 this figure expanded to A$1.2 billion after the Australian government settled a class-action lawsuit before it could go to trial. https://en.wikipedia.org/wiki/Robodebt_scheme https://en.wikipedia.org/wiki/Robodebt_scheme https://robodebt.royalcommission.gov.au/ https://robodebt.royalcommission.gov.au/
- bxjx 4y ago
  I too was going to mention Robodebt. In addition to wasting money, it caused all sorts of trauma to already vulnerable people - sadly in some cases leading to suicide (https://en.wikipedia.org/wiki/Robodebt_scheme#Allegations_of_misconduct https://en.wikipedia.org/wiki/Robodebt_scheme#Allegations_of...). My team at GetUp built a tool to automate initiating the process of challenging these requests https://fraudstop.com.au/ https://fraudstop.com.au/
  - RileyJames 4y ago
    I’d be very keen to know more about fraudstop. How it was built, the development process of the product and the green lighting of the project within getup. Was it internal only? Was anything open sourced? I’m aware of GetUp, but this is the first I’ve heard of a product they’ve developed. And that they have a team that’s capable of such developments.
  - defrost 4y ago
    Good on you all for making tools to assist. Robodebt was brutal in it's application and general air of intimidation, I wasn't personally affected but I fielded a few calls on behalf of several people I knew who felt crushed by what for them were significant debts (several times any savings they had in situations where they had zero income over enough to barely cover very basic cost of living).
  - EdwardDiego 4y ago
    Kudos to your team. ...and holy hell, they charged a 10% recovery fee? At least in NZ, when our social welfare system recovers an overpayment, there's no interest or fees. The only punitive measures used are criminal prosecution if you overtly committed fraud to a significant extent.
    - thombat 4y ago
      The Deputy Minister of Information explains it well in Terry Guillam's "Brazil": INTERVIEWER: And the cost of it all, Deputy Minister? Seven percent of the gross national produce ... HELPMANN: I understand this concern on behalf of the tax-payers. People want value for money and a cost-effective service. That is why we always insist on the principle of Information Retrieval Charges. These terrorists are not pulling their weight, and it's absolutely right and fair that those found guilty should pay for their periods of detention and the Information Retrieval Procedures used in their interrogation. And later on a well-meaning guard advises the protagonist: Don't fight it, son .... confess quickly ... If you hold out too long you could jeopardise your credit rating.
fjfaase 4y ago
This benefits scandal is far more about a government not wanting to recognize that things went wrong when the first signals occured many years ago (a legal advisor was ignored and demoted) and forgetting the human side of the application of the law. And when being exposed, trying to cover-up, and still hampering the compensation of those who were damaged by it. The latest estimates are that it will still take about seven years before everybody who suffered due to this scandal will be compensated. And again the main reason why it is taking so long, is due to a strickt adhearance to procedures and a fundamental mistrust of those being harmed, forcing many having to into legal procedures.
- ragebol 4y ago
  > strickt adhearance to procedures So still following an algorithm, though ran on human brains rather than electronics.
bragr 4y ago
Seems to be overloaded right now. Cached: https://web.archive.org/web/20230206232359/https://eulawenforcement.com/?p=7941 https://web.archive.org/web/20230206232359/https://eulawenfo...
roxgib 4y ago
A common theme seems to be humans taking output from an algorithm and blindly making decisions based on it, even when specifically told that the output shouldn't be used as the sole basis for a decision or even as evidence as all, other than cause to apply greater scrutiny. We've seen this in pharmacies flat out refusing to dispense medicine when an automated system flags a concern, even if it's simply supposed to be a prompt to ask more questions, or police trusting the results of facial recognition without further checking they have the right person. People talk about AI enhancing human abilities, and it can obviously do that, but it's really scary how willing people are to trust a system they've been told shouldn't be trusted.
- kranke155 4y ago
  The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being liable or responsible, and don’t have time to verify. Plus “no one gets fired for doing what algorithm says”.
  - patrakov 4y ago
    I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.
    - auggierose 4y ago
      That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
      - krisoft 4y ago
plantain 4y ago
Baffling difference in outcomes compared to Robodebt in Australia, where no one took any responsibility.
- gonzo41 4y ago
  Hopefully that's coming.
- yunohn 4y ago
  There was no responsibility really taken - the public re-elected the same government immediately after they did a “fake” resignation the month before elections.
jacquesm 4y ago
Algorithm is too much of a term to use here. They simply labelled everybody with a foreign (read: Islamic, Polish or Bulgarian) sounding name and/or an immigration background (including war refugees!) as a potential fraud case in response to some very limited actual fraud that exposed the tax office as being pretty bad at uncovering some (obvious) fraud cases. Long story short, this article contains some grains of truth but the reality is much simpler: the Dutch tax office systematically engaged in discrimination, persisted in doing so when this was pointed out to them and made extraordinary efforts to hide their failings rather than to fix it. The most ugly bit: the government that was in charge during all this fell due to the scandal and then was promptly re-elected claiming that they have full mandate to pursue their various un-ethical goals. Families broke apart over this, children (1100!) got placed out of their parent's homes (because having your parents labelled fraudsters gives extraordinary powers to the government to really mess up your life as if it wasn't enough yet). A number of people committed suicide, one of which was directly traced back to the scandal. If not for a single lawyer (González Pérez) persisting this whole thing would have never been brought to light. More than the algorithm I would blame the people who willingly implemented all of this. FWIW this is still unresolved, very few people (if any) have had any kind of compensation and the government is dragging its heels every step of the way.
- puffoflogic 4y ago
  [flagged]
- EdwardDiego 4y ago
  Thank you for the background :)
  - jacquesm 4y ago
    It's one of the most gruesome pages in (recent) Dutch history and it is far from over.
- arp242 4y ago
  By far the bigger problem is that once you got labelled as "fraudulent" it was nigh-impossible to prove your innocence, no matter what kind of proof you provided, that people were marked as "fraudulent" over simple banal mistakes or misunderstandings, and that people's benefits were not only stopped, they also had to return all previous benefits, often with a fine added. All of that was made worse by the reluctance to admit there was a structural problem at all for years. That was the second-biggest problem. Everything else are also problems, but this is really the root of the problem. It's worth pointing out that people with immigration backgrounds were far from the only victims, unlike your comment suggests (possession of dual nationality was the biggest "risk marker" they used AFAIK, not name, and it's not entirely clear just how biased the results were because people with migration backgrounds are also overrepresented as receivers of benefits). In short, it would still have been an enormous scandal even without any ethnic profiling. That is not to say that ethnic profiling wasn't a problem, just the lesser one out of several. > FWIW this is still unresolved, very few people have had any kind of compensation and the government is dragging its heels every step of the way. This is genuinely a difficult problem. 1,379 people are involved full-time just for the compensation[1] There's even been criticism it's too easy to get compensation, even if you weren't a victim (see e.g. [2]). Should it be faster? Yes. But it's hard to see how that would be realistically possible. I certainly don't think that the government has been "dragging its heels every step of the way" for the last 2 years. [1]: https://www.rijksoverheid.nl/binaries/rijksoverheid/documenten/rapporten/2023/02/03/voortgangsrapportage-hersteloperatie-toeslagen-2022-q4/voortgangsrapportage-hersteloperatie-toeslagen-2022-q4.pdf https://www.rijksoverheid.nl/binaries/rijksoverheid/document... page 43 [1]: https://decorrespondent.nl/13592/na-de-toeslagenaffaire-is-nu-de-compensatie-affaire-in-de-maak-en-weer-wil-niemand-het-horen/817712376008-96d24b4b https://decorrespondent.nl/13592/na-de-toeslagenaffaire-is-n...
smnrchrds 4y ago
Reminds me of the British Post Office scandal: > More than 700 branch managers were given criminal convictions when faulty accounting software made it look as though money was missing from their sites. https://www.bbc.com/news/business-56718036 https://www.bbc.com/news/business-56718036
- jimlongton 4y ago
  That scandal is different in nature but just as egregious. It didn't involve fancy ML or algorithms. The Post Office and Fujitsu knew the system was broken (no correct tracking of transactions) and maliciously prosecuted hundreds of people to cover up their enormous failure. "Expert witnesses" lied on the stand, investigations were obstructed and paperwork burned.
  - EdwardDiego 4y ago
    Are there more details on that somewhere? Feels like something that an investigative journalist (or, I guess these days, podcaster) would've dug right into the guts of.
    - synotna2 4y ago
      There is a very good book that covers it very well: the great post office scandal Hard to read without boiling your blood
      - EdwardDiego 4y ago
        Cheers!
    - arp242 4y ago
      People had been doing that since at least 2011 (there were some rumblings before, but no in-depth reporting by the likes of the BBC AFAIK). Still took close to 10 years to resolve the matter... You can read a report at e.g. https://www.private-eye.co.uk/pictures/special_reports/justice-lost-in-the-post.pdf https://www.private-eye.co.uk/pictures/special_reports/justi... As well as Wikipedia of course https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
jbverschoor 4y ago
Not really an algorithm. At least not a computer based one
aquarium87 4y ago
It seems that in reality, immigrants commit more fraud, and the AI correctly identified this. Who would have thought? I'm sure the same results would apply when viewing through the lense of class or income, so this isn't about race. Race just happens to be the angle used to most effectively right a wrong. The question then becomes what is the threshold to open an investigation? An AI represents reality better than a human, but also reminds us that certain classes and races of people are more likely to X Y and Z. If reality is racist, and there is no such thing as equality or equity, is it ever ok to use an AI for any task at all? Even while not feeding race data to the AI, it will still be racist because all datapoints should lead to an accurate representation of reality, and there is no such thing as equality or equity. In this example, even without race data, the AI should STILL come to the same conclusions. In fact, I doubt it knew the skin color or ethnicity of the names of the people it was analyzing. Where the Dutch fucked up was jumping to conclusions and labelling without cause or investigation 20,000 people as fraudsters. BUT, EVEN IF THEY CONDUCTED INVESTIGATIONS ON ALL 20,000, and they were guilty, does that make it any more right or less wrong? I posit no. It's still illegal and wrong. AI can only represent reality, reality reflects uncomfortable truths. AI has to be racist, or it isn't doing its job. The world is not equitable, all races have difference cultures and values and work ethics. Which I posit means that even if the AI is correct, and even if those 20,000 committed fraud, its still an illegal prosecution as there was never a legitimate predicate to investigate in the first place. AI is racist, even if it represents reality, and therefore cannot be used to initiate an investigation. To me, a more interesting question is whether AI should be allowed to assist police in investigations, as even with a predicate for opening an investigation, the AI is still racist/sexist/classiest etc, and is going to help put people in jail based on racist classest sexist biases. An example is a middle class white man that shares few traits with the average criminal, but is nonetheless a criminal. VS a black low income man that shares many traits with other criminals. In this example, if an investigative predicate was found for both, and an AI was used as an assistant to the police, the police would be looking at the low income guy with more scrutiny as they would believe that a more likely case to win. Theoretically. Even though we know them to be guilty, the police don't know that, and the AI don't know that, but the AI is allowing for more scrutiny and conviction against one of the two guilty parties. TLDR; AI is fucked. Shut it all down.
- 4y ago
celestialcheese 4y ago
Everytime a story like this pops up, I think of patio11's argument that "the optimal amount of fraud is non-zero". [1] I work with our local food bank, and got in a discussion with a family member who was shocked at there being no means testing or proof of need to participate. I'm certain there are people we serve who are over whatever threshold would exist, but the cost of excluding someone who's hungry because they couldn't print a paystub or have access to the right documents is _so_ much higher than any spillage from fraud. With certain aspects of society, such as with social-saftey-net benefits, when the costs of exclusion impact those teetering on the edge, "non-zero" fraud is absolutely the optimal amount. 1 - https://www.bitsaboutmoney.com/archive/optimal-amount-of-fraud/ https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
- MichaelZuo 4y ago
  > I'm certain there are people we serve who are over whatever threshold would exist, but the cost of excluding someone who's hungry because they couldn't print a paystub or have access to the right documents is _so_ much higher than any spillage from fraud. How did you determine this?
  - purpleflame1257 4y ago
    If the goal of the food bank is "ensure there's food for people who can't afford it," then turning away one person is a failure. As long as you're not running out of food, it would seem to me that means-testing for a food bank is unnecessary.
    - Ekaros 4y ago
      And means testing at food bank is bound to fail. As there are sufficient number of scenarios where technically person there should have means, but for one or other reason they do not at that specific moment. Like maybe home fire or something. Or they were fired and things are processing.
      - tremon 4y ago
at_a_remove 4y ago
I am going to say something deeply cynical: a non-trivial portion of the workforce would prefer not to have to think much or make tough decisions. As a programmer, I have heard this in so many variants of "Why can't we make the computer do it?" This naturally exists in tension with having jobs, so most people who have not automated themselves out of a position (and that has happened under my watch) would keep the "action" part for themselves, and the think-y bits would preferably be offloaded elsewhere. To some extent, a programmer is the natural predator of these individuals, but that's a side matter. As such, responsibility, the icky bits, those get pushed to The System, wherein accountability dissolves like cotton candy in hot water. The System said these people were frauds, I merely followed my marching orders. The bigger the organization, the more System to serve as a kind of landfill of responsibility, as well as the more inertia preventing real change. Essentially, the avoidance of the labor of thought, consideration, judgment and the like are what set all of this in motion. And people will still follow their GPS off of a cliff because of this.
kbuck 4y ago
Site appears to be down. Mirror: http://web.archive.org/web/20230206232359/https://eulawenforcement.com/?p=7941 http://web.archive.org/web/20230206232359/https://eulawenfor...
c3534l 4y ago
I'm confused. Was the AI wrong? Are we mad that immigrants commit a particular type of fraud at higher rates? Because then we're just mad that an algorithm discovered a truth our social norms don't normally allow us to openly acknowledge. If we're mad that the workers didn't perform a proper investigation or give people due process, then I hardly see why we need to make the predictive model the target of our blame, rather than the human beings with intent and responsibility chose not to treat people with the rights and priveleges that should be afforded to them. But I can't understand why everyone has decided the "correct" answer to this problem is to ignore reality and then try to delude machines into acting like how we want the world to work is how it actually works.
- LeetHacks 4y ago
  Yes, the AI was wrong and output was blindly accepted. There's no evidence migration background was a good predictor of fraud. The government was warned multiple times what they were doing was both illigal and ineffective. And even if it was a predictor, 26000 people were wrongly labeled as fraudsters, causing personal bankruptcy (which is very bad in the Netherlands), evictions, divorces, children wrongfully being taken away from the families (1100!) and suicides. All while systematically denying the issue, government officials that knowingly lied during public hearings and a complete inability to compensate those that were affected. Don't drag "wokeness" into this discussion, it has nothing to do with it.
  - concordDance 4y ago
    > There's no evidence migration background was a good predictor of fraud This would be a surprising statement to be true (theres technically evidence for Jesus being the son of god, its just the priors/hypothesis complexity are low/high. And the fact theres more evidence against the hypothesis). If you asked the makers of the law/algorithm what do you think they'd say?
  - ExoticPearTree 4y ago
    Wow. So the Dutch government considered people guilty by default for the simple reason of not being Dutch. What happened to the innocent until proven guilty way of handling matters?
rendall 4y ago
I would think, that to train this kind of algorithm, the investigators must never be told the findings of the algorithm. The training data must come only from historical data. It might be usable only when the algorithm can predict the findings of the investigators, with caution.
throwawayarnty 4y ago
Having lived in many European countries, Netherlands was by far the most openly racist of them all. Especially during coronavirus. Just browse a Facebook apartment rental group in Netherlands. Most would say “No internationals!”. Imagine a society where such blatant discrimination exists, and you will see why such scandals targeting foreigners is no surprise.
- deleted 4y ago
  [deleted]
- mtts 4y ago
  Yes, the Dutch are incredibly racist (I’m Dutch, btw) but the “no internationals” isn’t because of the color of their skin but because they don’t stay around for long. With “internationals” (meaning expats and foreign students) you have to go through the hassle of putting up your rental on the market much more often than with non-“internationals”. “No internationals” basically just means “long term rentals only”. Edit: but, in typical Dutch fashion, this essentially innocent statement is worded as rudely as possible.
heisenbit 4y ago
> The deference to an algorithm may weaken the agency’s internal accountability, by attributing decisions to a less explainable machine. On the long run this could affect the effectiveness of enforcement. I would argue not just the effectiveness but also the political acceptance of any enforcement and thus the existence of the state. Benefits as the plaything for politicians are an incredible and seemingly ever growing pile of sometimes contradictory, hindering or impossible to follow rules. Computers as humans will fail to implement them. The best we can do (besides thinking before writing rules) is to surround the rule application with layers of escalation to deal with the fallout and provide feedback paths to fix widespread or egregious failures. The unwillingness to monitor and fix failures is what made it a scandal. The economies of scale in computer applications made it a big one.
- jacquesm 4y ago
  The computer merely provided the pretext that they needed to do what they wanted to do anyway.
arianzagros 4y ago
It is the second step in order fulfillment, right after the picking process. The packing process takes place in the warehouse and typically consists of choosing appropriate materials and an appropriate container to pack the products, weighing the package, and labeling it with the relevant invoice or packing sliphttps://arianzagrosmachinery.com/ https://arianzagrosmachinery.com/ https://packationmachinery.com/ https://packationmachinery.com/
paganel 4y ago
Why can't the States take the false positives of "misuse" and move on? Ignoring this horrible Dutch example, I have the feeling that similar policies of trying to "correct" misuse have only managed to leave people in need without their last available support. The UK comes to mind in that respect, lots of similar cases in there (from what I could read).
RektBoy 4y ago
First, make your country refugee friendly. Then start to persecute these refugees. Dutch big brain time.
- jacquesm 4y ago
  And ignore those who make money off the refugees.
- mtts 4y ago
  - The Netherlands is in no way refugee friendly. They tried to ban families from reuniting (which international treaties explicitly forbid) and so many shelters were closed down that incoming refugees has to sleep out in the cold for weeks before they could be “processed”. - The victims of the benefit scandal were (often second or third generation) immigrants, not refugees.
deleted 4y ago
[deleted]
college_physics 4y ago
We live in a state of unawereness, denial or moronic apathy about the role of algorithms. This helps both state and commercial actors get away with actions that seriously violate prevailing moral or political preferences and equilibria. Human society involves huge numbers, it cant operate efficiently without some sort of automation. Algorithms, (whether simple or complicated versions doesnt matter to first order) have been used since a long time in sectors such as insurance and banking. When algorithmic decision making is absolutely necessary, regulations, independent control, transparency, accountability etc are various mechanisms to mitigate the risk. Today there is no excuse to keep dragging our feet about these matters.
instagraham 4y ago
How do you tackle bias in tackling bias? This is not even a machine problem. Not referring to this incident in particular, but rather, thinking ahead. In a hypothetical scenario, if it turns out that immigrants from a certain country are in fact high-risk for certain kinds of fraud, and the AI automatically picks this up, is a human intervention countering its bias an erroneous one? It's telling that so much of racism and stereotyping stems from a human desire for convenience and conformity - it is easier to shape your worldview when you've identified enemies and how to classify them. When you add algorithms with the goal of increasing efficiency into the mix, you create a real possibility that it could profile certain populations out of convenience. Of course, humans do that anyway. And when these humans train AI, they may pass on the bias. But assuming the objective truth does lead to some form of selection discrimination, how does one counter this?
- moring 4y ago
  > In a hypothetical scenario, if it turns out that immigrants from a certain country are in fact high-risk for certain kinds of fraud, and the AI automatically picks this up, is a human intervention countering its bias an erroneous one? > But assuming the objective truth does lead to some form of selection discrimination, how does one counter this? The basis of non-discrimination is to judge the individual. Even if 99% of people from some country are known to be fraudsters, you treat an individual from that country the same as an individual from another country. Anything else would be discrimination by definition. If somebody prefers to apply statistical results to the individual, there is no way to do that without discrimination because that is discrimination.
phazeedae 4y ago
The Netherlands are exceptional, at giving this public image of this amazing country, where there is practically no crime, high living standards, little to no corruption, etc. etc. etc. etc. However the longer you live here, the more you realise it's just a thin veil on the top, dig just a little bit past that, and it's a country faced with multiple serious issues. So something like this benefits scandal was not such a big surprise.
- nibbleshifter 4y ago
  I mean, once you spend more than 10 minutes looking at how their drug enforcement works, you come to the conclusion the Netherlands is a de-facto narcostate. While retail supply of soft drugs is permitted and taxed, production and wholesale supply to those retailers is illegal, which helped drug gangs to become immensely powerful. This could be resolved by legalising the production and wholesale supply aspects (with great benefits for the taxman), but the political will is lacking. It would also sharply hurt criminals. The half measures around drug legislation are indicative of how the rest of the country is ran.
m000 4y ago
One would expect higher-ups in the Netherlands would be more careful when using tools they don't fully understand. Not too long ago, there was also a notorious case of statistical conviction failure [1], where nurse Lucia de Berk was given the life sentence. Initial conviction was based on misinterpreted data which were amplified and extrapolated to a number of suspected murders through bogus statistical analysis. [1] https://en.wikipedia.org/wiki/Lucia_de_Berk https://en.wikipedia.org/wiki/Lucia_de_Berk
cudgy 4y ago
“More crucially, a disproportionate amount of those labelled as fraudsters had an immigration background.” Why is that more crucial than another group being disproportionately labeled? Also, why is it so shocking that one group would be disproportionately labeled? Otherwise, randomly audit for fraud and stop pursuing an algorithm. The focus should be on whether the algorithm is accurate.

165 points by sohkamyung 4y ago

Systems that were meant to detect misuse of the benefits scheme, mistakenly labelled over 20,000 parents as fraudsters. Amongst the upheaval, little attention was brought to the fact that the tax authority was making use of algorithms to guide its decision-making. In a report by the Dutch Data Protection Authority, it became clear that a ‘self-learning’ algorithm was used to classify the benefit claims. Whether it's pure AI or simple big spreadsheet chopping off tens of thousands that "match some filter" fraud detection is rarely that simple. When services impact real human beings and false positives have consequences that fall on the least resourced it's important to follow through with human eyeballs and good channels of appeal. In Australia the parallel mistake was the Robodebt Scheme: On 29 May 2020, the Morrison Government announced that it would scrap the debt recovery scheme, with 470,000 wrongly-issued debts to be repaid in full. Initially, the total sum of the repayments was estimated to be A$721 million, however in November 2020 this figure expanded to A$1.2 billion after the Australian government settled a class-action lawsuit before it could go to trial. https://en.wikipedia.org/wiki/Robodebt_scheme https://en.wikipedia.org/wiki/Robodebt_scheme https://robodebt.royalcommission.gov.au/ https://robodebt.royalcommission.gov.au/

bxjx 4y ago
I too was going to mention Robodebt. In addition to wasting money, it caused all sorts of trauma to already vulnerable people - sadly in some cases leading to suicide (https://en.wikipedia.org/wiki/Robodebt_scheme#Allegations_of_misconduct https://en.wikipedia.org/wiki/Robodebt_scheme#Allegations_of...). My team at GetUp built a tool to automate initiating the process of challenging these requests https://fraudstop.com.au/ https://fraudstop.com.au/
- RileyJames 4y ago
  I’d be very keen to know more about fraudstop. How it was built, the development process of the product and the green lighting of the project within getup. Was it internal only? Was anything open sourced? I’m aware of GetUp, but this is the first I’ve heard of a product they’ve developed. And that they have a team that’s capable of such developments.
- defrost 4y ago
  Good on you all for making tools to assist. Robodebt was brutal in it's application and general air of intimidation, I wasn't personally affected but I fielded a few calls on behalf of several people I knew who felt crushed by what for them were significant debts (several times any savings they had in situations where they had zero income over enough to barely cover very basic cost of living).
- EdwardDiego 4y ago
  Kudos to your team. ...and holy hell, they charged a 10% recovery fee? At least in NZ, when our social welfare system recovers an overpayment, there's no interest or fees. The only punitive measures used are criminal prosecution if you overtly committed fraud to a significant extent.
  - thombat 4y ago
    The Deputy Minister of Information explains it well in Terry Guillam's "Brazil": INTERVIEWER: And the cost of it all, Deputy Minister? Seven percent of the gross national produce ... HELPMANN: I understand this concern on behalf of the tax-payers. People want value for money and a cost-effective service. That is why we always insist on the principle of Information Retrieval Charges. These terrorists are not pulling their weight, and it's absolutely right and fair that those found guilty should pay for their periods of detention and the Information Retrieval Procedures used in their interrogation. And later on a well-meaning guard advises the protagonist: Don't fight it, son .... confess quickly ... If you hold out too long you could jeopardise your credit rating.

fjfaase 4y ago

This benefits scandal is far more about a government not wanting to recognize that things went wrong when the first signals occured many years ago (a legal advisor was ignored and demoted) and forgetting the human side of the application of the law. And when being exposed, trying to cover-up, and still hampering the compensation of those who were damaged by it. The latest estimates are that it will still take about seven years before everybody who suffered due to this scandal will be compensated. And again the main reason why it is taking so long, is due to a strickt adhearance to procedures and a fundamental mistrust of those being harmed, forcing many having to into legal procedures.

ragebol 4y ago
> strickt adhearance to procedures So still following an algorithm, though ran on human brains rather than electronics.

bragr 4y ago

Seems to be overloaded right now. Cached: https://web.archive.org/web/20230206232359/https://eulawenforcement.com/?p=7941 https://web.archive.org/web/20230206232359/https://eulawenfo...

roxgib 4y ago

A common theme seems to be humans taking output from an algorithm and blindly making decisions based on it, even when specifically told that the output shouldn't be used as the sole basis for a decision or even as evidence as all, other than cause to apply greater scrutiny. We've seen this in pharmacies flat out refusing to dispense medicine when an automated system flags a concern, even if it's simply supposed to be a prompt to ask more questions, or police trusting the results of facial recognition without further checking they have the right person. People talk about AI enhancing human abilities, and it can obviously do that, but it's really scary how willing people are to trust a system they've been told shouldn't be trusted.

kranke155 4y ago
The people making the algorithmic system are smart enough to see its limitations. The people they instruct to use it are scared of going against it and being liable or responsible, and don’t have time to verify. Plus “no one gets fired for doing what algorithm says”.
- patrakov 4y ago
  I believe one of the solutions to the problem of people automatically trusting the AI to make judgements, where in fact they are supposed to scrutinize, would be to introduce some deliberate random mistakes in the AI output, and log these deliberate mistakes in some not-really-accessible place. Then, later, we would be able to correlate the decisions made by a human with this log, and, if excessive trust is found, educate or fire the human.
  - auggierose 4y ago
    That doesn't sound like a solution, that sounds like hell. Just imagine a boss is doing that to you, instead of a machine. Giving you wrong facts from time to time on purpose, to see how badly you screw up.
    - krisoft 4y ago

plantain 4y ago

Baffling difference in outcomes compared to Robodebt in Australia, where no one took any responsibility.

gonzo41 4y ago
Hopefully that's coming.
yunohn 4y ago
There was no responsibility really taken - the public re-elected the same government immediately after they did a “fake” resignation the month before elections.

jacquesm 4y ago

Algorithm is too much of a term to use here. They simply labelled everybody with a foreign (read: Islamic, Polish or Bulgarian) sounding name and/or an immigration background (including war refugees!) as a potential fraud case in response to some very limited actual fraud that exposed the tax office as being pretty bad at uncovering some (obvious) fraud cases. Long story short, this article contains some grains of truth but the reality is much simpler: the Dutch tax office systematically engaged in discrimination, persisted in doing so when this was pointed out to them and made extraordinary efforts to hide their failings rather than to fix it. The most ugly bit: the government that was in charge during all this fell due to the scandal and then was promptly re-elected claiming that they have full mandate to pursue their various un-ethical goals. Families broke apart over this, children (1100!) got placed out of their parent's homes (because having your parents labelled fraudsters gives extraordinary powers to the government to really mess up your life as if it wasn't enough yet). A number of people committed suicide, one of which was directly traced back to the scandal. If not for a single lawyer (González Pérez) persisting this whole thing would have never been brought to light. More than the algorithm I would blame the people who willingly implemented all of this. FWIW this is still unresolved, very few people (if any) have had any kind of compensation and the government is dragging its heels every step of the way.

puffoflogic 4y ago
[flagged]
EdwardDiego 4y ago
Thank you for the background :)
- jacquesm 4y ago
  It's one of the most gruesome pages in (recent) Dutch history and it is far from over.
arp242 4y ago
By far the bigger problem is that once you got labelled as "fraudulent" it was nigh-impossible to prove your innocence, no matter what kind of proof you provided, that people were marked as "fraudulent" over simple banal mistakes or misunderstandings, and that people's benefits were not only stopped, they also had to return all previous benefits, often with a fine added. All of that was made worse by the reluctance to admit there was a structural problem at all for years. That was the second-biggest problem. Everything else are also problems, but this is really the root of the problem. It's worth pointing out that people with immigration backgrounds were far from the only victims, unlike your comment suggests (possession of dual nationality was the biggest "risk marker" they used AFAIK, not name, and it's not entirely clear just how biased the results were because people with migration backgrounds are also overrepresented as receivers of benefits). In short, it would still have been an enormous scandal even without any ethnic profiling. That is not to say that ethnic profiling wasn't a problem, just the lesser one out of several. > FWIW this is still unresolved, very few people have had any kind of compensation and the government is dragging its heels every step of the way. This is genuinely a difficult problem. 1,379 people are involved full-time just for the compensation[1] There's even been criticism it's too easy to get compensation, even if you weren't a victim (see e.g. [2]). Should it be faster? Yes. But it's hard to see how that would be realistically possible. I certainly don't think that the government has been "dragging its heels every step of the way" for the last 2 years. [1]: https://www.rijksoverheid.nl/binaries/rijksoverheid/documenten/rapporten/2023/02/03/voortgangsrapportage-hersteloperatie-toeslagen-2022-q4/voortgangsrapportage-hersteloperatie-toeslagen-2022-q4.pdf https://www.rijksoverheid.nl/binaries/rijksoverheid/document... page 43 [1]: https://decorrespondent.nl/13592/na-de-toeslagenaffaire-is-nu-de-compensatie-affaire-in-de-maak-en-weer-wil-niemand-het-horen/817712376008-96d24b4b https://decorrespondent.nl/13592/na-de-toeslagenaffaire-is-n...

smnrchrds 4y ago

Reminds me of the British Post Office scandal: > More than 700 branch managers were given criminal convictions when faulty accounting software made it look as though money was missing from their sites. https://www.bbc.com/news/business-56718036 https://www.bbc.com/news/business-56718036

jimlongton 4y ago
That scandal is different in nature but just as egregious. It didn't involve fancy ML or algorithms. The Post Office and Fujitsu knew the system was broken (no correct tracking of transactions) and maliciously prosecuted hundreds of people to cover up their enormous failure. "Expert witnesses" lied on the stand, investigations were obstructed and paperwork burned.
- EdwardDiego 4y ago
  Are there more details on that somewhere? Feels like something that an investigative journalist (or, I guess these days, podcaster) would've dug right into the guts of.
  - synotna2 4y ago
    There is a very good book that covers it very well: the great post office scandal Hard to read without boiling your blood
    - EdwardDiego 4y ago
      Cheers!
  - arp242 4y ago
    People had been doing that since at least 2011 (there were some rumblings before, but no in-depth reporting by the likes of the BBC AFAIK). Still took close to 10 years to resolve the matter... You can read a report at e.g. https://www.private-eye.co.uk/pictures/special_reports/justice-lost-in-the-post.pdf https://www.private-eye.co.uk/pictures/special_reports/justi... As well as Wikipedia of course https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal

jbverschoor 4y ago

Not really an algorithm. At least not a computer based one

aquarium87 4y ago

It seems that in reality, immigrants commit more fraud, and the AI correctly identified this. Who would have thought? I'm sure the same results would apply when viewing through the lense of class or income, so this isn't about race. Race just happens to be the angle used to most effectively right a wrong. The question then becomes what is the threshold to open an investigation? An AI represents reality better than a human, but also reminds us that certain classes and races of people are more likely to X Y and Z. If reality is racist, and there is no such thing as equality or equity, is it ever ok to use an AI for any task at all? Even while not feeding race data to the AI, it will still be racist because all datapoints should lead to an accurate representation of reality, and there is no such thing as equality or equity. In this example, even without race data, the AI should STILL come to the same conclusions. In fact, I doubt it knew the skin color or ethnicity of the names of the people it was analyzing. Where the Dutch fucked up was jumping to conclusions and labelling without cause or investigation 20,000 people as fraudsters. BUT, EVEN IF THEY CONDUCTED INVESTIGATIONS ON ALL 20,000, and they were guilty, does that make it any more right or less wrong? I posit no. It's still illegal and wrong. AI can only represent reality, reality reflects uncomfortable truths. AI has to be racist, or it isn't doing its job. The world is not equitable, all races have difference cultures and values and work ethics. Which I posit means that even if the AI is correct, and even if those 20,000 committed fraud, its still an illegal prosecution as there was never a legitimate predicate to investigate in the first place. AI is racist, even if it represents reality, and therefore cannot be used to initiate an investigation. To me, a more interesting question is whether AI should be allowed to assist police in investigations, as even with a predicate for opening an investigation, the AI is still racist/sexist/classiest etc, and is going to help put people in jail based on racist classest sexist biases. An example is a middle class white man that shares few traits with the average criminal, but is nonetheless a criminal. VS a black low income man that shares many traits with other criminals. In this example, if an investigative predicate was found for both, and an AI was used as an assistant to the police, the police would be looking at the low income guy with more scrutiny as they would believe that a more likely case to win. Theoretically. Even though we know them to be guilty, the police don't know that, and the AI don't know that, but the AI is allowing for more scrutiny and conviction against one of the two guilty parties. TLDR; AI is fucked. Shut it all down.

4y ago

celestialcheese 4y ago

Everytime a story like this pops up, I think of patio11's argument that "the optimal amount of fraud is non-zero". [1] I work with our local food bank, and got in a discussion with a family member who was shocked at there being no means testing or proof of need to participate. I'm certain there are people we serve who are over whatever threshold would exist, but the cost of excluding someone who's hungry because they couldn't print a paystub or have access to the right documents is _so_ much higher than any spillage from fraud. With certain aspects of society, such as with social-saftey-net benefits, when the costs of exclusion impact those teetering on the edge, "non-zero" fraud is absolutely the optimal amount. 1 - https://www.bitsaboutmoney.com/archive/optimal-amount-of-fraud/ https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...

MichaelZuo 4y ago
> I'm certain there are people we serve who are over whatever threshold would exist, but the cost of excluding someone who's hungry because they couldn't print a paystub or have access to the right documents is _so_ much higher than any spillage from fraud. How did you determine this?
- purpleflame1257 4y ago
  If the goal of the food bank is "ensure there's food for people who can't afford it," then turning away one person is a failure. As long as you're not running out of food, it would seem to me that means-testing for a food bank is unnecessary.
  - Ekaros 4y ago
    And means testing at food bank is bound to fail. As there are sufficient number of scenarios where technically person there should have means, but for one or other reason they do not at that specific moment. Like maybe home fire or something. Or they were fired and things are processing.
    - tremon 4y ago

at_a_remove 4y ago

I am going to say something deeply cynical: a non-trivial portion of the workforce would prefer not to have to think much or make tough decisions. As a programmer, I have heard this in so many variants of "Why can't we make the computer do it?" This naturally exists in tension with having jobs, so most people who have not automated themselves out of a position (and that has happened under my watch) would keep the "action" part for themselves, and the think-y bits would preferably be offloaded elsewhere. To some extent, a programmer is the natural predator of these individuals, but that's a side matter. As such, responsibility, the icky bits, those get pushed to The System, wherein accountability dissolves like cotton candy in hot water. The System said these people were frauds, I merely followed my marching orders. The bigger the organization, the more System to serve as a kind of landfill of responsibility, as well as the more inertia preventing real change. Essentially, the avoidance of the labor of thought, consideration, judgment and the like are what set all of this in motion. And people will still follow their GPS off of a cliff because of this.

kbuck 4y ago

Site appears to be down. Mirror: http://web.archive.org/web/20230206232359/https://eulawenforcement.com/?p=7941 http://web.archive.org/web/20230206232359/https://eulawenfor...

c3534l 4y ago

I'm confused. Was the AI wrong? Are we mad that immigrants commit a particular type of fraud at higher rates? Because then we're just mad that an algorithm discovered a truth our social norms don't normally allow us to openly acknowledge. If we're mad that the workers didn't perform a proper investigation or give people due process, then I hardly see why we need to make the predictive model the target of our blame, rather than the human beings with intent and responsibility chose not to treat people with the rights and priveleges that should be afforded to them. But I can't understand why everyone has decided the "correct" answer to this problem is to ignore reality and then try to delude machines into acting like how we want the world to work is how it actually works.

LeetHacks 4y ago
Yes, the AI was wrong and output was blindly accepted. There's no evidence migration background was a good predictor of fraud. The government was warned multiple times what they were doing was both illigal and ineffective. And even if it was a predictor, 26000 people were wrongly labeled as fraudsters, causing personal bankruptcy (which is very bad in the Netherlands), evictions, divorces, children wrongfully being taken away from the families (1100!) and suicides. All while systematically denying the issue, government officials that knowingly lied during public hearings and a complete inability to compensate those that were affected. Don't drag "wokeness" into this discussion, it has nothing to do with it.
- concordDance 4y ago
  > There's no evidence migration background was a good predictor of fraud This would be a surprising statement to be true (theres technically evidence for Jesus being the son of god, its just the priors/hypothesis complexity are low/high. And the fact theres more evidence against the hypothesis). If you asked the makers of the law/algorithm what do you think they'd say?
- ExoticPearTree 4y ago
  Wow. So the Dutch government considered people guilty by default for the simple reason of not being Dutch. What happened to the innocent until proven guilty way of handling matters?

rendall 4y ago

I would think, that to train this kind of algorithm, the investigators must never be told the findings of the algorithm. The training data must come only from historical data. It might be usable only when the algorithm can predict the findings of the investigators, with caution.

throwawayarnty 4y ago

Having lived in many European countries, Netherlands was by far the most openly racist of them all. Especially during coronavirus. Just browse a Facebook apartment rental group in Netherlands. Most would say “No internationals!”. Imagine a society where such blatant discrimination exists, and you will see why such scandals targeting foreigners is no surprise.

deleted 4y ago
[deleted]
mtts 4y ago
Yes, the Dutch are incredibly racist (I’m Dutch, btw) but the “no internationals” isn’t because of the color of their skin but because they don’t stay around for long. With “internationals” (meaning expats and foreign students) you have to go through the hassle of putting up your rental on the market much more often than with non-“internationals”. “No internationals” basically just means “long term rentals only”. Edit: but, in typical Dutch fashion, this essentially innocent statement is worded as rudely as possible.

heisenbit 4y ago

> The deference to an algorithm may weaken the agency’s internal accountability, by attributing decisions to a less explainable machine. On the long run this could affect the effectiveness of enforcement. I would argue not just the effectiveness but also the political acceptance of any enforcement and thus the existence of the state. Benefits as the plaything for politicians are an incredible and seemingly ever growing pile of sometimes contradictory, hindering or impossible to follow rules. Computers as humans will fail to implement them. The best we can do (besides thinking before writing rules) is to surround the rule application with layers of escalation to deal with the fallout and provide feedback paths to fix widespread or egregious failures. The unwillingness to monitor and fix failures is what made it a scandal. The economies of scale in computer applications made it a big one.

jacquesm 4y ago
The computer merely provided the pretext that they needed to do what they wanted to do anyway.

arianzagros 4y ago

It is the second step in order fulfillment, right after the picking process. The packing process takes place in the warehouse and typically consists of choosing appropriate materials and an appropriate container to pack the products, weighing the package, and labeling it with the relevant invoice or packing sliphttps://arianzagrosmachinery.com/ https://arianzagrosmachinery.com/ https://packationmachinery.com/ https://packationmachinery.com/

paganel 4y ago

Why can't the States take the false positives of "misuse" and move on? Ignoring this horrible Dutch example, I have the feeling that similar policies of trying to "correct" misuse have only managed to leave people in need without their last available support. The UK comes to mind in that respect, lots of similar cases in there (from what I could read).

RektBoy 4y ago

First, make your country refugee friendly. Then start to persecute these refugees. Dutch big brain time.

jacquesm 4y ago
And ignore those who make money off the refugees.
mtts 4y ago
- The Netherlands is in no way refugee friendly. They tried to ban families from reuniting (which international treaties explicitly forbid) and so many shelters were closed down that incoming refugees has to sleep out in the cold for weeks before they could be “processed”. - The victims of the benefit scandal were (often second or third generation) immigrants, not refugees.

deleted 4y ago

[deleted]

college_physics 4y ago

We live in a state of unawereness, denial or moronic apathy about the role of algorithms. This helps both state and commercial actors get away with actions that seriously violate prevailing moral or political preferences and equilibria. Human society involves huge numbers, it cant operate efficiently without some sort of automation. Algorithms, (whether simple or complicated versions doesnt matter to first order) have been used since a long time in sectors such as insurance and banking. When algorithmic decision making is absolutely necessary, regulations, independent control, transparency, accountability etc are various mechanisms to mitigate the risk. Today there is no excuse to keep dragging our feet about these matters.

instagraham 4y ago

How do you tackle bias in tackling bias? This is not even a machine problem. Not referring to this incident in particular, but rather, thinking ahead. In a hypothetical scenario, if it turns out that immigrants from a certain country are in fact high-risk for certain kinds of fraud, and the AI automatically picks this up, is a human intervention countering its bias an erroneous one? It's telling that so much of racism and stereotyping stems from a human desire for convenience and conformity - it is easier to shape your worldview when you've identified enemies and how to classify them. When you add algorithms with the goal of increasing efficiency into the mix, you create a real possibility that it could profile certain populations out of convenience. Of course, humans do that anyway. And when these humans train AI, they may pass on the bias. But assuming the objective truth does lead to some form of selection discrimination, how does one counter this?

moring 4y ago
> In a hypothetical scenario, if it turns out that immigrants from a certain country are in fact high-risk for certain kinds of fraud, and the AI automatically picks this up, is a human intervention countering its bias an erroneous one? > But assuming the objective truth does lead to some form of selection discrimination, how does one counter this? The basis of non-discrimination is to judge the individual. Even if 99% of people from some country are known to be fraudsters, you treat an individual from that country the same as an individual from another country. Anything else would be discrimination by definition. If somebody prefers to apply statistical results to the individual, there is no way to do that without discrimination because that is discrimination.

phazeedae 4y ago

The Netherlands are exceptional, at giving this public image of this amazing country, where there is practically no crime, high living standards, little to no corruption, etc. etc. etc. etc. However the longer you live here, the more you realise it's just a thin veil on the top, dig just a little bit past that, and it's a country faced with multiple serious issues. So something like this benefits scandal was not such a big surprise.

nibbleshifter 4y ago
I mean, once you spend more than 10 minutes looking at how their drug enforcement works, you come to the conclusion the Netherlands is a de-facto narcostate. While retail supply of soft drugs is permitted and taxed, production and wholesale supply to those retailers is illegal, which helped drug gangs to become immensely powerful. This could be resolved by legalising the production and wholesale supply aspects (with great benefits for the taxman), but the political will is lacking. It would also sharply hurt criminals. The half measures around drug legislation are indicative of how the rest of the country is ran.

m000 4y ago

One would expect higher-ups in the Netherlands would be more careful when using tools they don't fully understand. Not too long ago, there was also a notorious case of statistical conviction failure [1], where nurse Lucia de Berk was given the life sentence. Initial conviction was based on misinterpreted data which were amplified and extrapolated to a number of suspected murders through bogus statistical analysis. [1] https://en.wikipedia.org/wiki/Lucia_de_Berk https://en.wikipedia.org/wiki/Lucia_de_Berk

cudgy 4y ago

“More crucially, a disproportionate amount of those labelled as fraudsters had an immigration background.” Why is that more crucial than another group being disproportionately labeled? Also, why is it so shocking that one group would be disproportionately labeled? Otherwise, randomly audit for fraud and stop pursuing an algorithm. The focus should be on whether the algorithm is accurate.