8 ms·

We investigated France's mass profiling machine

200 points by sebg 3y ago

zelda-mazzy 3y ago
I read their report on Rotterdam's welfare fraud prediction system a few months ago and it was fascinating. Really opened my eyes to a lot of things, specifically how fast any bias can create a feedback loop that causes one demographic to be investigated more often than others.
- networkchad 3y ago
  [dead]
- Biologist123 3y ago
  The model is never the reality.
  - godelski 3y ago
    This is often stated but I think often it is forgotten what is a model. It's not just "the algorithm." The metric is a model (the Wasserstein distance between norm layers of a discriminator is not image quality. Entropy isn't language). The data is a model (scraping all the internet isn't human language nor thoughts). It's important to always remember that these things too are models. Often proxies for things that are intractable. They are fine to use, but not fine to forget that they are not perfectly aligned. They are maps and to use another clique, the map is not the territory (no matter how detailed that map is).
  - raybb 3y ago
    But some are useful https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong
- troupo 3y ago
  And this is the actual danger of AI, not the doom and gloom of "AI will wipe us out".
  - ilkke 3y ago
    You don't need AI, just a government willing to abuse the most vulnerable. Check out the Australian "robodebt" fiasco [1] [1]https://en.m.wikipedia.org/wiki/Robodebt_scheme https://en.m.wikipedia.org/wiki/Robodebt_scheme
    - quickthrower2 3y ago
      Post Office Scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal
    - DeathArrow 3y ago
      How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not. By requesting welfare money, you agree that the government can investigate you.
      - danielheath 3y ago
        Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated". Governments own lawyers advised them that their plans were illegal, in advance.
        DeathArrow 3y ago
        That sounds nasty. I was talking about the context about France's profiling algo.
        joshxyz 3y ago
- nonrandomstring 3y ago
  Seems two things are at play here, the model, and how the model is used. The article was quite dry, refreshingly analytical and not overly critical of the expected ground truth that the poor are punished for being poor. It's great these models are legible and open. Looks like civil service doing a job as well as possible. The other half of the story is how, in practice these models are interpreted and what actions follow. Are further modification of the model a direct result of that and how iterative is it? Doing that aggressively changes a bare "model" into an investigative tool, not simply a thresholding utility. Sorry if I missed it, but I don't remember reading anything about how it feeds back. But what I did read was unsettling - entering people's homes at random, quizzing neighbours, pretty fascist stuff. am I wrong? I guess my point is that if you look at a bare data set, or even an algorithm, sure you can probably infer a lot about biases and intent that might be built in, but you can't see the bigger model within which this functions - and that's the real story. Is it a persecutory investigative tool? Like Hicks said we could use cruise missiles to drop food into the mouths of hungry people... a benefit system that could identify people who are struggling (of which crime is an indicator), and, I dunno, give them some money and help? That would be preachy, and quite possible imho.
  - sam_lowry_ 3y ago
    > a benefit system that could identify people who are struggling > (of which crime is an indicator), and, I dunno, give them some > money and help? This is how the system should function, but then you get public outcry when the worst of us are found abusing the system of subsidies while perpetrating their crimes. I think specifically of Marc Dutroux here. On a side note, after the mass influx of Ukrainian refugees and wide public support, it is much more obvious that those in the greatest need tend to demand less, not more.
    - nonrandomstring 3y ago
      Just returning a context here [0]. Interesting case of which I was unaware, Thanks. [0] https://en.wikipedia.org/wiki/Marc_Dutroux https://en.wikipedia.org/wiki/Marc_Dutroux
- VoodooJuJu 3y ago
  I have limited resources for investigating robberies. I need to allocate these resources efficiently. 5x more robberies are reported in zone A than zone B. Why would I allocate an equal amount of resources to each zone? I have limited resources for investigating fraud. I need to allocate those resources efficiently. If profile A is more likely to commit fraud than profile B, why would I allocate an equal amount of resources for investigating fraud between them? I'm going on a hunting trip and have limited time and resources for hunting. I need to plan my trip accordingly. If deer are more abundant for hunting in Colorado, why would I go looking for them in the Sahara? Am I wrong to look for deer in Colorado rather than the Sahara? Am I biased for investigating fraud where it's more likely to occur? Am I evil for investigating robberies where they're reported? I have little time and resources on this planet and want to allocate my time and resources as efficiently as I can in order to do the most Good. Why would I allocate my time and resources inefficiently?
  - kamma4434 3y ago
    They actually do both - they spend three quarters of their time in Colorado, but they also sample random places and use that data to decide where is the next Colorado.
dr_kiszonka 3y ago
I am curious why they used a logistic regression. Given that their "IVs” are likely highly correlated, the interpretability of the model is not great, so they could have selected something more non-linear and possibly of higher predictive power. However, they used SAS, which - in the US - would suggest to me that some sort of a statistician was involved, so they would have known about the interpretability issue. (Or maybe I need to read up on logistic regression; it has been a few years.) > An important limitation of our approach is that we do not know the relationship between the different variables that we analysed. For example, having a child between 12 and 18 (which increases risk scores) is likely correlated to being above 34 (which decreases risk scores). Doesn't France publish demographic data that would help determine these relationships? Anyway, it was a good read and many things were very clearly explained, which I appreciated.
- ramraj07 3y ago
  Probably a physics PhD data scientist :)
- teruakohatu 3y ago
  The sad state of the industry is that terrible compromised models are delivered all the time by people who should know better. I sat through a presentation on a national govt AML machine learning system, they discovered after building it that the training data was all wrong. Luckily it hasn’t been put into production yet.
  - alexey-salmin 3y ago
    I don't at all see from the article how it's "terrible compromised".
- benob 3y ago
  Could it be that they are mandated by law, like the French banking sector for credit scoring, to use an interpretable method?
  - fmajid 3y ago
Onavo 3y ago
> Finally, the raw score is run through a squasher function that outputs a probabilistic risk score between 0 and 1, where 1 is the highest risk of fraud. A raw score of 0 corresponds to a risk score of 0.5. I vote that machine learning practitioners shall henceforth refer to the softmax formula as the "squasher function". It's so much more descriptive, kudos to the French!
- blt 3y ago
  It's a sigmoid function, not softmax. Softmax is for multi-class classification.
extr 3y ago
Honestly impressive depth into the model. I also liked that the page didn't really politicize the issue.
- DeathArrow 3y ago
  > I also liked that the page didn't really politicize the issue. Are we looking at the same page? >For vulnerable recipients who struggle to navigate the complex rules of the French social security system, the risk of their benefits being reclaimed or terminated is high — even if they unintentionally committed mistakes.
  - pas 3y ago
    how would you present this finding without "politicization"?
anonyme-honteux 3y ago
I'm french and I clicked on this link, worried to learn about mass surveillance programs, NSA or communist china style. But no, this time, it was about the welfare state doing its job, doing it mostly well, with details that were more impressive than I anticipated, and also opening its data to external analysts to continue improve it. That was quite an emotional ride !
- boudin 3y ago
  It's not the whole truth though, it's thanks to associations like La Quadrature du Net https://www.laquadrature.net/ https://www.laquadrature.net/ that they had to share their algorithm. It also shows a bias towards single familly with low income, adding pressure to families already in state of precarity. It doesn't mean it's intentional but it is still an issue. The fact that there are no proper audit of those algorithms before allowing those to be used looks quite bad in my opinion. This can be quite impactful on people's life.
  - anonyme-honteux 3y ago
    I missed the context where it was associations that put the issue on the table. In the end we have an audit that show a bias against single family with low income. The bias itself is bad, but having an audit is good, it's the first step to improve things somewhat
- nfeutry 3y ago
  They did not want to open it, they were forced to by the "Commission d'Accès aux Document Administratifs". And the published code is probably outdated.
  - anonyme-honteux 3y ago
    I missed that context, thanks!
- smallpipe 3y ago
  Did you read the article? The code was obtained using a FOI request, and the model targets vulnerable making mistakes in claiming benefits rather rather than actual fraud. Which bit is the one you're impressed with ? The part where you personally will be fine?
switch007 3y ago
If only the welfare departments employed the same rigour and determination when it comes to ensuring people receive benefits that they are entitled to but not claiming. Will they use they spend the same effort to ensure nobody goes hungry because of a benefit they're entitled to but aren't claiming ...? I doubt it —— Going after welfare claimants is wildly popular in the UK thanks to years of propaganda on TV dedicated to shows about people cheating the system. There is a widespread view that anyone claiming benefits is lazy and a cheat. This gives the departments carte Blanche to introduce inhumane claims processes, algorithms etc
- skywal_l 3y ago
  In the conclusion of the article: > It also pointed out that some investigations started by the model conclude that the CNAF actually owes the beneficiary money.
  - switch007 3y ago
    That’s a defensive statement from CNAF when they were asked to comment. What do they actually do with that data in those circumstances? It’s a completely different endeavour to look for fraud and to look for under-claimed benefits. I can’t believe the same project by the same people would be equally concerned with both tasks
DeathArrow 3y ago
Well, if investigations are triggered only by algorithms, if you intend to commit fraud, you can run their statistical model against your future data and see if alarms fire off. If so, just massage the data until they don't.
- a_dabbler 3y ago
  I don't think those likely to commit welfare fraud are going to be worrying about the statistical model and most of these variables aren't easily modifiable (except months since last emailed) assuming the source of information is reliable
- pas 3y ago
  there are these random sampling checks (I guess to get a reference baseline) > The CNAF trains its models on households who were randomly investigated as part of the Paiement à bon droit Fraudes (PBDF), an annual survey
- makomk 3y ago
  The big practical obstacle to this is that the things this flags people as potentially committing fraud based on look to be the same things an actual fraudster would manipulate in order to get payments they're not actually entitled to - stuff line income, living situation, kids etc. So the obvious route to not setting off fraud alarms would likely involve not committing fraud, or at least not the most obvious and profitable kinds.
knallfrosch 3y ago
> which groups are more likely to be flagged. Did I miss the part where they checked whether the risk assessment was correct? I see a lof of studies like this: »'Male drivers aged 18-25 leaving clubs at night' are a group that is overrepresented in police DUI (Driving Under the Influence of alcohol) checks.« Stopping there is okay if you want to claim that any such targeting is wrongful. But the more useful information would be whether these groups are justly targeted.
- mistrial9 3y ago
  no - add race, spoken language, education, social status, income, politics.. those indicators are always present, and are controversial today as differentiation for street law enforcement
  - snowpid 3y ago
    French republic and many other European nations dont collect on race. And "racial thinking in Europe" does not fit very well into American thinking. Where are you from?
    - mistrial9 3y ago
      California here, born American
    - bashinator 3y ago
      > French republic and many other European nations dont collect on race. Which is a convenient way for them not to track patterns of racial discrimination in their social systems.
      - snowpid 3y ago
        why should the government empowers pseudo biological bullshit? For me it is very weird to write human race into driver licence, census or university application.
gafage 3y ago
[flagged]
kemenaran 3y ago
As noted by the other investigating organization (La Quadrature du Net), the problem is worse than just the question of the algorithm implementation. The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. Why that? Because fraud is harder to detect and to prove (you have to provide evidence it was intentional). So smaller and poorer families are disproportionately affected by the controls, because the algorithm was designed to do so. No computational adjustment can fix that: it is the initial intent that is broken. Source: https://www-laquadrature-net.translate.goog/2023/11/27/notation-des-allocataires-lindecence-des-pratiques-de-la-caf-desormais-indeniable/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=fr&_x_tr_pto=wapp https://www-laquadrature-net.translate.goog/2023/11/27/notat... (auto-translated in english)
- wslh 3y ago
  > The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. I think your observation is the greatest one: you don't target "Madoffs or SBFs" you just look at the low hanging fruit where you look at simple probabilistic causality A => B instead of, or at the same time, targetting big malicioous actors. Big and/or complex crimes and corruption are protected. The irony is that many times the crimes commited by big actors could be simpler to analyze based on tax and financial information.
- addcommitpush 3y ago
  > The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. The French welfare system is incredibly complex - see for instance this [0] simplified description of housing allowances which is 80 (!) pages long. This is not the most complex part of the system. With such a system, there are massive amounts of errors, both too-much-money-given and not-enough-money-given. The scale is so large that the French Court of Accounts refused to certify the CNAF accounts last year[1]: thoses errors represent about 7.5% of the CAF budget. So basically the probability to have an error is just a function of how complex your situation is, and thus the "algorithm" targets more complex situations - change in your marital situation, having adult children (which may or may not need to be taken into account when applying for benefits depending on a bazillion variables), and so on, increases your probability to be targeted. [0] https://www.ecologie.gouv.fr/sites/default/files/Brochure-bareme-avril-2023-APL.pdf https://www.ecologie.gouv.fr/sites/default/files/Brochure-ba... [1] https://www.ccomptes.fr/fr/publications/certification-des-comptes-2022-du-regime-general-de-securite-sociale-et-du-cpsti https://www.ccomptes.fr/fr/publications/certification-des-co...
sylware 3y ago
So, they concluded that the people most likely to commit fraud are those under the most economic pressure? They need an algorithm to do that? As far as I know this is expected behavior of normal human beings. Geniuses all over the board. We all know the core of the issue is the following balance: - expecting money to circulate in a sane society, namely where there is not a massive amount of BS jobs, that to give a decent living to all households, is just utopia. Thinking otherwise is actually dangerous. - one of the bad sides of human nature: some "nasty" and/or really "hard" "essential" jobs have to be done, and most people will try to avoid them (many won't be able to do them anyway). Not to mention, for a "developped" country, the level of efficacy is very high and only a small amount of people rotating on those jobs (but they need to build experience anyway) is actually "needed" to make things work properly.
- pas 3y ago
  the problem is that this is an important system (because it directs ~80% of investigations) without transparency, audits, and on top of this when these groups started to ask questions the attitude of the government was - surprise surprise - to try to deny their requests for transparency and on top of (or beneath) all this there's the question of model performance, calibration, etc. and cost-benefit of this whole investigation department. as in the recent case of California welfare fraud ~90% (or more) was perpetrated by organized crime groups. (which is absolutely not surprising, after all it requires a big operation to file hundreds of thousands of completely fake but seemingly-valid claims.) their claim that their model is targeting ~600 or more EUR/month frauds, which covers ~98% of the cases (based on some past historic data), but there's no mention of how much sense does this make. (it's quite likely that they could get ~90% of fraud value with drastically less effort spent on investigating those who are already likely to turn out to be rightful recipients, etc.) so all in all there ought to be a very strong bias against the classic "looking for the keys under the streetlamp" mistake, which in this case means there's likely a real negative cost-benefit to spending most investigative resources on those who are already very likely to be rightful recipients of welfare. (not to mention the usual completely avoidable - and thus totally infuriating and idiotic - discontinuity problems, like single parent with a kid just turned 18, or whatever the limit is, suddenly there's a huge drop in income, huge motivation to try to smudge the numbers, and somehow get more months of payments, etc.) of course, hard data is always welcome, I'm just a regular keyboard warrior
thriftwy 3y ago
The whole idea of welfare-driven society is nuts and untenable. Welfare-driven countries should deleverage their economies and depromote price growth until they no longer need to pay welfare on mass scale due to people being able to afford housing without much strain.
logicalmonster 3y ago
Anybody notice that all of the word choices in the headline make this sound particularly dark and ominous? Wouldn't a more accurate headline be something like "Here is the math behind how France's fraud detection system works"?
- pyrale 3y ago
  As it should be. This system targets these people and not other types of fraud which are a larger source of fraud in terms of lost money. This correlates strongly with political discourse, in terms of who is accused of being "problematic".
  - logicalmonster 3y ago
    > This system targets these people Let's keep in mind that this algorithm isn't in and of itself punishing people. It's just the starting point for a human-led investigation. Isn't that a proper application of technology? My bigger concern is that any human-led investigation is fair and reasonable and doesn't go hog-wild trying to punish people for minor misunderstandings of the rules and policies that are in place. (Somebody in this thread said that the rules just for how housing allowances can be used are like 80 pages long, so it's probably the case that it's pretty much impossible for any random citizen to know all of these rules) > other types of fraud which are a larger source of fraud in terms of lost money Surely you're not suggesting that welfare fraud shouldn't be investigated so long as say Military-Industrial-Complex fraud or Big-Pharma fraud is even greater?
    - pyrale 3y ago
      > Surely you're not suggesting that welfare fraud shouldn't be investigated so long as say Military-Industrial-Complex fraud or Big-Pharma fraud is even greater? No, I'm talking about other types of fraud effecting the social system funds. Some of them by individuals, some of them by companies. > Isn't that a proper application of technology? It's hard to apply technology properly when you don't start with good intents.
AndrewKemendo 3y ago
These kind of profile based forecasting systems *literally* encode the social biases embedded in the law into a measurement system that perfectly alienates individuals. The laws and structure of society ENSURE this is the outcome of any hierarchical capital/property based system. So it’s not the algorithm that’s the problem. The problem is having a system like this to begin with.
- skrebbel 3y ago
  > the social biases embedded in the law I’m not sure what you mean by this, do you have any examples?
  - AndrewKemendo 3y ago
    Example: Redlining was a racist social bias embedded in the city infrastructure planning and implementation process. Or “slaves are legally 3/5th of a human” was encoded into the US constitution By encoding into the legal structure a social bias, and then using that legal precedent as the goal state, implemented via orthogonal behavior measures, you are creating a measurement system that will enforce whatever goal condition the law encoded , including any biases that were encoded in the original lawmaking process [1] https://en.m.wikipedia.org/wiki/Redlining https://en.m.wikipedia.org/wiki/Redlining [2] https://en.m.wikipedia.org/wiki/Three-fifths_Compromise https://en.m.wikipedia.org/wiki/Three-fifths_Compromise
    - skrebbel 3y ago
      To be fair I think it’s a bit odd that you only quote American examples (and century old ones at that) in a discussion about a profiling system in France. I can vaguely imagine that France may have laws, today, that are biased too, but I struggle to imagine that they’re anywhere near as bad as the blatantly racist laws America grew up with. Agree that it dehumanizes citizens though.
      - AndrewKemendo 3y ago
        I find it odd you find it odd. This is a website primarily used by Americans, which I am, and therefore have the most experience in that context. I am less familiar with French laws and you agree that there are likely laws which encoded bias. So we agree that the examples I used of bias encoding are invariant to the source and you accept the premise So the only thing you response does is give a snide jab at what you perceive as relative foundational inequities in the law. Was that your intent? To simply shit on America?
Roark66 3y ago
Sadly, all benefit systems that pay out on certain conditions are subject to fraud. For me as long as there is a human involved in the decision and the benefits recipient has the right to appeal or sue there is no problem in using ML models for "finding most likely candidates for fraud". I don't know about France, but I assume it is similar to my country Poland where one can sue the state very easily with minimal expense. My parents did when I was a child and we were quite poor (self represented and all that).
bertil 3y ago
I have serious reservation about the system, but the key ones are: - It’s wrongly presented as a tool against fraud: it’s a tool to identify complex cases that are more likely to need human review. It is presented as unfair because files with mistakes can lead to asking people to pay back the money they have received in error, but it’s a big moral step to see that as the state abusing its power. - There’s no attempt to match that risk to the actual complexity of cases and decide if the model is fair. It feels unprofessional not to connect the model with observations and bias: a model where the Police makes arrest perpetuates bias because it doesn’t take crimes without arrests in un-patrolled neighborhoods. If that were the case, the article would focus on the weight in training of a sample audited at random, or about the value of families suspected to be confused that were not (less than zero). The argument “they are targeting confused people in need of help rather than big fraud cases” doesn’t make much sense: those are never very large sums. Significant fraud cases would have to be spread around many recipients told how to abuse the system, and from whom fraudsters collect. It is fair to make that point against large tax frauds, but 1. That's a different administration and 2. You don’t want a model estimating if the recipient is confused. Fraudsters are not confused; they know exactly what they are doing and why.
kamma4434 3y ago
If, as they say, “ The CNAF trains its models on households who were randomly investigated”, then the model is fair in assessing the risks of fraud. The fact that street junkie vs Google engineer have a different propensity in mugging elderly ladies is definitely partially explaine by variable household_income is not economic discrimination. You’d have a problem if you did regressions on a sample selected by the results of previous regressions, but this is not the case. And in this case you are looking for a biased sample of high-risk individuals.

200 points by sebg 3y ago

I read their report on Rotterdam's welfare fraud prediction system a few months ago and it was fascinating. Really opened my eyes to a lot of things, specifically how fast any bias can create a feedback loop that causes one demographic to be investigated more often than others.

networkchad 3y ago
[dead]
Biologist123 3y ago
The model is never the reality.
- godelski 3y ago
  This is often stated but I think often it is forgotten what is a model. It's not just "the algorithm." The metric is a model (the Wasserstein distance between norm layers of a discriminator is not image quality. Entropy isn't language). The data is a model (scraping all the internet isn't human language nor thoughts). It's important to always remember that these things too are models. Often proxies for things that are intractable. They are fine to use, but not fine to forget that they are not perfectly aligned. They are maps and to use another clique, the map is not the territory (no matter how detailed that map is).
- raybb 3y ago
  But some are useful https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong
troupo 3y ago
And this is the actual danger of AI, not the doom and gloom of "AI will wipe us out".
- ilkke 3y ago
  You don't need AI, just a government willing to abuse the most vulnerable. Check out the Australian "robodebt" fiasco [1] [1]https://en.m.wikipedia.org/wiki/Robodebt_scheme https://en.m.wikipedia.org/wiki/Robodebt_scheme
  - quickthrower2 3y ago
    Post Office Scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal
  - DeathArrow 3y ago
    How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not. By requesting welfare money, you agree that the government can investigate you.
    - danielheath 3y ago
      Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated". Governments own lawyers advised them that their plans were illegal, in advance.
      - DeathArrow 3y ago
        That sounds nasty. I was talking about the context about France's profiling algo.
      - joshxyz 3y ago
nonrandomstring 3y ago
Seems two things are at play here, the model, and how the model is used. The article was quite dry, refreshingly analytical and not overly critical of the expected ground truth that the poor are punished for being poor. It's great these models are legible and open. Looks like civil service doing a job as well as possible. The other half of the story is how, in practice these models are interpreted and what actions follow. Are further modification of the model a direct result of that and how iterative is it? Doing that aggressively changes a bare "model" into an investigative tool, not simply a thresholding utility. Sorry if I missed it, but I don't remember reading anything about how it feeds back. But what I did read was unsettling - entering people's homes at random, quizzing neighbours, pretty fascist stuff. am I wrong? I guess my point is that if you look at a bare data set, or even an algorithm, sure you can probably infer a lot about biases and intent that might be built in, but you can't see the bigger model within which this functions - and that's the real story. Is it a persecutory investigative tool? Like Hicks said we could use cruise missiles to drop food into the mouths of hungry people... a benefit system that could identify people who are struggling (of which crime is an indicator), and, I dunno, give them some money and help? That would be preachy, and quite possible imho.
- sam_lowry_ 3y ago
  > a benefit system that could identify people who are struggling > (of which crime is an indicator), and, I dunno, give them some > money and help? This is how the system should function, but then you get public outcry when the worst of us are found abusing the system of subsidies while perpetrating their crimes. I think specifically of Marc Dutroux here. On a side note, after the mass influx of Ukrainian refugees and wide public support, it is much more obvious that those in the greatest need tend to demand less, not more.
  - nonrandomstring 3y ago
    Just returning a context here [0]. Interesting case of which I was unaware, Thanks. [0] https://en.wikipedia.org/wiki/Marc_Dutroux https://en.wikipedia.org/wiki/Marc_Dutroux
VoodooJuJu 3y ago
I have limited resources for investigating robberies. I need to allocate these resources efficiently. 5x more robberies are reported in zone A than zone B. Why would I allocate an equal amount of resources to each zone? I have limited resources for investigating fraud. I need to allocate those resources efficiently. If profile A is more likely to commit fraud than profile B, why would I allocate an equal amount of resources for investigating fraud between them? I'm going on a hunting trip and have limited time and resources for hunting. I need to plan my trip accordingly. If deer are more abundant for hunting in Colorado, why would I go looking for them in the Sahara? Am I wrong to look for deer in Colorado rather than the Sahara? Am I biased for investigating fraud where it's more likely to occur? Am I evil for investigating robberies where they're reported? I have little time and resources on this planet and want to allocate my time and resources as efficiently as I can in order to do the most Good. Why would I allocate my time and resources inefficiently?
- kamma4434 3y ago
  They actually do both - they spend three quarters of their time in Colorado, but they also sample random places and use that data to decide where is the next Colorado.

dr_kiszonka 3y ago

I am curious why they used a logistic regression. Given that their "IVs” are likely highly correlated, the interpretability of the model is not great, so they could have selected something more non-linear and possibly of higher predictive power. However, they used SAS, which - in the US - would suggest to me that some sort of a statistician was involved, so they would have known about the interpretability issue. (Or maybe I need to read up on logistic regression; it has been a few years.) > An important limitation of our approach is that we do not know the relationship between the different variables that we analysed. For example, having a child between 12 and 18 (which increases risk scores) is likely correlated to being above 34 (which decreases risk scores). Doesn't France publish demographic data that would help determine these relationships? Anyway, it was a good read and many things were very clearly explained, which I appreciated.

ramraj07 3y ago
Probably a physics PhD data scientist :)
teruakohatu 3y ago
The sad state of the industry is that terrible compromised models are delivered all the time by people who should know better. I sat through a presentation on a national govt AML machine learning system, they discovered after building it that the training data was all wrong. Luckily it hasn’t been put into production yet.
- alexey-salmin 3y ago
  I don't at all see from the article how it's "terrible compromised".
benob 3y ago
Could it be that they are mandated by law, like the French banking sector for credit scoring, to use an interpretable method?
- fmajid 3y ago

Onavo 3y ago

> Finally, the raw score is run through a squasher function that outputs a probabilistic risk score between 0 and 1, where 1 is the highest risk of fraud. A raw score of 0 corresponds to a risk score of 0.5. I vote that machine learning practitioners shall henceforth refer to the softmax formula as the "squasher function". It's so much more descriptive, kudos to the French!

blt 3y ago
It's a sigmoid function, not softmax. Softmax is for multi-class classification.

extr 3y ago

Honestly impressive depth into the model. I also liked that the page didn't really politicize the issue.

DeathArrow 3y ago
> I also liked that the page didn't really politicize the issue. Are we looking at the same page? >For vulnerable recipients who struggle to navigate the complex rules of the French social security system, the risk of their benefits being reclaimed or terminated is high — even if they unintentionally committed mistakes.
- pas 3y ago
  how would you present this finding without "politicization"?

anonyme-honteux 3y ago

I'm french and I clicked on this link, worried to learn about mass surveillance programs, NSA or communist china style. But no, this time, it was about the welfare state doing its job, doing it mostly well, with details that were more impressive than I anticipated, and also opening its data to external analysts to continue improve it. That was quite an emotional ride !

boudin 3y ago
It's not the whole truth though, it's thanks to associations like La Quadrature du Net https://www.laquadrature.net/ https://www.laquadrature.net/ that they had to share their algorithm. It also shows a bias towards single familly with low income, adding pressure to families already in state of precarity. It doesn't mean it's intentional but it is still an issue. The fact that there are no proper audit of those algorithms before allowing those to be used looks quite bad in my opinion. This can be quite impactful on people's life.
- anonyme-honteux 3y ago
  I missed the context where it was associations that put the issue on the table. In the end we have an audit that show a bias against single family with low income. The bias itself is bad, but having an audit is good, it's the first step to improve things somewhat
nfeutry 3y ago
They did not want to open it, they were forced to by the "Commission d'Accès aux Document Administratifs". And the published code is probably outdated.
- anonyme-honteux 3y ago
  I missed that context, thanks!
smallpipe 3y ago
Did you read the article? The code was obtained using a FOI request, and the model targets vulnerable making mistakes in claiming benefits rather rather than actual fraud. Which bit is the one you're impressed with ? The part where you personally will be fine?

switch007 3y ago

If only the welfare departments employed the same rigour and determination when it comes to ensuring people receive benefits that they are entitled to but not claiming. Will they use they spend the same effort to ensure nobody goes hungry because of a benefit they're entitled to but aren't claiming ...? I doubt it —— Going after welfare claimants is wildly popular in the UK thanks to years of propaganda on TV dedicated to shows about people cheating the system. There is a widespread view that anyone claiming benefits is lazy and a cheat. This gives the departments carte Blanche to introduce inhumane claims processes, algorithms etc

skywal_l 3y ago
In the conclusion of the article: > It also pointed out that some investigations started by the model conclude that the CNAF actually owes the beneficiary money.
- switch007 3y ago
  That’s a defensive statement from CNAF when they were asked to comment. What do they actually do with that data in those circumstances? It’s a completely different endeavour to look for fraud and to look for under-claimed benefits. I can’t believe the same project by the same people would be equally concerned with both tasks

DeathArrow 3y ago

Well, if investigations are triggered only by algorithms, if you intend to commit fraud, you can run their statistical model against your future data and see if alarms fire off. If so, just massage the data until they don't.

a_dabbler 3y ago
I don't think those likely to commit welfare fraud are going to be worrying about the statistical model and most of these variables aren't easily modifiable (except months since last emailed) assuming the source of information is reliable
pas 3y ago
there are these random sampling checks (I guess to get a reference baseline) > The CNAF trains its models on households who were randomly investigated as part of the Paiement à bon droit Fraudes (PBDF), an annual survey
makomk 3y ago
The big practical obstacle to this is that the things this flags people as potentially committing fraud based on look to be the same things an actual fraudster would manipulate in order to get payments they're not actually entitled to - stuff line income, living situation, kids etc. So the obvious route to not setting off fraud alarms would likely involve not committing fraud, or at least not the most obvious and profitable kinds.

knallfrosch 3y ago

> which groups are more likely to be flagged. Did I miss the part where they checked whether the risk assessment was correct? I see a lof of studies like this: »'Male drivers aged 18-25 leaving clubs at night' are a group that is overrepresented in police DUI (Driving Under the Influence of alcohol) checks.« Stopping there is okay if you want to claim that any such targeting is wrongful. But the more useful information would be whether these groups are justly targeted.

mistrial9 3y ago
no - add race, spoken language, education, social status, income, politics.. those indicators are always present, and are controversial today as differentiation for street law enforcement
- snowpid 3y ago
  French republic and many other European nations dont collect on race. And "racial thinking in Europe" does not fit very well into American thinking. Where are you from?
  - mistrial9 3y ago
    California here, born American
  - bashinator 3y ago
    > French republic and many other European nations dont collect on race. Which is a convenient way for them not to track patterns of racial discrimination in their social systems.
    - snowpid 3y ago
      why should the government empowers pseudo biological bullshit? For me it is very weird to write human race into driver licence, census or university application.

gafage 3y ago

[flagged]

kemenaran 3y ago

As noted by the other investigating organization (La Quadrature du Net), the problem is worse than just the question of the algorithm implementation. The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. Why that? Because fraud is harder to detect and to prove (you have to provide evidence it was intentional). So smaller and poorer families are disproportionately affected by the controls, because the algorithm was designed to do so. No computational adjustment can fix that: it is the initial intent that is broken. Source: https://www-laquadrature-net.translate.goog/2023/11/27/notation-des-allocataires-lindecence-des-pratiques-de-la-caf-desormais-indeniable/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=fr&_x_tr_pto=wapp https://www-laquadrature-net.translate.goog/2023/11/27/notat... (auto-translated in english)

wslh 3y ago
> The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. I think your observation is the greatest one: you don't target "Madoffs or SBFs" you just look at the low hanging fruit where you look at simple probabilistic causality A => B instead of, or at the same time, targetting big malicioous actors. Big and/or complex crimes and corruption are protected. The irony is that many times the crimes commited by big actors could be simpler to analyze based on tax and financial information.
addcommitpush 3y ago
> The problem is that the CNAF deliberately targets small unintentional errors rather than large-scale intentional fraud. The French welfare system is incredibly complex - see for instance this [0] simplified description of housing allowances which is 80 (!) pages long. This is not the most complex part of the system. With such a system, there are massive amounts of errors, both too-much-money-given and not-enough-money-given. The scale is so large that the French Court of Accounts refused to certify the CNAF accounts last year[1]: thoses errors represent about 7.5% of the CAF budget. So basically the probability to have an error is just a function of how complex your situation is, and thus the "algorithm" targets more complex situations - change in your marital situation, having adult children (which may or may not need to be taken into account when applying for benefits depending on a bazillion variables), and so on, increases your probability to be targeted. [0] https://www.ecologie.gouv.fr/sites/default/files/Brochure-bareme-avril-2023-APL.pdf https://www.ecologie.gouv.fr/sites/default/files/Brochure-ba... [1] https://www.ccomptes.fr/fr/publications/certification-des-comptes-2022-du-regime-general-de-securite-sociale-et-du-cpsti https://www.ccomptes.fr/fr/publications/certification-des-co...

sylware 3y ago

So, they concluded that the people most likely to commit fraud are those under the most economic pressure? They need an algorithm to do that? As far as I know this is expected behavior of normal human beings. Geniuses all over the board. We all know the core of the issue is the following balance: - expecting money to circulate in a sane society, namely where there is not a massive amount of BS jobs, that to give a decent living to all households, is just utopia. Thinking otherwise is actually dangerous. - one of the bad sides of human nature: some "nasty" and/or really "hard" "essential" jobs have to be done, and most people will try to avoid them (many won't be able to do them anyway). Not to mention, for a "developped" country, the level of efficacy is very high and only a small amount of people rotating on those jobs (but they need to build experience anyway) is actually "needed" to make things work properly.

pas 3y ago
the problem is that this is an important system (because it directs ~80% of investigations) without transparency, audits, and on top of this when these groups started to ask questions the attitude of the government was - surprise surprise - to try to deny their requests for transparency and on top of (or beneath) all this there's the question of model performance, calibration, etc. and cost-benefit of this whole investigation department. as in the recent case of California welfare fraud ~90% (or more) was perpetrated by organized crime groups. (which is absolutely not surprising, after all it requires a big operation to file hundreds of thousands of completely fake but seemingly-valid claims.) their claim that their model is targeting ~600 or more EUR/month frauds, which covers ~98% of the cases (based on some past historic data), but there's no mention of how much sense does this make. (it's quite likely that they could get ~90% of fraud value with drastically less effort spent on investigating those who are already likely to turn out to be rightful recipients, etc.) so all in all there ought to be a very strong bias against the classic "looking for the keys under the streetlamp" mistake, which in this case means there's likely a real negative cost-benefit to spending most investigative resources on those who are already very likely to be rightful recipients of welfare. (not to mention the usual completely avoidable - and thus totally infuriating and idiotic - discontinuity problems, like single parent with a kid just turned 18, or whatever the limit is, suddenly there's a huge drop in income, huge motivation to try to smudge the numbers, and somehow get more months of payments, etc.) of course, hard data is always welcome, I'm just a regular keyboard warrior

thriftwy 3y ago

The whole idea of welfare-driven society is nuts and untenable. Welfare-driven countries should deleverage their economies and depromote price growth until they no longer need to pay welfare on mass scale due to people being able to afford housing without much strain.

logicalmonster 3y ago

Anybody notice that all of the word choices in the headline make this sound particularly dark and ominous? Wouldn't a more accurate headline be something like "Here is the math behind how France's fraud detection system works"?

pyrale 3y ago
As it should be. This system targets these people and not other types of fraud which are a larger source of fraud in terms of lost money. This correlates strongly with political discourse, in terms of who is accused of being "problematic".
- logicalmonster 3y ago
  > This system targets these people Let's keep in mind that this algorithm isn't in and of itself punishing people. It's just the starting point for a human-led investigation. Isn't that a proper application of technology? My bigger concern is that any human-led investigation is fair and reasonable and doesn't go hog-wild trying to punish people for minor misunderstandings of the rules and policies that are in place. (Somebody in this thread said that the rules just for how housing allowances can be used are like 80 pages long, so it's probably the case that it's pretty much impossible for any random citizen to know all of these rules) > other types of fraud which are a larger source of fraud in terms of lost money Surely you're not suggesting that welfare fraud shouldn't be investigated so long as say Military-Industrial-Complex fraud or Big-Pharma fraud is even greater?
  - pyrale 3y ago
    > Surely you're not suggesting that welfare fraud shouldn't be investigated so long as say Military-Industrial-Complex fraud or Big-Pharma fraud is even greater? No, I'm talking about other types of fraud effecting the social system funds. Some of them by individuals, some of them by companies. > Isn't that a proper application of technology? It's hard to apply technology properly when you don't start with good intents.

AndrewKemendo 3y ago

These kind of profile based forecasting systems *literally* encode the social biases embedded in the law into a measurement system that perfectly alienates individuals. The laws and structure of society ENSURE this is the outcome of any hierarchical capital/property based system. So it’s not the algorithm that’s the problem. The problem is having a system like this to begin with.

skrebbel 3y ago
> the social biases embedded in the law I’m not sure what you mean by this, do you have any examples?
- AndrewKemendo 3y ago
  Example: Redlining was a racist social bias embedded in the city infrastructure planning and implementation process. Or “slaves are legally 3/5th of a human” was encoded into the US constitution By encoding into the legal structure a social bias, and then using that legal precedent as the goal state, implemented via orthogonal behavior measures, you are creating a measurement system that will enforce whatever goal condition the law encoded , including any biases that were encoded in the original lawmaking process [1] https://en.m.wikipedia.org/wiki/Redlining https://en.m.wikipedia.org/wiki/Redlining [2] https://en.m.wikipedia.org/wiki/Three-fifths_Compromise https://en.m.wikipedia.org/wiki/Three-fifths_Compromise
  - skrebbel 3y ago
    To be fair I think it’s a bit odd that you only quote American examples (and century old ones at that) in a discussion about a profiling system in France. I can vaguely imagine that France may have laws, today, that are biased too, but I struggle to imagine that they’re anywhere near as bad as the blatantly racist laws America grew up with. Agree that it dehumanizes citizens though.
    - AndrewKemendo 3y ago
      I find it odd you find it odd. This is a website primarily used by Americans, which I am, and therefore have the most experience in that context. I am less familiar with French laws and you agree that there are likely laws which encoded bias. So we agree that the examples I used of bias encoding are invariant to the source and you accept the premise So the only thing you response does is give a snide jab at what you perceive as relative foundational inequities in the law. Was that your intent? To simply shit on America?

Roark66 3y ago

Sadly, all benefit systems that pay out on certain conditions are subject to fraud. For me as long as there is a human involved in the decision and the benefits recipient has the right to appeal or sue there is no problem in using ML models for "finding most likely candidates for fraud". I don't know about France, but I assume it is similar to my country Poland where one can sue the state very easily with minimal expense. My parents did when I was a child and we were quite poor (self represented and all that).

bertil 3y ago

I have serious reservation about the system, but the key ones are: - It’s wrongly presented as a tool against fraud: it’s a tool to identify complex cases that are more likely to need human review. It is presented as unfair because files with mistakes can lead to asking people to pay back the money they have received in error, but it’s a big moral step to see that as the state abusing its power. - There’s no attempt to match that risk to the actual complexity of cases and decide if the model is fair. It feels unprofessional not to connect the model with observations and bias: a model where the Police makes arrest perpetuates bias because it doesn’t take crimes without arrests in un-patrolled neighborhoods. If that were the case, the article would focus on the weight in training of a sample audited at random, or about the value of families suspected to be confused that were not (less than zero). The argument “they are targeting confused people in need of help rather than big fraud cases” doesn’t make much sense: those are never very large sums. Significant fraud cases would have to be spread around many recipients told how to abuse the system, and from whom fraudsters collect. It is fair to make that point against large tax frauds, but 1. That's a different administration and 2. You don’t want a model estimating if the recipient is confused. Fraudsters are not confused; they know exactly what they are doing and why.

kamma4434 3y ago

If, as they say, “ The CNAF trains its models on households who were randomly investigated”, then the model is fair in assessing the risks of fraud. The fact that street junkie vs Google engineer have a different propensity in mugging elderly ladies is definitely partially explaine by variable household_income is not economic discrimination. You’d have a problem if you did regressions on a sample selected by the results of previous regressions, but this is not the case. And in this case you are looking for a biased sample of high-risk individuals.