4 ms·

I am curious why they used a logistic regression. Given that their "IVs” are likely highly correlated, the interpretability of the model is not great, so they c

I am curious why they used a logistic regression. Given that their "IVs” are likely highly correlated, the interpretability of the model is not great, so they could have selected something more non-linear and possibly of higher predictive power. However, they used SAS, which - in the US - would suggest to me that some sort of a statistician was involved, so they would have known about the interpretability issue. (Or maybe I need to read up on logistic regression; it has been a few years.) > An important limitation of our approach is that we do not know the relationship between the different variables that we analysed. For example, having a child between 12 and 18 (which increases risk scores) is likely correlated to being above 34 (which decreases risk scores). Doesn't France publish demographic data that would help determine these relationships? Anyway, it was a good read and many things were very clearly explained, which I appreciated.

ramraj07 3y ago
Probably a physics PhD data scientist :)
teruakohatu 3y ago
The sad state of the industry is that terrible compromised models are delivered all the time by people who should know better. I sat through a presentation on a national govt AML machine learning system, they discovered after building it that the training data was all wrong. Luckily it hasn’t been put into production yet.
- alexey-salmin 3y ago
  I don't at all see from the article how it's "terrible compromised".
benob 3y ago
Could it be that they are mandated by law, like the French banking sector for credit scoring, to use an interpretable method?
- fmajid 3y ago
  The EU as a whole requires models used for automated decision-making to be interpretable, and logistic regression is highly so. Benefits fraud is a big problem in France, and this investigation shows an agency doing its job properly.
  - orwin 3y ago
    The social fraud in France, combining all type of social fraud, is estimated at 1.1 billions (probably more, because Healthcare fraud is massively underestimated and growing, due to private clinics helping a lot with producing misleading claims and documents, but considering The total healthcare budget, it cannot be that much higher). Corporate tax fraud is 27 billions Social security contribution fraud is estimated at 14 billions (to compare to 200millions for the share of social security fraud) Income tax fraud is 17 billions (and wages are automatically declared, so this fraud is mostly on non-work income).
    - CalRobert 3y ago
      Perhaps it's related to visibility and a feeling of "fairness". I completely understand all of the things you wrote, and the corporate tax fraud is a MUCH bigger problem. But for some reason when my neighbour told me she'd been lying about not being able to work for the last 12 years so she could collect Jobseeker's allowance (basically "unemployment insurance" for those in the US) it really stung and felt like I, personally, was paying for her. I got a lot more conservative after I moved to the countryside and realized how casual fraud and tax evasion were (I was routinely asked to pay for several-thousand-euro jobs in cash, and tradespeople balked at me for wanting a receipt)
      - orwin 3y ago
        Ahah, i'm from the countryside, I know what you're talking about. Weirdly, social contribution fraud and income tax fraud ('travail au noir') is present in really big cities (mostly because the cost of getting caught is quite low compared to the margin of construction business in Paris/Lyon), and in rural areas, because no one ever check (it's changing, slowly). >> felt like I, personally, was paying for her. You also helped found Vlc, and probably half of the French startups that exist today. I agree that this is not fair (I do not ask for insurance when I'm in between jobs, because I have the means to ignore it now), but also, it will last two year at most.
extr 3y ago
It's most certainly done for interpretability reasons. A non-linear model would provide huge lift here. Another option could be to run a non-linear model off to the side and understand the most powerful interaction terms and incorporate that into the linear model. This same problem is faced by insurance companies vs insurance regulators.
- dr_kiszonka 3y ago
  Right, but they can't really interpret this model, even if it does a good job at prediction. Two references, in case they ever read this :) 1. https://www.tandfonline.com/doi/abs/10.1080/09720502.2010.10700699 https://www.tandfonline.com/doi/abs/10.1080/09720502.2010.10... [1051 citations] 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4888898/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4888898/ [1032 citations]
  - pas 3y ago
    for those less blessed with a priory knowledge or time (and/or motivation to chew through these articles) can I ask you for a tl;dr? maybe could you show an example where the linear model fails? and what's an non-linear model that would overcome this problem? (also I have no idea whether it's related, but ... my very sparse knowledge of this field just regurgitates the Statmodeling blog (by Andrew Gelman and co.), which is basically something something monte carlo + hierarchical models + calibration... but of course it's easy to shout buzzwords.)
    - extr 3y ago
      The tl;dr is multicollinearity is a big problem for interpreting models like this. Aka you have variables in the model that are highly correlated and because of this the model does not necessarily assign weights to the variables in a way that means anything. Eg: In the article they show "Age: 34 or Older" = -.11 and also that "Married" = -.49. Being over 34 and married is -.60. But as far as we know the model doesn't account for the fact that people older than 34 are more likely to be married in the first place. Perhaps the model would have worked just as well if "Age: 34 or Older" = -.30 and also that "Married" = -.30 (still -.60). And so on with every possible combination of attributes (they are all correlated). This isn't the end of the story. In situations where interpretability really matters or you have the time/resources, it's possible to mostly prevent this by building the model slowly and diagnosing multicollinearity issues as they arise. Perhaps you purposefully leave out variables because you see they are already highly correlated with existing ones and leave the model weights unstable. It requires exhaustive analysis and necessarily involves subjectivity, though. The researchers here could have done something like this, it's certainly done in industry for the aforementioned financial regulatory environments. Non-linear models like trees don't have multicollinearity problems because they implicitly model interactions between variables with the branching behavior. Instead the problem is interpretation, it's hard to sum up all those branches in a way that is intuitive and meaningful.
addcommitpush 3y ago
> I am curious why they used a logistic regression. CNAF is not a competitive employer. French public service is essentially split in two: the State, and the various "Social Security" bodies which manages most of the welfare system (among them, the Caisse Nationale des Allocations Familiales which manages part of social & family welfare). The "State" side has a very competitive track for stats/econ/ML employees, namely the national statistics office members (INSEE). As the name does _not_ indicate, they work everywhere in the government, not just at INSEE. So if you're good at stats and in public service, the odds you end up in the CNAF team instead of the statistiscal office of the Social Ministry is about zero. Hence SAS. Hence logit. And so forth.

ramraj07 3y ago

Probably a physics PhD data scientist :)

teruakohatu 3y ago

The sad state of the industry is that terrible compromised models are delivered all the time by people who should know better. I sat through a presentation on a national govt AML machine learning system, they discovered after building it that the training data was all wrong. Luckily it hasn’t been put into production yet.

alexey-salmin 3y ago
I don't at all see from the article how it's "terrible compromised".

benob 3y ago

Could it be that they are mandated by law, like the French banking sector for credit scoring, to use an interpretable method?

fmajid 3y ago
The EU as a whole requires models used for automated decision-making to be interpretable, and logistic regression is highly so. Benefits fraud is a big problem in France, and this investigation shows an agency doing its job properly.
- orwin 3y ago
  The social fraud in France, combining all type of social fraud, is estimated at 1.1 billions (probably more, because Healthcare fraud is massively underestimated and growing, due to private clinics helping a lot with producing misleading claims and documents, but considering The total healthcare budget, it cannot be that much higher). Corporate tax fraud is 27 billions Social security contribution fraud is estimated at 14 billions (to compare to 200millions for the share of social security fraud) Income tax fraud is 17 billions (and wages are automatically declared, so this fraud is mostly on non-work income).
  - CalRobert 3y ago
    Perhaps it's related to visibility and a feeling of "fairness". I completely understand all of the things you wrote, and the corporate tax fraud is a MUCH bigger problem. But for some reason when my neighbour told me she'd been lying about not being able to work for the last 12 years so she could collect Jobseeker's allowance (basically "unemployment insurance" for those in the US) it really stung and felt like I, personally, was paying for her. I got a lot more conservative after I moved to the countryside and realized how casual fraud and tax evasion were (I was routinely asked to pay for several-thousand-euro jobs in cash, and tradespeople balked at me for wanting a receipt)
    - orwin 3y ago
      Ahah, i'm from the countryside, I know what you're talking about. Weirdly, social contribution fraud and income tax fraud ('travail au noir') is present in really big cities (mostly because the cost of getting caught is quite low compared to the margin of construction business in Paris/Lyon), and in rural areas, because no one ever check (it's changing, slowly). >> felt like I, personally, was paying for her. You also helped found Vlc, and probably half of the French startups that exist today. I agree that this is not fair (I do not ask for insurance when I'm in between jobs, because I have the means to ignore it now), but also, it will last two year at most.

extr 3y ago

It's most certainly done for interpretability reasons. A non-linear model would provide huge lift here. Another option could be to run a non-linear model off to the side and understand the most powerful interaction terms and incorporate that into the linear model. This same problem is faced by insurance companies vs insurance regulators.

dr_kiszonka 3y ago
Right, but they can't really interpret this model, even if it does a good job at prediction. Two references, in case they ever read this :) 1. https://www.tandfonline.com/doi/abs/10.1080/09720502.2010.10700699 https://www.tandfonline.com/doi/abs/10.1080/09720502.2010.10... [1051 citations] 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4888898/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4888898/ [1032 citations]
- pas 3y ago
  for those less blessed with a priory knowledge or time (and/or motivation to chew through these articles) can I ask you for a tl;dr? maybe could you show an example where the linear model fails? and what's an non-linear model that would overcome this problem? (also I have no idea whether it's related, but ... my very sparse knowledge of this field just regurgitates the Statmodeling blog (by Andrew Gelman and co.), which is basically something something monte carlo + hierarchical models + calibration... but of course it's easy to shout buzzwords.)
  - extr 3y ago
    The tl;dr is multicollinearity is a big problem for interpreting models like this. Aka you have variables in the model that are highly correlated and because of this the model does not necessarily assign weights to the variables in a way that means anything. Eg: In the article they show "Age: 34 or Older" = -.11 and also that "Married" = -.49. Being over 34 and married is -.60. But as far as we know the model doesn't account for the fact that people older than 34 are more likely to be married in the first place. Perhaps the model would have worked just as well if "Age: 34 or Older" = -.30 and also that "Married" = -.30 (still -.60). And so on with every possible combination of attributes (they are all correlated). This isn't the end of the story. In situations where interpretability really matters or you have the time/resources, it's possible to mostly prevent this by building the model slowly and diagnosing multicollinearity issues as they arise. Perhaps you purposefully leave out variables because you see they are already highly correlated with existing ones and leave the model weights unstable. It requires exhaustive analysis and necessarily involves subjectivity, though. The researchers here could have done something like this, it's certainly done in industry for the aforementioned financial regulatory environments. Non-linear models like trees don't have multicollinearity problems because they implicitly model interactions between variables with the branching behavior. Instead the problem is interpretation, it's hard to sum up all those branches in a way that is intuitive and meaningful.

addcommitpush 3y ago

> I am curious why they used a logistic regression. CNAF is not a competitive employer. French public service is essentially split in two: the State, and the various "Social Security" bodies which manages most of the welfare system (among them, the Caisse Nationale des Allocations Familiales which manages part of social & family welfare). The "State" side has a very competitive track for stats/econ/ML employees, namely the national statistics office members (INSEE). As the name does _not_ indicate, they work everywhere in the government, not just at INSEE. So if you're good at stats and in public service, the odds you end up in the CNAF team instead of the statistiscal office of the Social Ministry is about zero. Hence SAS. Hence logit. And so forth.