6 ms·

I read their report on Rotterdam's welfare fraud prediction system a few months ago and it was fascinating. Really opened my eyes to a lot of things, specifical

I read their report on Rotterdam's welfare fraud prediction system a few months ago and it was fascinating. Really opened my eyes to a lot of things, specifically how fast any bias can create a feedback loop that causes one demographic to be investigated more often than others.

networkchad 3y ago
[dead]
Biologist123 3y ago
The model is never the reality.
- godelski 3y ago
  This is often stated but I think often it is forgotten what is a model. It's not just "the algorithm." The metric is a model (the Wasserstein distance between norm layers of a discriminator is not image quality. Entropy isn't language). The data is a model (scraping all the internet isn't human language nor thoughts). It's important to always remember that these things too are models. Often proxies for things that are intractable. They are fine to use, but not fine to forget that they are not perfectly aligned. They are maps and to use another clique, the map is not the territory (no matter how detailed that map is).
- raybb 3y ago
  But some are useful https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong
troupo 3y ago
And this is the actual danger of AI, not the doom and gloom of "AI will wipe us out".
- ilkke 3y ago
  You don't need AI, just a government willing to abuse the most vulnerable. Check out the Australian "robodebt" fiasco [1] [1]https://en.m.wikipedia.org/wiki/Robodebt_scheme https://en.m.wikipedia.org/wiki/Robodebt_scheme
  - quickthrower2 3y ago
    Post Office Scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal
  - DeathArrow 3y ago
    How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not. By requesting welfare money, you agree that the government can investigate you.
    - danielheath 3y ago
      Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated". Governments own lawyers advised them that their plans were illegal, in advance.
      - DeathArrow 3y ago
        That sounds nasty. I was talking about the context about France's profiling algo.
      - joshxyz 3y ago
        guilty until proven innocent, yikes
    - quadcore 3y ago
nonrandomstring 3y ago
Seems two things are at play here, the model, and how the model is used. The article was quite dry, refreshingly analytical and not overly critical of the expected ground truth that the poor are punished for being poor. It's great these models are legible and open. Looks like civil service doing a job as well as possible. The other half of the story is how, in practice these models are interpreted and what actions follow. Are further modification of the model a direct result of that and how iterative is it? Doing that aggressively changes a bare "model" into an investigative tool, not simply a thresholding utility. Sorry if I missed it, but I don't remember reading anything about how it feeds back. But what I did read was unsettling - entering people's homes at random, quizzing neighbours, pretty fascist stuff. am I wrong? I guess my point is that if you look at a bare data set, or even an algorithm, sure you can probably infer a lot about biases and intent that might be built in, but you can't see the bigger model within which this functions - and that's the real story. Is it a persecutory investigative tool? Like Hicks said we could use cruise missiles to drop food into the mouths of hungry people... a benefit system that could identify people who are struggling (of which crime is an indicator), and, I dunno, give them some money and help? That would be preachy, and quite possible imho.
- sam_lowry_ 3y ago
  > a benefit system that could identify people who are struggling > (of which crime is an indicator), and, I dunno, give them some > money and help? This is how the system should function, but then you get public outcry when the worst of us are found abusing the system of subsidies while perpetrating their crimes. I think specifically of Marc Dutroux here. On a side note, after the mass influx of Ukrainian refugees and wide public support, it is much more obvious that those in the greatest need tend to demand less, not more.
  - nonrandomstring 3y ago
    Just returning a context here [0]. Interesting case of which I was unaware, Thanks. [0] https://en.wikipedia.org/wiki/Marc_Dutroux https://en.wikipedia.org/wiki/Marc_Dutroux
VoodooJuJu 3y ago
I have limited resources for investigating robberies. I need to allocate these resources efficiently. 5x more robberies are reported in zone A than zone B. Why would I allocate an equal amount of resources to each zone? I have limited resources for investigating fraud. I need to allocate those resources efficiently. If profile A is more likely to commit fraud than profile B, why would I allocate an equal amount of resources for investigating fraud between them? I'm going on a hunting trip and have limited time and resources for hunting. I need to plan my trip accordingly. If deer are more abundant for hunting in Colorado, why would I go looking for them in the Sahara? Am I wrong to look for deer in Colorado rather than the Sahara? Am I biased for investigating fraud where it's more likely to occur? Am I evil for investigating robberies where they're reported? I have little time and resources on this planet and want to allocate my time and resources as efficiently as I can in order to do the most Good. Why would I allocate my time and resources inefficiently?
- kamma4434 3y ago
  They actually do both - they spend three quarters of their time in Colorado, but they also sample random places and use that data to decide where is the next Colorado.

networkchad 3y ago

[dead]

Biologist123 3y ago

The model is never the reality.

godelski 3y ago
This is often stated but I think often it is forgotten what is a model. It's not just "the algorithm." The metric is a model (the Wasserstein distance between norm layers of a discriminator is not image quality. Entropy isn't language). The data is a model (scraping all the internet isn't human language nor thoughts). It's important to always remember that these things too are models. Often proxies for things that are intractable. They are fine to use, but not fine to forget that they are not perfectly aligned. They are maps and to use another clique, the map is not the territory (no matter how detailed that map is).
raybb 3y ago
But some are useful https://en.m.wikipedia.org/wiki/All_models_are_wrong https://en.m.wikipedia.org/wiki/All_models_are_wrong

troupo 3y ago

And this is the actual danger of AI, not the doom and gloom of "AI will wipe us out".

ilkke 3y ago
You don't need AI, just a government willing to abuse the most vulnerable. Check out the Australian "robodebt" fiasco [1] [1]https://en.m.wikipedia.org/wiki/Robodebt_scheme https://en.m.wikipedia.org/wiki/Robodebt_scheme
- quickthrower2 3y ago
  Post Office Scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal https://en.m.wikipedia.org/wiki/British_Post_Office_scandal
- DeathArrow 3y ago
  How does an investigation equal abuse? If you did no wrong, you don't have to fear any investigation about what you did with the money, and whether you deserve it or not. By requesting welfare money, you agree that the government can investigate you.
  - danielheath 3y ago
    Robodebt wasn't "an investigation". It was "Our algorithm (which we know is very frequently wrong) guessed you might have money, so we've retroactively cancelled your benefits and will come after you for fraud until you can prove your innocence via an appeal process that's been deliberately obfuscated". Governments own lawyers advised them that their plans were illegal, in advance.
    - DeathArrow 3y ago
      That sounds nasty. I was talking about the context about France's profiling algo.
    - joshxyz 3y ago
      guilty until proven innocent, yikes
  - quadcore 3y ago

nonrandomstring 3y ago

Seems two things are at play here, the model, and how the model is used. The article was quite dry, refreshingly analytical and not overly critical of the expected ground truth that the poor are punished for being poor. It's great these models are legible and open. Looks like civil service doing a job as well as possible. The other half of the story is how, in practice these models are interpreted and what actions follow. Are further modification of the model a direct result of that and how iterative is it? Doing that aggressively changes a bare "model" into an investigative tool, not simply a thresholding utility. Sorry if I missed it, but I don't remember reading anything about how it feeds back. But what I did read was unsettling - entering people's homes at random, quizzing neighbours, pretty fascist stuff. am I wrong? I guess my point is that if you look at a bare data set, or even an algorithm, sure you can probably infer a lot about biases and intent that might be built in, but you can't see the bigger model within which this functions - and that's the real story. Is it a persecutory investigative tool? Like Hicks said we could use cruise missiles to drop food into the mouths of hungry people... a benefit system that could identify people who are struggling (of which crime is an indicator), and, I dunno, give them some money and help? That would be preachy, and quite possible imho.

sam_lowry_ 3y ago
> a benefit system that could identify people who are struggling > (of which crime is an indicator), and, I dunno, give them some > money and help? This is how the system should function, but then you get public outcry when the worst of us are found abusing the system of subsidies while perpetrating their crimes. I think specifically of Marc Dutroux here. On a side note, after the mass influx of Ukrainian refugees and wide public support, it is much more obvious that those in the greatest need tend to demand less, not more.
- nonrandomstring 3y ago
  Just returning a context here [0]. Interesting case of which I was unaware, Thanks. [0] https://en.wikipedia.org/wiki/Marc_Dutroux https://en.wikipedia.org/wiki/Marc_Dutroux

VoodooJuJu 3y ago

I have limited resources for investigating robberies. I need to allocate these resources efficiently. 5x more robberies are reported in zone A than zone B. Why would I allocate an equal amount of resources to each zone? I have limited resources for investigating fraud. I need to allocate those resources efficiently. If profile A is more likely to commit fraud than profile B, why would I allocate an equal amount of resources for investigating fraud between them? I'm going on a hunting trip and have limited time and resources for hunting. I need to plan my trip accordingly. If deer are more abundant for hunting in Colorado, why would I go looking for them in the Sahara? Am I wrong to look for deer in Colorado rather than the Sahara? Am I biased for investigating fraud where it's more likely to occur? Am I evil for investigating robberies where they're reported? I have little time and resources on this planet and want to allocate my time and resources as efficiently as I can in order to do the most Good. Why would I allocate my time and resources inefficiently?

kamma4434 3y ago
They actually do both - they spend three quarters of their time in Colorado, but they also sample random places and use that data to decide where is the next Colorado.