3 ms·

It's a bit like all the arbitrary account terminations we hear about with Google. A machine makes a decision with virtually no recourse for the user. When gover

It's a bit like all the arbitrary account terminations we hear about with Google. A machine makes a decision with virtually no recourse for the user. When governments are applying this same concept to welfare, it does indeed create a dystopian scenario where people are left to starve because some algorithm got it wrong. In essence, it's about cost cutting. Governments seem to think that machines can replace humans in making decisions where there is nuance involved and the stakes are extremely high. It is frightening and should not be accepted. The furthest the automation should go is to flag irregularities for review. Instead, the machines are given far too much autonomy, with the robodebt collection being a scary example.

Clubber 7y ago
One of the problems with the nature of macroeconomic / statistical thinking is it removes any humanity from the decision, because we focus on just numbers. Anytime we discuss policy as citizens, we really need to keep in mind that every 1 in a number represents an actual individual which an actual family and life and life experiences. An example is raising the social security age from 65 to 67. How many lives did that simple change affect? What repercussions did that have on that individual's family? We don't really think about it, we just think about it's effects on the bottom line. "One death is a tragedy; a million is a statistic" -Joseph Stalin (1879-1953).
hcarvalhoalves 7y ago
A human isn't necessarily less arbitrary, in fact it can be much worse. You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human". Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) and benchmark against that - the same way you would measure if a department comprised of humans were doing the same job.
- kazagistar 7y ago
  It's a matter of incentives. In non automation, someone can be on the hook if things go wrong, so they have an incentive to be right, even if they are flawed and make lots of mistakes. In automation you spread responsibility out, and things aren't so clear. Automation can be better, more accurate, and in general kinder, but it can also be more profitable, efficient, uncaring and cruel, and when its pointed out the people at fault can avoid blame far far easier.
- Majromax 7y ago
  > You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human". Not necessarily. Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal. But what happens when the decision is made by an algorithm, especially one that wasn't built to give an explicable reason? > Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) That's not a good metric. Suppose the algorithm were in fact perfect and nobody starved -- except you. For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm would be doing a fantastic job compared to an imperfect human system, but it would also be profoundly unjust.
  - hcarvalhoalves 7y ago
    > Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal. So the problem is not automation, is lack of recourse. Lack of recourse is a real problem that already exists today. People do go unattended by not knowing how to navigate bureaucratic structures, and not having money to pay a lawyer. It also affects the most the countries the Guardian article targets as "automating poverty". > For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm This is not an algorithmic issue. If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either.
    - Faark 7y ago
      > So the problem is not automation, is lack of recourse. The problem is automation, if it exaggerates the "lack of recourse". It means that, without being careful, adding automation would screw up most of such systems. Thus such articles to boost awareness and make sure automation is done well. I'm disappointed with what a hard time many on HN seem to have with accepting the need to be cautious. Especially with Gov, were you cannot just find or build a better alternative on the open market. > If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either. Is that actually the case in the US? The programs (run by the church and heavily supported by german gov) I've come in contact with certainly did not. Never was at a soup kitchen, but would be surprised for them to ask for IDs.

Clubber 7y ago

One of the problems with the nature of macroeconomic / statistical thinking is it removes any humanity from the decision, because we focus on just numbers. Anytime we discuss policy as citizens, we really need to keep in mind that every 1 in a number represents an actual individual which an actual family and life and life experiences. An example is raising the social security age from 65 to 67. How many lives did that simple change affect? What repercussions did that have on that individual's family? We don't really think about it, we just think about it's effects on the bottom line. "One death is a tragedy; a million is a statistic" -Joseph Stalin (1879-1953).

hcarvalhoalves 7y ago

A human isn't necessarily less arbitrary, in fact it can be much worse. You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human". Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) and benchmark against that - the same way you would measure if a department comprised of humans were doing the same job.

kazagistar 7y ago
It's a matter of incentives. In non automation, someone can be on the hook if things go wrong, so they have an incentive to be right, even if they are flawed and make lots of mistakes. In automation you spread responsibility out, and things aren't so clear. Automation can be better, more accurate, and in general kinder, but it can also be more profitable, efficient, uncaring and cruel, and when its pointed out the people at fault can avoid blame far far easier.
Majromax 7y ago
> You (and Guardian, it seems) are falling for the fallacy of the danger of automation - "it's acceptable for bad decisions to be made as long as it's been decided by a human". Not necessarily. Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal. But what happens when the decision is made by an algorithm, especially one that wasn't built to give an explicable reason? > Fear mongering is not how you prove/disprove automation, you should pick a useful metric (e.g. are people starving?) That's not a good metric. Suppose the algorithm were in fact perfect and nobody starved -- except you. For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm would be doing a fantastic job compared to an imperfect human system, but it would also be profoundly unjust.
- hcarvalhoalves 7y ago
  > Even bureaucratic systems tend to recognize that humans make imperfect decisions -- we're susceptible to everything from bribery to exhaustion. These systems then tend to not treat first decisions as final, and they allow an opportunity to appeal. So the problem is not automation, is lack of recourse. Lack of recourse is a real problem that already exists today. People do go unattended by not knowing how to navigate bureaucratic structures, and not having money to pay a lawyer. It also affects the most the countries the Guardian article targets as "automating poverty". > For some reason, it couldn't recognize your ID (and only your ID). By the "are people starving" metric, the algorithm This is not an algorithmic issue. If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either.
  - Faark 7y ago
    > So the problem is not automation, is lack of recourse. The problem is automation, if it exaggerates the "lack of recourse". It means that, without being careful, adding automation would screw up most of such systems. Thus such articles to boost awareness and make sure automation is done well. I'm disappointed with what a hard time many on HN seem to have with accepting the need to be cautious. Especially with Gov, were you cannot just find or build a better alternative on the open market. > If you lost your ID, you wouldn't get your benefit even if you walked into a social security branch either. Is that actually the case in the US? The programs (run by the church and heavily supported by german gov) I've come in contact with certainly did not. Never was at a soup kitchen, but would be surprised for them to ask for IDs.