3 ms·

The coming war on the hidden algorithms that trap people in poverty

16 points by Digit-Al 6y ago

avmich 6y ago
Things like this make you wonder how system works at all, if a decision can't be explained. Shouldn't it - obviously - be thrown out in a case of a dispute for this reason alone?
- CogitoCogito 6y ago
  Yeah I believe that essentially all of the actors in that article (private or public) should be legally required to explain all reasoning for decisions made and provide redress for mistakes. This should be done quickly. It all should be transparent. If any of these requirements means the current business models or contracts are untenable, so be it. Efficiency isn't everything. It definitely isn't worth leaving random victims at the wayside.
  - ClumsyPilot 6y ago
    >"Efficiency isn't everything." Can they even prove it's 'more efficient', and not marketing hype or a mistake?
    - CogitoCogito 6y ago
      I think that’s a good point and I’m not sure I can answer the question (without following back on some unsupported and self-fulfilling a priori assumption like “the market chooses the most efficient solutions”). However, my point is just that I think that’s the wrong focus. Even if it could be proved, I don’t think that’s enough.
jjk166 6y ago
> He realized that nurses were telling patients, “Well, the computer did it—it’s not me.” “That’s what tipped us off,” he says. This seems to be the real crux of the issue. These industrial-scale algorithms should be treated the same as any other industrial equipment. If you run over someone with a bulldozer, you can't say "the bulldozer did it, not me." Using the excuse "I don't even know how to operate a bulldozer" only makes it worse. I think it's perfectly reasonable that if an institution doesn't train people to properly use an algorithm, take reasonable precautions to make sure it doesn't cause severe damage, and/or has no reasonable system in place for fixing problems it creates, they should be liable for negligence the same way any other operator of potentially dangerous equipment would.
- Digit-Al 6y ago
  A good point well made.
sthnblllII 6y ago
The economy has become impersonal. In the past, there was one county bank, one regional college, and usually one large employer in town. If someone never paid their debts, the bank manager wouldn't lend them any more. If they didn't do well in school, the dean wouldn't admit them to the college. If they didn't show up for work, the foreman wouldn't hire them back. Did people accuse these decision makers of "keeping" unreliable people poor? No. Today there are many banks, but if you don't pay back a loan, your bank tells the others and you wont get another from any of them. There are many universities, but the all have similar admission criteria, etc. The fact that computers instead of people make these simple calculations isn't a problem. My concern is that algorithms will be used to create affirmative action for things that were never possible before. Some people will start paying higher prices on Amazon and not know it. Groups that organize politically will be able to tip the scales in their favor by claiming the unbiased algorithm "keeps them down" or that the biases people develop against them must be offset by opposite biases in the algorithms. Whereas before every outcome was the product of decisions that every human being got to make about their voluntary interactions, now the reigns of power in the economy are centralized in the hands of whoever gets to design the algorithms and the economy is superseded by a zero-sum political struggle. That is the biggest danger of algorithms.

16 points by Digit-Al 6y ago

Things like this make you wonder how system works at all, if a decision can't be explained. Shouldn't it - obviously - be thrown out in a case of a dispute for this reason alone?

CogitoCogito 6y ago
Yeah I believe that essentially all of the actors in that article (private or public) should be legally required to explain all reasoning for decisions made and provide redress for mistakes. This should be done quickly. It all should be transparent. If any of these requirements means the current business models or contracts are untenable, so be it. Efficiency isn't everything. It definitely isn't worth leaving random victims at the wayside.
- ClumsyPilot 6y ago
  >"Efficiency isn't everything." Can they even prove it's 'more efficient', and not marketing hype or a mistake?
  - CogitoCogito 6y ago
    I think that’s a good point and I’m not sure I can answer the question (without following back on some unsupported and self-fulfilling a priori assumption like “the market chooses the most efficient solutions”). However, my point is just that I think that’s the wrong focus. Even if it could be proved, I don’t think that’s enough.

jjk166 6y ago

> He realized that nurses were telling patients, “Well, the computer did it—it’s not me.” “That’s what tipped us off,” he says. This seems to be the real crux of the issue. These industrial-scale algorithms should be treated the same as any other industrial equipment. If you run over someone with a bulldozer, you can't say "the bulldozer did it, not me." Using the excuse "I don't even know how to operate a bulldozer" only makes it worse. I think it's perfectly reasonable that if an institution doesn't train people to properly use an algorithm, take reasonable precautions to make sure it doesn't cause severe damage, and/or has no reasonable system in place for fixing problems it creates, they should be liable for negligence the same way any other operator of potentially dangerous equipment would.

Digit-Al 6y ago
A good point well made.

sthnblllII 6y ago

The economy has become impersonal. In the past, there was one county bank, one regional college, and usually one large employer in town. If someone never paid their debts, the bank manager wouldn't lend them any more. If they didn't do well in school, the dean wouldn't admit them to the college. If they didn't show up for work, the foreman wouldn't hire them back. Did people accuse these decision makers of "keeping" unreliable people poor? No. Today there are many banks, but if you don't pay back a loan, your bank tells the others and you wont get another from any of them. There are many universities, but the all have similar admission criteria, etc. The fact that computers instead of people make these simple calculations isn't a problem. My concern is that algorithms will be used to create affirmative action for things that were never possible before. Some people will start paying higher prices on Amazon and not know it. Groups that organize politically will be able to tip the scales in their favor by claiming the unbiased algorithm "keeps them down" or that the biases people develop against them must be offset by opposite biases in the algorithms. Whereas before every outcome was the product of decisions that every human being got to make about their voluntary interactions, now the reigns of power in the economy are centralized in the hands of whoever gets to design the algorithms and the economy is superseded by a zero-sum political struggle. That is the biggest danger of algorithms.