4 ms·

There is a scene at the beginning of the movie Brazil [1] where a literal bug falls into a tele-type, causes an error that starts a chain reaction that frames t

There is a scene at the beginning of the movie Brazil [1] where a literal bug falls into a tele-type, causes an error that starts a chain reaction that frames the events of the movie. The bug is a metaphor in the movie, just some nice visual story-telling to indicate how errors in automated processes can have unintended consequences. I honestly fear that entire lives will be ruined by AI systems mis-analyzing some data and locking people out from education, work, credit and health opportunities. For some reason, which will be mostly inexplicable to even the engineers that trained the ML system, people will get denied or flagged for spurious reasons. As we automate everything this is inevitable. We're actively creating digital gatekeepers. 99% of the time you won't even realize it has happened to you nor would you have any recourse. It will be as innocent as an application you don't get a response from, or a simple generic "sorry" email. Then the brick wall of literal no customer service, or automated customer service that refuses to escalate you to a human tier. Then maybe if you are lucky a clueless customer service rep that compassionately explains there is nothing they can do and they wish you the best with your continued search. Fair enough their TOS likely allows them to deny service to anyone for any reason. Why would they waste the effort to figure out why their multi-million dollar system targeted you as an edge case? It works for 98% of cases which is probably good enough for them. Brazil is a fictionalization that imagines an extreme case and the events of the movie get quite dark. The reality will be more mundane but IMO just as insidious. In the not very distant future AI will be choosing who is healthy and who is rich. 1. https://www.youtube.com/watch?v=XGge4rj4v_Y https://www.youtube.com/watch?v=XGge4rj4v_Y

snagglegaggle 7y ago
I'm not sure we can solve this as it relates to AI/ML if we can't even solve it as it exists in human society today. Various heuristics are used, knowingly or not, to gate access to employment, education, healthcare, etc. As an example I had to drop out of education (in the US) due to mental health issues. 6-7 years later, I'm still having problems getting back in as I wasn't able to get the problems documented in time. I also am unable to get any money as most of my classmates were despite my far above average performance pre-uni.
zxcb1 7y ago
Will these systems ever forgive or forget, like humans?
ssss11 7y ago
We need to stop that future don’t we?
Accujack 7y ago
>As we automate everything this is inevitable. Which is why it's important to ask the question "should we" instead of just "can we". Even more important is not dividing people into groups that we treat differently. Everyone should have the same education, work, credit and health opportunities. If we can free ourselves from the desire to segregate and separate people in the name of progress, whether or not an "AI" guesses wrong about you is irrelevant. It comes down to ego... people have to accept that no one can predict someone's potential or value to the world with enough certainty to make restricting people's opportunities worth it.
phs318u 7y ago
We’re already seeing the impacts of these kinds of systems affecting many of the most vulnerable in the Australian community. I’m referring to the Dept. of Human Services’ punitive “robodebt” system. https://mobile.abc.net.au/news/2017-03-03/centrelink-debt-controversy-what-is-robodebt/8317764 https://mobile.abc.net.au/news/2017-03-03/centrelink-debt-co...
falcor84 7y ago
But is this really about AI? Almost all aspects of our lives are already controlled by the enormously complex soulless machines of bureaucracy we call firms and governments. Is there any essential difference between a computer bug and a clerical error? If anything, I would assume the former is easier to fix.
- OnlineGladiator 7y ago
  I agree. I think the real question (which is much broader and beyond the scope of OP) is how does humanity have to operate differently with every order of magnitude it grows in size? Humanity seems to be transitioning from single digit billions to crossing the double digit billions barrier in the foreseeable future. Until 1900 humanity had been on the order of a billion people, and we are currently transitioning towards 10x as many people sustaining the planet. I do not mean this to be about finite resources and our planet. I am referring to the way that organizations require more layers of bureaucracy as they grow. A rule of thumb is for every 10x in people, you have another layer of management. So what will this extra layer of management be for people, not just from a government perspective but also a capitalistic corporation perspective? How do Google, Facebook, Apple, Stripe, Amazon, and all of those other companies handle so many customers? How much do they use automation, how many of their rules are archaic, how hard is it to challenge the system if you think there was an error? I am not convinced AI can solve many of the fundamental problems humanity are facing - and I think overpopulation might actually be the problem. Modern capitalism has conditioned us to believe "growth is good" but growth for the sake of growth is no different from cancer.
- ljm 7y ago
  But that’s not how it works, it’s not one in place of another; it’s one that depends on another. If your physical bureaucratic entity delegates decision making to an AI, you have no hope of redress unless due process was mandated. If you cut the bureaucratic middleman then you have to build an AI that can question itself and correct errors. AI will never do that. So what you get instead is negligence because a computer makes a mathematical computation and the human element treats it as infallble.
- rexpop 7y ago
temp626 7y ago
Humans are not much better gatekeepers. Studies have shown that for example having certain name can be serious disadvantage when applying for work. Algorithms can be bad as well, but at least you can look into them to understand the reasons.
- bradknowles 7y ago
  Maybe. They can also be inscrutable. And that's just basic algorithms and regular human-written code. Then you've got ML systems which are a whole 'nother level of inscrutable.
  - carlmr 7y ago
    Or buggy. We're already running plenty of buggy code, but at least the errors can usually be found. If an AI overflows on some intermediate calculation sometimes because of implementation error I'm almost sure nobody will find that.
1290cc 7y ago
Its already happening to hundreds of business owners who sell their own products on Amazon. Anyone who had a successful FBA business is now diversifying away from Amazon as it seems almost impossible to resolve mistakes caused by their algos.
momokoko 7y ago
> AI systems mis-analyzing some data and locking people out from education, work, credit and health opportunities. College admissions? Resume screens? Mortgage applications? We already have that at scale. There wouldn't be so much money in gaming those things if the AI wasn't doing such a poor job. You likely just happen to be someone that isn't on the wrong side of it.
- Sniffnoy 7y ago
  > There wouldn't be so much money in gaming those things if the AI wasn't doing such a poor job. This isn't true at all. When the process works well, those who it excludes often attempt to spend money to subvert it. There's no reason that interfering with the process should only happen when it's working poorly. A simple example is the recent college admissions scandal. You often hear this scandal demonstrates that college admissions are just a way to let in rich people, but actually it demonstrates the opposite -- that the normal admissions process had (presumably correctly?) kept these people out and so they had to pay to subvert it. Of course, you're not claiming that these things are just ways of separating the rich from the rest; just that they're messed up in some way or another. Which, if not very specific, seems broadly accurate. But the following claim isn't -- people on the losing end of a judgment are going to do their best to subvert that judgment whether it's correct or not, so one cannot infer much of anything about the judgment's correctness based on whether there's then an attempt at subverting it.
- qrbLPHiKpiux 7y ago
  Call AI for what it is. Garbage in, garbage out. GIGO
RachelF 7y ago
This has already happened in Australia. The AI that runs the government equivalent of social security and the tax office, has been programmed to send debt notices to tens of thousands of people on welfare. The Aussies call it "Robodebt".
avshyz 7y ago
"The chains of tormented mankind are made out of red tape" - If Kafka were to live today, he'd write about AI bureaucracy.
nonprofiteer 7y ago
My go-to example of this scenario: https://splinternews.com/how-an-internet-mapping-glitch-turned-a-random-kansas-f-1793856052 https://splinternews.com/how-an-internet-mapping-glitch-turn...

snagglegaggle 7y ago

I'm not sure we can solve this as it relates to AI/ML if we can't even solve it as it exists in human society today. Various heuristics are used, knowingly or not, to gate access to employment, education, healthcare, etc. As an example I had to drop out of education (in the US) due to mental health issues. 6-7 years later, I'm still having problems getting back in as I wasn't able to get the problems documented in time. I also am unable to get any money as most of my classmates were despite my far above average performance pre-uni.

zxcb1 7y ago

Will these systems ever forgive or forget, like humans?

ssss11 7y ago

We need to stop that future don’t we?

Accujack 7y ago

>As we automate everything this is inevitable. Which is why it's important to ask the question "should we" instead of just "can we". Even more important is not dividing people into groups that we treat differently. Everyone should have the same education, work, credit and health opportunities. If we can free ourselves from the desire to segregate and separate people in the name of progress, whether or not an "AI" guesses wrong about you is irrelevant. It comes down to ego... people have to accept that no one can predict someone's potential or value to the world with enough certainty to make restricting people's opportunities worth it.

phs318u 7y ago

We’re already seeing the impacts of these kinds of systems affecting many of the most vulnerable in the Australian community. I’m referring to the Dept. of Human Services’ punitive “robodebt” system. https://mobile.abc.net.au/news/2017-03-03/centrelink-debt-controversy-what-is-robodebt/8317764 https://mobile.abc.net.au/news/2017-03-03/centrelink-debt-co...

falcor84 7y ago

But is this really about AI? Almost all aspects of our lives are already controlled by the enormously complex soulless machines of bureaucracy we call firms and governments. Is there any essential difference between a computer bug and a clerical error? If anything, I would assume the former is easier to fix.

OnlineGladiator 7y ago
I agree. I think the real question (which is much broader and beyond the scope of OP) is how does humanity have to operate differently with every order of magnitude it grows in size? Humanity seems to be transitioning from single digit billions to crossing the double digit billions barrier in the foreseeable future. Until 1900 humanity had been on the order of a billion people, and we are currently transitioning towards 10x as many people sustaining the planet. I do not mean this to be about finite resources and our planet. I am referring to the way that organizations require more layers of bureaucracy as they grow. A rule of thumb is for every 10x in people, you have another layer of management. So what will this extra layer of management be for people, not just from a government perspective but also a capitalistic corporation perspective? How do Google, Facebook, Apple, Stripe, Amazon, and all of those other companies handle so many customers? How much do they use automation, how many of their rules are archaic, how hard is it to challenge the system if you think there was an error? I am not convinced AI can solve many of the fundamental problems humanity are facing - and I think overpopulation might actually be the problem. Modern capitalism has conditioned us to believe "growth is good" but growth for the sake of growth is no different from cancer.
ljm 7y ago
But that’s not how it works, it’s not one in place of another; it’s one that depends on another. If your physical bureaucratic entity delegates decision making to an AI, you have no hope of redress unless due process was mandated. If you cut the bureaucratic middleman then you have to build an AI that can question itself and correct errors. AI will never do that. So what you get instead is negligence because a computer makes a mathematical computation and the human element treats it as infallble.
rexpop 7y ago

temp626 7y ago

Humans are not much better gatekeepers. Studies have shown that for example having certain name can be serious disadvantage when applying for work. Algorithms can be bad as well, but at least you can look into them to understand the reasons.

bradknowles 7y ago
Maybe. They can also be inscrutable. And that's just basic algorithms and regular human-written code. Then you've got ML systems which are a whole 'nother level of inscrutable.
- carlmr 7y ago
  Or buggy. We're already running plenty of buggy code, but at least the errors can usually be found. If an AI overflows on some intermediate calculation sometimes because of implementation error I'm almost sure nobody will find that.

1290cc 7y ago

Its already happening to hundreds of business owners who sell their own products on Amazon. Anyone who had a successful FBA business is now diversifying away from Amazon as it seems almost impossible to resolve mistakes caused by their algos.

momokoko 7y ago

> AI systems mis-analyzing some data and locking people out from education, work, credit and health opportunities. College admissions? Resume screens? Mortgage applications? We already have that at scale. There wouldn't be so much money in gaming those things if the AI wasn't doing such a poor job. You likely just happen to be someone that isn't on the wrong side of it.

Sniffnoy 7y ago
> There wouldn't be so much money in gaming those things if the AI wasn't doing such a poor job. This isn't true at all. When the process works well, those who it excludes often attempt to spend money to subvert it. There's no reason that interfering with the process should only happen when it's working poorly. A simple example is the recent college admissions scandal. You often hear this scandal demonstrates that college admissions are just a way to let in rich people, but actually it demonstrates the opposite -- that the normal admissions process had (presumably correctly?) kept these people out and so they had to pay to subvert it. Of course, you're not claiming that these things are just ways of separating the rich from the rest; just that they're messed up in some way or another. Which, if not very specific, seems broadly accurate. But the following claim isn't -- people on the losing end of a judgment are going to do their best to subvert that judgment whether it's correct or not, so one cannot infer much of anything about the judgment's correctness based on whether there's then an attempt at subverting it.
qrbLPHiKpiux 7y ago
Call AI for what it is. Garbage in, garbage out. GIGO

RachelF 7y ago

This has already happened in Australia. The AI that runs the government equivalent of social security and the tax office, has been programmed to send debt notices to tens of thousands of people on welfare. The Aussies call it "Robodebt".

avshyz 7y ago

"The chains of tormented mankind are made out of red tape" - If Kafka were to live today, he'd write about AI bureaucracy.

nonprofiteer 7y ago

My go-to example of this scenario: https://splinternews.com/how-an-internet-mapping-glitch-turned-a-random-kansas-f-1793856052 https://splinternews.com/how-an-internet-mapping-glitch-turn...