4 ms·

I'm sure this is an incredibly unpopular opinion here, but I really believe that these things can be a net good. At least in some cases, like your example of cr

I'm sure this is an incredibly unpopular opinion here, but I really believe that these things can be a net good. At least in some cases, like your example of credit checking. I recall a startup nonprofit that gave loans to people in third world countries. The main problem was detecting fraud, and the only tool they had was checking social media profiles. Clearly this is a good thing if it maximizes the number of people who get loans. Even if a few good people get accidentally filtered, there is a net benefit. And yet this was really controversial in the HN comments. There is also a net benefit to having more accurate predictions on who will make a good tenant. Rent is the largest cost of living for the poor in many places, and anything that helps reduce that cost is a net benefit. Yet people react based on emotions. If a single false positive out of millions causes someone to be worse off, it's considered unfair and bad. Even if the net utility is positive and more people benefit than are hurt. It also rests on some underlying assumption that the existing system is perfectly fair, which is obviously false. Human's are way more biased at judging than algorithms. Bad loans and bad tenants have to be paid for by everyone else, which is hardly fair either. Social Credit has nothing to do with it. The goal of social credit is to discourage dissent. Whereas credit rating agencies don't care what your political opinions are, just how likely you are to pay your bills.

kristianc 10y ago
> There is also a net benefit to having more accurate predictions on who will make a good tenant. > Social Credit has nothing to do with it. The goal of social credit is to discourage dissent. That's the law of unintended consequences though. A well-intentioned system designed to determine who is and isn't a good tenant ends up having a chilling effect on online conversation as a whole because people are worried their prospective landlord, business partner, romantic partner won't want to deal with a Tory / Labour voter / UKIP voter / Trump supporter delete as appropriate. As Schneier points out in his book, it's the same technology used by Disneyland to match people's to their photos when they've been on Big Thunder Mountain as is used to pick out dissidents at political demos in Russia. Inherent in cost-benefit calculations is a tendency to overestimate short term individual benefits and underestimate long term social harms.
a3_nm 10y ago
You are suggesting that a background check system is doing a net good if more people benefit than are hurt. The problem is that a system like this would strongly encourage all people, good or bad, to avoid doing things that the system could object to. It's not enough to count the actual false positives, you have to take this loss of freedom into account as well. I don't think it's a good idea to have people self-censor because they worry how a surveillance algorithm will react to what they do. (The same way that you think twice, nowadays, about using problematic keywords or expressing a dangerously controversial opinion in a text message to a friend.)
- louthy 10y ago
  Exactly. This is exactly why systems like this pose such a threat. And what about the few that decide to say "Fuck the system!", well they become a social underclass - they can't get loans, mortgages, jobs, etc. It's a very slippery slope. Everyone should be entitled to a private life. And not everybody's private life would chime well with 'the establishment'. I could imagine myself falling foul of this because I produce music. It wouldn't take long on my facebook page to see that, so if a landlord could mine that information then I'd be unlikely to get a tenancy based on my potential to be a nuisance to my neighbours.
  - visarga 10y ago
    You gotta be a model FB citizen to rent or loan. FB's got it made, nobody can ignore them any more, even if they want to.
- visarga 10y ago
  Also, part of the problem is in the extraordinary scalability of these algorithms. You can apply them to 1000, 1 million or 1 billion people with little difference in cost. So the harm they do can be multiplied and forced upon the population at a large scale, with no human in the loop.
- Houshalter 10y ago
  That's a fair point, you've changed my mind. However what about if such a system was voluntary? Like you could opt-out of it, or opt into it. That's how car insurance is starting to work. You can get a thing put in your car that will monitor your driving, and in exchange you get lower rates. Perhaps you could opt into the social media scanner and get lower interest rates. If you don't, you just get grouped with all the other people who voluntarily opted out, and your rates will probably not be any higher than before. And in any case, landlords and employers already look at social media profiles. It's not like that is going to go away if you ban this, but an algorithm has a chance to be more fair and unbiased at it than a human.
blub 10y ago
I also remember a company that extended loans, but all ober the world. They were using analytics based on social media, browser fingerprinting, etc. They had some cases where they had no idea why the algorithm rejected some loan applications and then analysed the logs to find out why. It turned out that there was a correlation between specific system fonts and defaulting ob loans. Giving power over people to opaque algorithms and hustler start-ups is foolish.
- visarga 10y ago
  That's why algorithms that judge people should be held to public scrutiny. There should be validation tests to ensure the algorithm is fair. There should be error analysis, performed by third parties. Each judged human should be entitled to a feedback report to show how we were rated and why. It's not that I agree with this application, but it's inevitable that we will be judged by algorithms and we need to be sure they don't act in absurd ways. Public outcry should keep them in check.
  - Houshalter 10y ago
    The first problem is that the alternative is far less fair. Humans are hundreds of times more biased than algorithms. Judges give much harsher sentences after they have eaten lunch. Ugly people are discriminated against far more than attractive people. People of different political affiliations are discriminated against more than race is. And when asked to make predictions about candidates, we are only barely better than random (but think we are much better.) The second is that unfairness isn't a problem. Whatever has the most utility should be preferred. If one person is worse off and 99 are better off, it's a net good.
- Houshalter 10y ago
  This isn't an argument against it. So what if font size correlates with bad loans? Then it is a feature that should be used and will reduce the number of bad loans.
pjc50 10y ago
There is also a net benefit to having more accurate predictions on who will make a good tenant. Rent is the largest cost of living for the poor in many places, and anything that helps reduce that cost is a net benefit. Rent levels have very little to do with actual cost, it's a market-clearing price, only affected by the availability of alternatives. And what happens to the people who are banned by the system? They don't cease to exist. Rentiers used to overtly discriminate against people by ethnic group until this was banned, and undoubtedly still do to the extent that they can get away with it. If you algorithm disproportionately rejects members of ethnic minorities, is that even legal?
- antisthenes 10y ago
  > If you algorithm disproportionately rejects members of ethnic minorities, is that even legal? I don't know, but AirBnB might.
- Houshalter 10y ago
  >Rentiers used to overtly discriminate against people by ethnic group until this was banned, and undoubtedly still do to the extent that they can get away with it. But that's exactly my point. Humans are terrible and biased. The algorithm isn't racist, it just cares about providing the most accurate prediction possible. Leaving such decisions up to humans is horribly unfair and should be banned, not the algorithms. >And what happens to the people who are banned by the system? They don't cease to exist. Is that not already the case with credit ratings and applications and stuff? I'm only arguing that the existing system should be made more efficient. >Rent levels have very little to do with actual cost, it's a market-clearing price, only affected by the availability of alternatives. Rent level must be influenced by the number of bad tenants. Evicting people is difficult, and bad tenants can cause a lot of pain to the landlord. Which is why they care so much about using systems like this in the first place.

kristianc 10y ago

> There is also a net benefit to having more accurate predictions on who will make a good tenant. > Social Credit has nothing to do with it. The goal of social credit is to discourage dissent. That's the law of unintended consequences though. A well-intentioned system designed to determine who is and isn't a good tenant ends up having a chilling effect on online conversation as a whole because people are worried their prospective landlord, business partner, romantic partner won't want to deal with a Tory / Labour voter / UKIP voter / Trump supporter delete as appropriate. As Schneier points out in his book, it's the same technology used by Disneyland to match people's to their photos when they've been on Big Thunder Mountain as is used to pick out dissidents at political demos in Russia. Inherent in cost-benefit calculations is a tendency to overestimate short term individual benefits and underestimate long term social harms.

a3_nm 10y ago

You are suggesting that a background check system is doing a net good if more people benefit than are hurt. The problem is that a system like this would strongly encourage all people, good or bad, to avoid doing things that the system could object to. It's not enough to count the actual false positives, you have to take this loss of freedom into account as well. I don't think it's a good idea to have people self-censor because they worry how a surveillance algorithm will react to what they do. (The same way that you think twice, nowadays, about using problematic keywords or expressing a dangerously controversial opinion in a text message to a friend.)

louthy 10y ago
Exactly. This is exactly why systems like this pose such a threat. And what about the few that decide to say "Fuck the system!", well they become a social underclass - they can't get loans, mortgages, jobs, etc. It's a very slippery slope. Everyone should be entitled to a private life. And not everybody's private life would chime well with 'the establishment'. I could imagine myself falling foul of this because I produce music. It wouldn't take long on my facebook page to see that, so if a landlord could mine that information then I'd be unlikely to get a tenancy based on my potential to be a nuisance to my neighbours.
- visarga 10y ago
  You gotta be a model FB citizen to rent or loan. FB's got it made, nobody can ignore them any more, even if they want to.
visarga 10y ago
Also, part of the problem is in the extraordinary scalability of these algorithms. You can apply them to 1000, 1 million or 1 billion people with little difference in cost. So the harm they do can be multiplied and forced upon the population at a large scale, with no human in the loop.
Houshalter 10y ago
That's a fair point, you've changed my mind. However what about if such a system was voluntary? Like you could opt-out of it, or opt into it. That's how car insurance is starting to work. You can get a thing put in your car that will monitor your driving, and in exchange you get lower rates. Perhaps you could opt into the social media scanner and get lower interest rates. If you don't, you just get grouped with all the other people who voluntarily opted out, and your rates will probably not be any higher than before. And in any case, landlords and employers already look at social media profiles. It's not like that is going to go away if you ban this, but an algorithm has a chance to be more fair and unbiased at it than a human.

blub 10y ago

I also remember a company that extended loans, but all ober the world. They were using analytics based on social media, browser fingerprinting, etc. They had some cases where they had no idea why the algorithm rejected some loan applications and then analysed the logs to find out why. It turned out that there was a correlation between specific system fonts and defaulting ob loans. Giving power over people to opaque algorithms and hustler start-ups is foolish.

visarga 10y ago
That's why algorithms that judge people should be held to public scrutiny. There should be validation tests to ensure the algorithm is fair. There should be error analysis, performed by third parties. Each judged human should be entitled to a feedback report to show how we were rated and why. It's not that I agree with this application, but it's inevitable that we will be judged by algorithms and we need to be sure they don't act in absurd ways. Public outcry should keep them in check.
- Houshalter 10y ago
  The first problem is that the alternative is far less fair. Humans are hundreds of times more biased than algorithms. Judges give much harsher sentences after they have eaten lunch. Ugly people are discriminated against far more than attractive people. People of different political affiliations are discriminated against more than race is. And when asked to make predictions about candidates, we are only barely better than random (but think we are much better.) The second is that unfairness isn't a problem. Whatever has the most utility should be preferred. If one person is worse off and 99 are better off, it's a net good.
Houshalter 10y ago
This isn't an argument against it. So what if font size correlates with bad loans? Then it is a feature that should be used and will reduce the number of bad loans.

pjc50 10y ago

There is also a net benefit to having more accurate predictions on who will make a good tenant. Rent is the largest cost of living for the poor in many places, and anything that helps reduce that cost is a net benefit. Rent levels have very little to do with actual cost, it's a market-clearing price, only affected by the availability of alternatives. And what happens to the people who are banned by the system? They don't cease to exist. Rentiers used to overtly discriminate against people by ethnic group until this was banned, and undoubtedly still do to the extent that they can get away with it. If you algorithm disproportionately rejects members of ethnic minorities, is that even legal?

antisthenes 10y ago
> If you algorithm disproportionately rejects members of ethnic minorities, is that even legal? I don't know, but AirBnB might.
Houshalter 10y ago
>Rentiers used to overtly discriminate against people by ethnic group until this was banned, and undoubtedly still do to the extent that they can get away with it. But that's exactly my point. Humans are terrible and biased. The algorithm isn't racist, it just cares about providing the most accurate prediction possible. Leaving such decisions up to humans is horribly unfair and should be banned, not the algorithms. >And what happens to the people who are banned by the system? They don't cease to exist. Is that not already the case with credit ratings and applications and stuff? I'm only arguing that the existing system should be made more efficient. >Rent levels have very little to do with actual cost, it's a market-clearing price, only affected by the availability of alternatives. Rent level must be influenced by the number of bad tenants. Evicting people is difficult, and bad tenants can cause a lot of pain to the landlord. Which is why they care so much about using systems like this in the first place.