8 ms·

Replacing Judgment with Algorithms

79 points by henrik_w 11y ago

ikeboy 11y ago
>The first step is to make these algorithms public. Companies and governments both balk at this, fearing that people will deliberately try to game them, but the alternative is much worse. This needs a lot more than the one sentence given to justify. How do we know that an open source security algorithm will be better? The justification isn't powerful enough for the claim.
- imron 11y ago
  It's not necessarily better, but you can study it, understand it, criticise it and suggest improvements. All of which are difficult/impossible if the algorithm stays secret.
- bnegreve 11y ago
  It seems to me that the whole article is about proving this point. Relevant quotes: > The secrecy of the algorithms further pushes people toward conformity. If you are worried that the US government will classify you as a potential terrorist, you're less likely to friend Muslims on Facebook. > If you know that your Sesame Credit score is partly based on your not buying "subversive" products or being friends with dissidents, you're more likely to overcompensate by not buying anything but the most innocuous books or corresponding with the most boring people. > Uber is an example of how this works. Passengers rate drivers and drivers rate passengers; both risk getting booted out of the system if their rankings get too low. > Many have documented a chilling effect among American Muslims, with them avoiding certain discussion topics lest they be taken the wrong way.
  - ikeboy 11y ago
    Those are trying to establish a loss from having these algorithms. They don't show whether the benefits of open source outweigh the costs. That part only seems to be handwaved in the sentence I quoted.
    - bnegreve 11y ago
      I think you're being unfair. The author has given us many clear issues with closed source algorithm that would not exist with open source algorithms. Showing that the benefits outweigh the costs would require a model of the impact the (closed or open) algorithmic judgments on the society, and thus a model of the society itself. Such model would inevitably be subject to even more subjectivity and discussions. If you're not buying it, it's your choice, and you may even be right. But that doesn't make the whole argument unreceivable. Instead, you should mention the benefits of closed source algorithms for the society, they're not that obvious to me.
      - ikeboy 11y ago
        "the alternative is much worse." That's a definite statement, and shouldn't be made without such a model. If you only have handwavey models, you can't claim one is "much worse" than the others. The level of confidence isn't justified. I'm fine with a list of possible bad outcomes, but don't claim way A is much worse than way B without stronger evidence. >Instead, you should mention the benefits of closed source algorithms for the society, they're not that obvious to me. One is mentioned in the part I quoted: that they're harder to circumvent. This isn't the same as regular security software where they theoretically are secure, but practically have holes, and so there's gains by opening it up and letting people report holes. There would be no theoretical security model here, it would only be probabilistic. There may not be a "patch" for the kind of holes (I'll give an example soon). So security by obscurity might make more sense here than in general. Example: suppose our data shows that people that drive car X have a greater security risk. If this is kept secret, we are able to more effectively target people (probably, many different factors will combine until someone is an extreme risk, but this is simplified.) The fact that X is weak evidence of risk is information that becomes useless relatively soon after it is publicized (because potential terrorists just put "don't buy X" on their checklists.) So publicizing the algorithm would concretely lead to worse results. Realistically, car model probably isn't too predictive, but possibly the union of all behaviors that are easy to change when known predicts risk reasonably well.
Houshalter 11y ago
This article is mainly about algorithms used by the government for surveillance and control. Bad stuff, of course. But in general, I think algorithms are something to embrace rather than fear: Even very simple algorithms reliably outperform experts by a large margin, in certain domains: http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/ http://lesswrong.com/lw/3gv/statistical_prediction_rules_out... Despite this, humans are heavily biased against algorithms. Even when we know they perform much better than humans. We also tend to vastly overestimate human performance: http://lesswrong.com/lw/lsc/link_algorithm_aversion/ http://lesswrong.com/lw/lsc/link_algorithm_aversion/ This has a lot of consequences. Humans are terribly biased, and often aren't even aware of our biases. Not just stuff like race, but even people that are ugly are heavily discriminated against: http://lesswrong.com/lw/lj/the_halo_effect/ http://lesswrong.com/lw/lj/the_halo_effect/ We discriminate against people of different political parties even more than we do based on race: http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/ http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything... Or just random factors we can't predict. Nothing about being judged by a human is "fair". And beyond fairness, they just give better predictions. Humans just aren't very good at prediction, even when we think we are. And in some domains that's extremely important, like in medicine. The first link referenced several places where statistical methods did better at diagnosis than doctors who specialized in it. Yet there has always been massive resistance to using algorithms in diagnosis.
- rando289 11y ago
  > I think algorithms are something to embrace rather than fear He has the same sentiment in his article.
jraines 11y ago
Replacing judgment with algorithms has always been part of bureaucracy. Once encoded as programs, though, they have a chance -- however unlikely, and still not without flaws -- of being audited either by the public or at least by independent auditors.
- pavel_lishin 11y ago
  Why do you supposed that programs written in a programming language will be audited more effectively than programs written down on paper?
Nutmog 11y ago
A large part of our lives is already devoted to gaming the human judgement system. The clothes we wear, the way we speak, the way we walk. Who we associate with - "Associate with deadbeats, and you're more likely to be judged as one." - has always happened in real life too. Women wear makeup to work to game the judgement system and men try to show off their confidence. All just to try to influence how other people see them. So adding computers isn't really a big deal. It's already a major part of our daily lives and efforts.
logn 11y ago
I think we can probably get transparency and ethics built into the algorithms. But I think we'll have a much harder time controlling how pervasive and intrusive the algorithms are. And as a result, our behavior will be more effectively controlled by corporations and governments.
secondtimeuse 11y ago
Paraphrasing Sussman-Minsky story [1], just because you rely on judgement does not means there is no algorithm, it's just that you don't know what the algorithm is. In a country like China with rampant corruption, Algorithms are a welcome relief. Schneier having never faced any kind of corruption at individual level completely ignores this giant problem. While the exact details of the algorithm might be debatable, yet having an algorithm as opposed to corrupt bureaucrats making arbitrary decisions is a welcome step. [1] "So Sussman began working on a program. Not long after, this odd-looking bald guy came over. Sussman figured the guy was going to boot him out, but instead the man sat down, asking, "Hey, what are you doing?" Sussman talked over his program with the man, Marvin Minsky. At one point in the discussion, Sussman told Minsky that he was using a certain randomizing technique in his program because he didn't want the machine to have any preconceived notions. Minsky said, "Well, it has them, it's just that you don't know what they are." It was the most profound thing Gerry Sussman had ever heard. And Minsky continued, telling him that the world is built a certain way, and the most important thing we can do with the world is avoid randomness, and figure out ways by which things can be planned. Wisdom like this has its effect on seventeen-year-old freshmen, and from then on Sussman was hooked."
- adrianN 11y ago
  Regarding the Sussman story, it made it into a Hacker Koan https://en.wikipedia.org/wiki/Hacker_koan#Uncarved_block https://en.wikipedia.org/wiki/Hacker_koan#Uncarved_block
yummyfajitas 11y ago
Schneier makes a startling claim: It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. This concept needs to be expanded. We as a society need to understand what we expect out of the algorithms that automatically judge us and ensure that those expectations are met. Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above. If we want to mandate the outcome of our decision process, let's just do that openly and honestly. E.g. for credit, just impose a quota of credit that must be issued to each protected class. If we want fair and unbiased decisions then we need to instead examine the internals of our process and accept the outcomes, whatever they may be. But we need to stop pretending we can do both.
- Nutmog 11y ago
  This is going to get really hairy. Who gets to decide the protected classes? The classic ones grouped by race, religion, gender, etc will be too crude when computers are judging. Imagine some weird class like people who are old but don't wear prescription glasses - perhaps indicating that they're too irresponsible with money to be able to pay for them. Nevermind that they're an exception whose eyesight is OK if that data isn't visible to the computer. There could be all kinds of unexpected special cases like this that lead to individuals getting generally bad treatment throughout life.
  - barry-cotter 11y ago
    Protected classes aren't about justice, they're about identity politics. If a group can deliver votes or violence as a bloc and demand group representation then they'll get it. But invisible groups like left-handers or citizens who are not native born have immense difficulty coordinating and mostly don't.
- masterzora 11y ago
  > Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. That's an interesting interpretation. To me it looks more like he's discussing that bias can (probably will) be baked into algorithms even unintentionally and often subtly. And that highlights a significant difficulty with the "unbiased predictors" thing you say: can you distinguish between an unbiased algorithm discovering that something that looks like bias isn't and bias being subtly baked into the algorithm? I think that's where the "we need to understand what we expect out of the algorithms and ensure the expectations are met" bit comes in, at least in part.
zdw 11y ago
Dupe: https://news.ycombinator.com/item?id=10864933 https://news.ycombinator.com/item?id=10864933
- throwaway2048 11y ago
  calling attention to a story without upvotes or comments isn't very useful.
rubberstamp 11y ago
If any such algorithms are going to be built, it better be open source or be able to explain to me how it arrived on that judgment. I am not going to trust some blackbox on this.
hiddencost 11y ago
There's already very active mathematical work in this area. See fatml.org . I appreciate jeremy kun calling this out in Schneier'so comments.

79 points by henrik_w 11y ago

>The first step is to make these algorithms public. Companies and governments both balk at this, fearing that people will deliberately try to game them, but the alternative is much worse. This needs a lot more than the one sentence given to justify. How do we know that an open source security algorithm will be better? The justification isn't powerful enough for the claim.

imron 11y ago
It's not necessarily better, but you can study it, understand it, criticise it and suggest improvements. All of which are difficult/impossible if the algorithm stays secret.
bnegreve 11y ago
It seems to me that the whole article is about proving this point. Relevant quotes: > The secrecy of the algorithms further pushes people toward conformity. If you are worried that the US government will classify you as a potential terrorist, you're less likely to friend Muslims on Facebook. > If you know that your Sesame Credit score is partly based on your not buying "subversive" products or being friends with dissidents, you're more likely to overcompensate by not buying anything but the most innocuous books or corresponding with the most boring people. > Uber is an example of how this works. Passengers rate drivers and drivers rate passengers; both risk getting booted out of the system if their rankings get too low. > Many have documented a chilling effect among American Muslims, with them avoiding certain discussion topics lest they be taken the wrong way.
- ikeboy 11y ago
  Those are trying to establish a loss from having these algorithms. They don't show whether the benefits of open source outweigh the costs. That part only seems to be handwaved in the sentence I quoted.
  - bnegreve 11y ago
    I think you're being unfair. The author has given us many clear issues with closed source algorithm that would not exist with open source algorithms. Showing that the benefits outweigh the costs would require a model of the impact the (closed or open) algorithmic judgments on the society, and thus a model of the society itself. Such model would inevitably be subject to even more subjectivity and discussions. If you're not buying it, it's your choice, and you may even be right. But that doesn't make the whole argument unreceivable. Instead, you should mention the benefits of closed source algorithms for the society, they're not that obvious to me.
    - ikeboy 11y ago
      "the alternative is much worse." That's a definite statement, and shouldn't be made without such a model. If you only have handwavey models, you can't claim one is "much worse" than the others. The level of confidence isn't justified. I'm fine with a list of possible bad outcomes, but don't claim way A is much worse than way B without stronger evidence. >Instead, you should mention the benefits of closed source algorithms for the society, they're not that obvious to me. One is mentioned in the part I quoted: that they're harder to circumvent. This isn't the same as regular security software where they theoretically are secure, but practically have holes, and so there's gains by opening it up and letting people report holes. There would be no theoretical security model here, it would only be probabilistic. There may not be a "patch" for the kind of holes (I'll give an example soon). So security by obscurity might make more sense here than in general. Example: suppose our data shows that people that drive car X have a greater security risk. If this is kept secret, we are able to more effectively target people (probably, many different factors will combine until someone is an extreme risk, but this is simplified.) The fact that X is weak evidence of risk is information that becomes useless relatively soon after it is publicized (because potential terrorists just put "don't buy X" on their checklists.) So publicizing the algorithm would concretely lead to worse results. Realistically, car model probably isn't too predictive, but possibly the union of all behaviors that are easy to change when known predicts risk reasonably well.

Houshalter 11y ago

This article is mainly about algorithms used by the government for surveillance and control. Bad stuff, of course. But in general, I think algorithms are something to embrace rather than fear: Even very simple algorithms reliably outperform experts by a large margin, in certain domains: http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/ http://lesswrong.com/lw/3gv/statistical_prediction_rules_out... Despite this, humans are heavily biased against algorithms. Even when we know they perform much better than humans. We also tend to vastly overestimate human performance: http://lesswrong.com/lw/lsc/link_algorithm_aversion/ http://lesswrong.com/lw/lsc/link_algorithm_aversion/ This has a lot of consequences. Humans are terribly biased, and often aren't even aware of our biases. Not just stuff like race, but even people that are ugly are heavily discriminated against: http://lesswrong.com/lw/lj/the_halo_effect/ http://lesswrong.com/lw/lj/the_halo_effect/ We discriminate against people of different political parties even more than we do based on race: http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/ http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything... Or just random factors we can't predict. Nothing about being judged by a human is "fair". And beyond fairness, they just give better predictions. Humans just aren't very good at prediction, even when we think we are. And in some domains that's extremely important, like in medicine. The first link referenced several places where statistical methods did better at diagnosis than doctors who specialized in it. Yet there has always been massive resistance to using algorithms in diagnosis.

rando289 11y ago
> I think algorithms are something to embrace rather than fear He has the same sentiment in his article.

jraines 11y ago

Replacing judgment with algorithms has always been part of bureaucracy. Once encoded as programs, though, they have a chance -- however unlikely, and still not without flaws -- of being audited either by the public or at least by independent auditors.

pavel_lishin 11y ago
Why do you supposed that programs written in a programming language will be audited more effectively than programs written down on paper?

Nutmog 11y ago

A large part of our lives is already devoted to gaming the human judgement system. The clothes we wear, the way we speak, the way we walk. Who we associate with - "Associate with deadbeats, and you're more likely to be judged as one." - has always happened in real life too. Women wear makeup to work to game the judgement system and men try to show off their confidence. All just to try to influence how other people see them. So adding computers isn't really a big deal. It's already a major part of our daily lives and efforts.

logn 11y ago

I think we can probably get transparency and ethics built into the algorithms. But I think we'll have a much harder time controlling how pervasive and intrusive the algorithms are. And as a result, our behavior will be more effectively controlled by corporations and governments.

secondtimeuse 11y ago

Paraphrasing Sussman-Minsky story [1], just because you rely on judgement does not means there is no algorithm, it's just that you don't know what the algorithm is. In a country like China with rampant corruption, Algorithms are a welcome relief. Schneier having never faced any kind of corruption at individual level completely ignores this giant problem. While the exact details of the algorithm might be debatable, yet having an algorithm as opposed to corrupt bureaucrats making arbitrary decisions is a welcome step. [1] "So Sussman began working on a program. Not long after, this odd-looking bald guy came over. Sussman figured the guy was going to boot him out, but instead the man sat down, asking, "Hey, what are you doing?" Sussman talked over his program with the man, Marvin Minsky. At one point in the discussion, Sussman told Minsky that he was using a certain randomizing technique in his program because he didn't want the machine to have any preconceived notions. Minsky said, "Well, it has them, it's just that you don't know what they are." It was the most profound thing Gerry Sussman had ever heard. And Minsky continued, telling him that the world is built a certain way, and the most important thing we can do with the world is avoid randomness, and figure out ways by which things can be planned. Wisdom like this has its effect on seventeen-year-old freshmen, and from then on Sussman was hooked."

adrianN 11y ago
Regarding the Sussman story, it made it into a Hacker Koan https://en.wikipedia.org/wiki/Hacker_koan#Uncarved_block https://en.wikipedia.org/wiki/Hacker_koan#Uncarved_block

yummyfajitas 11y ago

Schneier makes a startling claim: It's already illegal for these algorithms to have discriminatory outcomes, even if they're not deliberately designed in. This concept needs to be expanded. We as a society need to understand what we expect out of the algorithms that automatically judge us and ensure that those expectations are met. Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. But we don't like to admit this, so we instead make silly statements like the above. If we want to mandate the outcome of our decision process, let's just do that openly and honestly. E.g. for credit, just impose a quota of credit that must be issued to each protected class. If we want fair and unbiased decisions then we need to instead examine the internals of our process and accept the outcomes, whatever they may be. But we need to stop pretending we can do both.

Nutmog 11y ago
This is going to get really hairy. Who gets to decide the protected classes? The classic ones grouped by race, religion, gender, etc will be too crude when computers are judging. Imagine some weird class like people who are old but don't wear prescription glasses - perhaps indicating that they're too irresponsible with money to be able to pay for them. Nevermind that they're an exception whose eyesight is OK if that data isn't visible to the computer. There could be all kinds of unexpected special cases like this that lead to individuals getting generally bad treatment throughout life.
- barry-cotter 11y ago
  Protected classes aren't about justice, they're about identity politics. If a group can deliver votes or violence as a bloc and demand group representation then they'll get it. But invisible groups like left-handers or citizens who are not native born have immense difficulty coordinating and mostly don't.
masterzora 11y ago
> Schneier is discussing an unpleasant fact; unbiased algorithms often discover that things we previously attributed to bias were actually unbiased predictors. That's an interesting interpretation. To me it looks more like he's discussing that bias can (probably will) be baked into algorithms even unintentionally and often subtly. And that highlights a significant difficulty with the "unbiased predictors" thing you say: can you distinguish between an unbiased algorithm discovering that something that looks like bias isn't and bias being subtly baked into the algorithm? I think that's where the "we need to understand what we expect out of the algorithms and ensure the expectations are met" bit comes in, at least in part.

zdw 11y ago

Dupe: https://news.ycombinator.com/item?id=10864933 https://news.ycombinator.com/item?id=10864933

throwaway2048 11y ago
calling attention to a story without upvotes or comments isn't very useful.

rubberstamp 11y ago

If any such algorithms are going to be built, it better be open source or be able to explain to me how it arrived on that judgment. I am not going to trust some blackbox on this.

hiddencost 11y ago

There's already very active mathematical work in this area. See fatml.org . I appreciate jeremy kun calling this out in Schneier'so comments.