11 ms·

Be Careful What You Code For

53 points by miraj 10y ago

rndmize 10y ago
I thought this was going to be something interesting until I hit the "Environmental Consequences" part. I mean, really? We have environmental problems as a result on agriculture and manufacturing and transportation and construction and literally any other industry, and you're going to complain about the cost of running some servers? We are orders of magnitude away in impact. Actually, I suspect its even worse - the more computing power we make available to those industries, the more efficient they become. Please, tell me how making digital designs is environmentally worse than physical prototypes, or simulations are less power efficient than real-world tests, or data-driven irrigation is more wasteful than spraying water into the air for hours.
- PeCaN 10y ago
  This comment raises a good point.¹ Yes, lots of technology brings an environmental impact. Does the author of this article have any fucking clue how much paper and gas would be used for mail if Gmail didn't exist? ¹ Diatribe time: Why the fuck is rndmize's comment negative currently? Are people really that oblivious that they can't think for themselves and realize how flawed the arguments in the article are just because it lets them circlejerk² about political correctness? ² Or whatever the equivalent is for women, gotta be inclusive.
jordigh 10y ago
There's those funny/sad stories of how face recognition software has categorised black people as either invisible[1] or gorillas: http://www.businessinsider.com/google-tags-black-people-as-gorillas-2015-7 http://www.businessinsider.com/google-tags-black-people-as-g... http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html The prejudice of the programmer (unintentional, as all prejudice really is) can show up in the code just because they forgot to include enough data in their training set. --- [1] In light of which Ralph Ellison's novel is particularly prescient: https://en.wikipedia.org/wiki/Invisible_Man https://en.wikipedia.org/wiki/Invisible_Man
- hodwik 10y ago
  I ran into this recently on a video chat with my brother, he was trying to pin a mustache on his face but the algorithm couldn't see him. Fortunately, he didn't know why. I don't know how I would've reacted had he understood.
- prodigal_erik 10y ago
  If you forget I exist, you are not judging me in any way. If my training data consists of one face, I'm obviously incompetent at machine learning but not prejudiced against everyone else in the world.
beat 10y ago
The money quote: "I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system." This is extremely important. The weight of existing prejudice drives and reinforces future prejudice. Garbage in, garbage out. If you start with biased, flawed data, and you look for patterns like the biased, flawed data, you'll just add to the biases and flaws. You need to account for the poor data quality explicitly.
- ben_jones 10y ago
  I love how we work in an industry where it's ok to manipulate massive numbers of people because as engineers we know what's best for them. \s.
- TruthAndDare 10y ago
  I don't follow you. When do you start with biased data? Furthermore, wouldn't a advertisement system that somehow progresses to an increasingly erroneous view be broken anyway?
  - rm_-rf_slash 10y ago
    If historic and contemporary data assumes American blacks live in the city and whites live in the suburbs, data alone will not "know" the histories of race-based discrimination. So if it advertises apartments to blacks and houses to whites, it does not "understand" that it is propagating a stereotype. A human has to intervene.
    - TruthAndDare 10y ago
      The point of the algorithm is simply to choose advertisements as successfully as possible.
  - ubernostrum 10y ago
    Current distributions of property, wealth, access to services, etc. are all rooted in discriminatory laws and policies of the (not always distant) past. Which creates a risk that blindly pointing an algorithm at these data sets will result in an algorithm which assumes these are the expected/natural distributions and will optimize to perpetuate them, essentially becoming an algorithmic implementation of the prior discriminatory laws and policies. Thus, unless the goal is to perpetuate the effects of discrimination, algorithms need to be carefully set up and trained to understand that "this isn't how things are supposed to be".
niftich 10y ago
I don't care what your politics are. If you're building a data-driven system and you forget that empirically observed covariation is a necessary but not sufficient condition for causality, you're building flawed system, which is inaccurate at best, discriminatory at worst.
PeCaN 10y ago
I feel like this is a politically-correct rant thinly disguised as programming related. > I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system. Fuck that. Build _correct_ systems first. If you’re making a system that intentionally distorts data for the sake of “combating prejudice”, you’re _lying_. That doesn’t help anyone.
- jordigh 10y ago
  The programmer's prejudice may show up in code. There are funny/sad examples of blacks getting identified as gorillas or not being recognised at all by face recognition software... all because the person designing the code forgot to include enough black people in the training set: http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://www.businessinsider.com/google-tags-black-people-as-gorillas-2015-7 http://www.businessinsider.com/google-tags-black-people-as-g...
- MereInterest 10y ago
  What you call "intentionally distorting data", I would call "correcting for a confounding variable". Unrelated Example: Every morning, I weigh myself. Today, I found that my scale is inaccurate, and understates my weight by 5%. Therefore, if I am to predict my weight in the future, I must account for the inaccuracy of the scale. Related Example: There exist police statistics for arrests. Studies have found that arrests are racially biased, and therefore those police statistics do not accurately represent the crime rate, only the biased arrest rate. Therefore, if I am to predict the crime rate in the future, I must account for the inaccuracy of the data set.
  - PeCaN 10y ago
    I'm not going to get into a debate if you're going to appeal to the “actual crime rate” vs the “racially biased arrest rate”. There's not enough data to know, and too many interacting variables. It's enough for me to accept that blacks represent a disproportional amount of inmates and that black neighborhoods are frequently stuck in poverty cycles. I think the solution actually has nothing to do with correcting variables for race or anything of the sort; rather, help people—regardless of race—out of poverty and the statistics will level out. I'm extremely skeptical that arrest “bias” is skin color related. Also, citation needed for “studies have found that arrests are racially biased”. That is entirely too hard to measure. Well, it's not that I doubt such studies exist, I simply doubt they're valid in any way, and there are probably other studies with the opposite conclusion.
analognoise 10y ago
Ugh, more garbage from the technical bubble children who are perpetually an exposed nerve.
NetTechM 10y ago
It seems perfectly reasonable to station police in places which are indisputable hotspots for crime. I mean, is it discriminatory to say that Compton has a lot of crime and police should patrol it? Or is this just loosely using the tech community as a demographic to drum up clicks while scraping the bottom of the barrel as far as looking for topics to write about?
- rconti 10y ago
  Hotspots for crime? Or hotspots for arrests?
  - PeCaN 10y ago
    Why not both? What if all races commit roughly proportional amounts of crime, but there's a higher density of crime in black neighborhoods? There are way too many variables to say anything about whether “blacks commit more crime” OR “police arrest blacks too much”.
  - ktRolster 10y ago
    An external proxy for 'crime hotspots' could be looking whether an area has bars on the windows or not.
hueving 10y ago
"ignore the data and do what's good for the feels." If you are writing a system that is designed to predict arrests for any reason other than telling police officers where to go, then arrest records are precisely the data you want to use without these political correctness coefficients. So advertising legal help for criminal arrests to someone that matches arrest demographics is the correct thing to do. Anything else is stupidity.
- deleted 10y ago
  [deleted]
- CoryG89 10y ago
  This is a good point. If you are trying to design a system that "tells police officers where to go" then you shouldn't be using data to predict arrests, you should be predicting crime, which we don't have unbiased data for.

53 points by miraj 10y ago

I thought this was going to be something interesting until I hit the "Environmental Consequences" part. I mean, really? We have environmental problems as a result on agriculture and manufacturing and transportation and construction and literally any other industry, and you're going to complain about the cost of running some servers? We are orders of magnitude away in impact. Actually, I suspect its even worse - the more computing power we make available to those industries, the more efficient they become. Please, tell me how making digital designs is environmentally worse than physical prototypes, or simulations are less power efficient than real-world tests, or data-driven irrigation is more wasteful than spraying water into the air for hours.

PeCaN 10y ago
This comment raises a good point.¹ Yes, lots of technology brings an environmental impact. Does the author of this article have any fucking clue how much paper and gas would be used for mail if Gmail didn't exist? ¹ Diatribe time: Why the fuck is rndmize's comment negative currently? Are people really that oblivious that they can't think for themselves and realize how flawed the arguments in the article are just because it lets them circlejerk² about political correctness? ² Or whatever the equivalent is for women, gotta be inclusive.

jordigh 10y ago

There's those funny/sad stories of how face recognition software has categorised black people as either invisible[1] or gorillas: http://www.businessinsider.com/google-tags-black-people-as-gorillas-2015-7 http://www.businessinsider.com/google-tags-black-people-as-g... http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html The prejudice of the programmer (unintentional, as all prejudice really is) can show up in the code just because they forgot to include enough data in their training set. --- [1] In light of which Ralph Ellison's novel is particularly prescient: https://en.wikipedia.org/wiki/Invisible_Man https://en.wikipedia.org/wiki/Invisible_Man

hodwik 10y ago
I ran into this recently on a video chat with my brother, he was trying to pin a mustache on his face but the algorithm couldn't see him. Fortunately, he didn't know why. I don't know how I would've reacted had he understood.
prodigal_erik 10y ago
If you forget I exist, you are not judging me in any way. If my training data consists of one face, I'm obviously incompetent at machine learning but not prejudiced against everyone else in the world.

beat 10y ago

The money quote: "I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system." This is extremely important. The weight of existing prejudice drives and reinforces future prejudice. Garbage in, garbage out. If you start with biased, flawed data, and you look for patterns like the biased, flawed data, you'll just add to the biases and flaws. You need to account for the poor data quality explicitly.

ben_jones 10y ago
I love how we work in an industry where it's ok to manipulate massive numbers of people because as engineers we know what's best for them. \s.
TruthAndDare 10y ago
I don't follow you. When do you start with biased data? Furthermore, wouldn't a advertisement system that somehow progresses to an increasingly erroneous view be broken anyway?
- rm_-rf_slash 10y ago
  If historic and contemporary data assumes American blacks live in the city and whites live in the suburbs, data alone will not "know" the histories of race-based discrimination. So if it advertises apartments to blacks and houses to whites, it does not "understand" that it is propagating a stereotype. A human has to intervene.
  - TruthAndDare 10y ago
    The point of the algorithm is simply to choose advertisements as successfully as possible.
- ubernostrum 10y ago
  Current distributions of property, wealth, access to services, etc. are all rooted in discriminatory laws and policies of the (not always distant) past. Which creates a risk that blindly pointing an algorithm at these data sets will result in an algorithm which assumes these are the expected/natural distributions and will optimize to perpetuate them, essentially becoming an algorithmic implementation of the prior discriminatory laws and policies. Thus, unless the goal is to perpetuate the effects of discrimination, algorithms need to be carefully set up and trained to understand that "this isn't how things are supposed to be".

niftich 10y ago

I don't care what your politics are. If you're building a data-driven system and you forget that empirically observed covariation is a necessary but not sufficient condition for causality, you're building flawed system, which is inaccurate at best, discriminatory at worst.

PeCaN 10y ago

I feel like this is a politically-correct rant thinly disguised as programming related. > I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system. Fuck that. Build _correct_ systems first. If you’re making a system that intentionally distorts data for the sake of “combating prejudice”, you’re _lying_. That doesn’t help anyone.

jordigh 10y ago
The programmer's prejudice may show up in code. There are funny/sad examples of blacks getting identified as gorillas or not being recognised at all by face recognition software... all because the person designing the code forgot to include enough black people in the training set: http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://edition.cnn.com/2009/TECH/12/22/hp.webcams/index.html http://www.businessinsider.com/google-tags-black-people-as-gorillas-2015-7 http://www.businessinsider.com/google-tags-black-people-as-g...
MereInterest 10y ago
What you call "intentionally distorting data", I would call "correcting for a confounding variable". Unrelated Example: Every morning, I weigh myself. Today, I found that my scale is inaccurate, and understates my weight by 5%. Therefore, if I am to predict my weight in the future, I must account for the inaccuracy of the scale. Related Example: There exist police statistics for arrests. Studies have found that arrests are racially biased, and therefore those police statistics do not accurately represent the crime rate, only the biased arrest rate. Therefore, if I am to predict the crime rate in the future, I must account for the inaccuracy of the data set.
- PeCaN 10y ago
  I'm not going to get into a debate if you're going to appeal to the “actual crime rate” vs the “racially biased arrest rate”. There's not enough data to know, and too many interacting variables. It's enough for me to accept that blacks represent a disproportional amount of inmates and that black neighborhoods are frequently stuck in poverty cycles. I think the solution actually has nothing to do with correcting variables for race or anything of the sort; rather, help people—regardless of race—out of poverty and the statistics will level out. I'm extremely skeptical that arrest “bias” is skin color related. Also, citation needed for “studies have found that arrests are racially biased”. That is entirely too hard to measure. Well, it's not that I doubt such studies exist, I simply doubt they're valid in any way, and there are probably other studies with the opposite conclusion.

analognoise 10y ago

Ugh, more garbage from the technical bubble children who are perpetually an exposed nerve.

NetTechM 10y ago

It seems perfectly reasonable to station police in places which are indisputable hotspots for crime. I mean, is it discriminatory to say that Compton has a lot of crime and police should patrol it? Or is this just loosely using the tech community as a demographic to drum up clicks while scraping the bottom of the barrel as far as looking for topics to write about?

rconti 10y ago
Hotspots for crime? Or hotspots for arrests?
- PeCaN 10y ago
  Why not both? What if all races commit roughly proportional amounts of crime, but there's a higher density of crime in black neighborhoods? There are way too many variables to say anything about whether “blacks commit more crime” OR “police arrest blacks too much”.
- ktRolster 10y ago
  An external proxy for 'crime hotspots' could be looking whether an area has bars on the windows or not.

hueving 10y ago

"ignore the data and do what's good for the feels." If you are writing a system that is designed to predict arrests for any reason other than telling police officers where to go, then arrest records are precisely the data you want to use without these political correctness coefficients. So advertising legal help for criminal arrests to someone that matches arrest demographics is the correct thing to do. Anything else is stupidity.

deleted 10y ago
[deleted]
CoryG89 10y ago
This is a good point. If you are trying to design a system that "tells police officers where to go" then you shouldn't be using data to predict arrests, you should be predicting crime, which we don't have unbiased data for.