4 ms·

If "big data" informs you that a group of people (race, geographic location, nationality, occupation, gender, etc) is less likely to say, pay back a loan, or su

If "big data" informs you that a group of people (race, geographic location, nationality, occupation, gender, etc) is less likely to say, pay back a loan, or successfully complete 4 years at university, or avoid insurance claims... must that be anything-ist? Or just a fact? Or, as your post suggests, are you obligated to throw out the conclusion and just assume the inputs must have been "flawed data"?

mmt 8y ago
I think, perhaps, you misunderstood the OP's objection, particularly with respect to feedback loops. It may not be safe to assume the inputs are biased [1], but, then, it's also not safe merely to assume that they aren't. What's particularly dangerous is to assume unbiased inputs if the subsequent conclusions (the outputs) then influences future inputs [2]. That's the feedback loop that can serve to amplify biases in the data/conclusions with each cycle. > must that be anything-ist? Or just a fact? To answer your question, no, it doesn't have to be anything-ist and is, of course, a fact. However, to characterize it as just a fact ignores that it may well be an anything-ist fact, if it had been based on anything-ist data. Ultimately, this is just a GIGO [3] problem, a concept from early in "computing" (aka "data processing"). [1] which I'm substituting for "flawed", since "unbiased" was the OP's term [2] for example, by setting interest rates which have an influence on loan repayment likelihood [3] https://en.wikipedia.org/wiki/Garbage_in,_garbage_out https://en.wikipedia.org/wiki/Garbage_in,_garbage_out
throwawayjava 8y ago
This topic has been discussed at length on HN and has been the topic of many flame wars. If you're genuinely curious for an answer, in the United States it's absolutely the case that you are legally > obligated to throw out the conclusion when it comes to many of the things you mentioned (e.g., you can't consider race, or often gender and sexual orientation depending on the state, when you evaluate a loan/employment application). The reason for those laws is that, historically, overtly and openly racist people used the loan application process to discriminate in housing. Asking to be able to use any accurate statistical model for any application is an inherently political request, because bad actors in the past have attempted to exclude people based upon skin color. We can't make that discrimination legal whenever someone can dream up a plausible mathematical model justifying an ultimately racial intent. The real point of my post is that a discussion on this topic devolves into a flame war unless everyone agrees that we can't turn p-hacking into a legally justifiable way of allowing racial discrimination in important social processes like housing loan or employment applications.
- listenallyall 8y ago
  > We can't make that discrimination legal > an ultimately racial intent. I reject your assertion that by studying data, data scientists or analysts are somehow complicit in discrimination or any -ism. Their job is to make business decisions based on data, WITHOUT relying on "gut feel" or other human biases. I also reject the expectation that data scientists have an obligation (or the ability) to "fix" whatever biases may be revealed by the data. > p-hacking Again, you are making assumptions that data scientists are somehow evil, engaging in shady or illegal tactics to promote discrimination rather than simply doing their job in a straightforward manner. P-hacking is often done by individual researchers looking to get a study published in a scholarly journal (as opposed to months/years or research failing to reach a statistically-valid conclusion), rarely by companies who are looking to make profitable decisions based on what the data is indicating.
  - tanana 8y ago
    It appears that you make the false assumption that the data itself are unbiased and are always factually correct. This is untrue. Data does not appear magically in a dataset. It is the interpretation of the world by humans and may therefore carry the original bias, intended or not, of humans. This is why analysts have to think about what the data means whenever they do their analyses. I would say that is their ethical responsibility.
    - listenallyall 8y ago
      > think about what the data means of course they do that, that is literally their job. But to reverse-engineer the biases that may or may not exist in an original data set -- please explain how this should be accomplished, because I don't see how someone could accurately quantify the amount or degree of race/sex/age/religion/nationality-ism without introducing additional "bias" based on that person's own opinion. > Data does not appear magically in a dataset. Right, so why isn't the boss, or exec, or department head, or 3rd-party, who sourced the data responsible for de-biasing the data before even handing it off to the the data scientist, so s/he can just do the job of data science-ing, and not political science-ing? You're putting a whole lot of "ethical responsibility" on just one person -- ironically, the one least likely to be good at interpreting human emotional tendencies -- within a much larger ecosystem.
chrischen 8y ago
Big data may conclude that the data shows black people are more likely to commit crimes, but it doesn’t tell you the why, which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life. The AI, based on statistics can tell you the effects, but not the causes. So if i were to go and beat up every person named John, and then machine learning takes a crack at it and tells us that people named John are more likely to get injured, we may end up discriminating against Johns without realizing it was I who just have a thing against people named John. If this happens in a system and creates a feedback loop, It can lead to something becoming a self-fulfilling prophecy when it need not have been. Based on a naive conclusion the solution may be to deny health coverage to people named John, but obviously the real solution is to put me in jail.
- ekianjo 8y ago
  Regardless of the data used, not knowing the why does not invalidate a strong correlation at all. If you stopped at doing thing only when you know the why there would no progress in science, because there are always new WHYs you uncover as you go.
  - tanana 8y ago
    That's why when studies that make discoveries of new correlations, they have to replicate such findingers in different study designs that actually allow you to say something about causation. In medicine, that would often mean a randomized study between treatment A and a placebo. However, with factors such as race, or sex this is obviously impossible.
    - ekianjo 8y ago
      Intead of looking at race or sex, looking at individual behaviors usually explains things better but it is harder to capture.
- curiousgal 8y ago
  > which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life Also know as a cofounding effect. All of these problems have been encountered by statisticians decades ago.

mmt 8y ago

I think, perhaps, you misunderstood the OP's objection, particularly with respect to feedback loops. It may not be safe to assume the inputs are biased [1], but, then, it's also not safe merely to assume that they aren't. What's particularly dangerous is to assume unbiased inputs if the subsequent conclusions (the outputs) then influences future inputs [2]. That's the feedback loop that can serve to amplify biases in the data/conclusions with each cycle. > must that be anything-ist? Or just a fact? To answer your question, no, it doesn't have to be anything-ist and is, of course, a fact. However, to characterize it as just a fact ignores that it may well be an anything-ist fact, if it had been based on anything-ist data. Ultimately, this is just a GIGO [3] problem, a concept from early in "computing" (aka "data processing"). [1] which I'm substituting for "flawed", since "unbiased" was the OP's term [2] for example, by setting interest rates which have an influence on loan repayment likelihood [3] https://en.wikipedia.org/wiki/Garbage_in,_garbage_out https://en.wikipedia.org/wiki/Garbage_in,_garbage_out

throwawayjava 8y ago

This topic has been discussed at length on HN and has been the topic of many flame wars. If you're genuinely curious for an answer, in the United States it's absolutely the case that you are legally > obligated to throw out the conclusion when it comes to many of the things you mentioned (e.g., you can't consider race, or often gender and sexual orientation depending on the state, when you evaluate a loan/employment application). The reason for those laws is that, historically, overtly and openly racist people used the loan application process to discriminate in housing. Asking to be able to use any accurate statistical model for any application is an inherently political request, because bad actors in the past have attempted to exclude people based upon skin color. We can't make that discrimination legal whenever someone can dream up a plausible mathematical model justifying an ultimately racial intent. The real point of my post is that a discussion on this topic devolves into a flame war unless everyone agrees that we can't turn p-hacking into a legally justifiable way of allowing racial discrimination in important social processes like housing loan or employment applications.

listenallyall 8y ago
> We can't make that discrimination legal > an ultimately racial intent. I reject your assertion that by studying data, data scientists or analysts are somehow complicit in discrimination or any -ism. Their job is to make business decisions based on data, WITHOUT relying on "gut feel" or other human biases. I also reject the expectation that data scientists have an obligation (or the ability) to "fix" whatever biases may be revealed by the data. > p-hacking Again, you are making assumptions that data scientists are somehow evil, engaging in shady or illegal tactics to promote discrimination rather than simply doing their job in a straightforward manner. P-hacking is often done by individual researchers looking to get a study published in a scholarly journal (as opposed to months/years or research failing to reach a statistically-valid conclusion), rarely by companies who are looking to make profitable decisions based on what the data is indicating.
- tanana 8y ago
  It appears that you make the false assumption that the data itself are unbiased and are always factually correct. This is untrue. Data does not appear magically in a dataset. It is the interpretation of the world by humans and may therefore carry the original bias, intended or not, of humans. This is why analysts have to think about what the data means whenever they do their analyses. I would say that is their ethical responsibility.
  - listenallyall 8y ago
    > think about what the data means of course they do that, that is literally their job. But to reverse-engineer the biases that may or may not exist in an original data set -- please explain how this should be accomplished, because I don't see how someone could accurately quantify the amount or degree of race/sex/age/religion/nationality-ism without introducing additional "bias" based on that person's own opinion. > Data does not appear magically in a dataset. Right, so why isn't the boss, or exec, or department head, or 3rd-party, who sourced the data responsible for de-biasing the data before even handing it off to the the data scientist, so s/he can just do the job of data science-ing, and not political science-ing? You're putting a whole lot of "ethical responsibility" on just one person -- ironically, the one least likely to be good at interpreting human emotional tendencies -- within a much larger ecosystem.

chrischen 8y ago

Big data may conclude that the data shows black people are more likely to commit crimes, but it doesn’t tell you the why, which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life. The AI, based on statistics can tell you the effects, but not the causes. So if i were to go and beat up every person named John, and then machine learning takes a crack at it and tells us that people named John are more likely to get injured, we may end up discriminating against Johns without realizing it was I who just have a thing against people named John. If this happens in a system and creates a feedback loop, It can lead to something becoming a self-fulfilling prophecy when it need not have been. Based on a naive conclusion the solution may be to deny health coverage to people named John, but obviously the real solution is to put me in jail.

ekianjo 8y ago
Regardless of the data used, not knowing the why does not invalidate a strong correlation at all. If you stopped at doing thing only when you know the why there would no progress in science, because there are always new WHYs you uncover as you go.
- tanana 8y ago
  That's why when studies that make discoveries of new correlations, they have to replicate such findingers in different study designs that actually allow you to say something about causation. In medicine, that would often mean a randomized study between treatment A and a placebo. However, with factors such as race, or sex this is obviously impossible.
  - ekianjo 8y ago
    Intead of looking at race or sex, looking at individual behaviors usually explains things better but it is harder to capture.
curiousgal 8y ago
> which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life Also know as a cofounding effect. All of these problems have been encountered by statisticians decades ago.