4 ms·

I just started reading Weapons of Math Destruction by mathematician Cathy O'Neil. She warns about big data systems that codify racism and classism from flawed d

I just started reading Weapons of Math Destruction by mathematician Cathy O'Neil. She warns about big data systems that codify racism and classism from flawed data and self-fulfilling feedback loops. The systems' "unbiased" decisions are opaque, proprietary, and often unchallengeable. https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

listenallyall 8y ago
If "big data" informs you that a group of people (race, geographic location, nationality, occupation, gender, etc) is less likely to say, pay back a loan, or successfully complete 4 years at university, or avoid insurance claims... must that be anything-ist? Or just a fact? Or, as your post suggests, are you obligated to throw out the conclusion and just assume the inputs must have been "flawed data"?
- mmt 8y ago
  I think, perhaps, you misunderstood the OP's objection, particularly with respect to feedback loops. It may not be safe to assume the inputs are biased [1], but, then, it's also not safe merely to assume that they aren't. What's particularly dangerous is to assume unbiased inputs if the subsequent conclusions (the outputs) then influences future inputs [2]. That's the feedback loop that can serve to amplify biases in the data/conclusions with each cycle. > must that be anything-ist? Or just a fact? To answer your question, no, it doesn't have to be anything-ist and is, of course, a fact. However, to characterize it as just a fact ignores that it may well be an anything-ist fact, if it had been based on anything-ist data. Ultimately, this is just a GIGO [3] problem, a concept from early in "computing" (aka "data processing"). [1] which I'm substituting for "flawed", since "unbiased" was the OP's term [2] for example, by setting interest rates which have an influence on loan repayment likelihood [3] https://en.wikipedia.org/wiki/Garbage_in,_garbage_out https://en.wikipedia.org/wiki/Garbage_in,_garbage_out
- throwawayjava 8y ago
  This topic has been discussed at length on HN and has been the topic of many flame wars. If you're genuinely curious for an answer, in the United States it's absolutely the case that you are legally > obligated to throw out the conclusion when it comes to many of the things you mentioned (e.g., you can't consider race, or often gender and sexual orientation depending on the state, when you evaluate a loan/employment application). The reason for those laws is that, historically, overtly and openly racist people used the loan application process to discriminate in housing. Asking to be able to use any accurate statistical model for any application is an inherently political request, because bad actors in the past have attempted to exclude people based upon skin color. We can't make that discrimination legal whenever someone can dream up a plausible mathematical model justifying an ultimately racial intent. The real point of my post is that a discussion on this topic devolves into a flame war unless everyone agrees that we can't turn p-hacking into a legally justifiable way of allowing racial discrimination in important social processes like housing loan or employment applications.
  - listenallyall 8y ago
    > We can't make that discrimination legal > an ultimately racial intent. I reject your assertion that by studying data, data scientists or analysts are somehow complicit in discrimination or any -ism. Their job is to make business decisions based on data, WITHOUT relying on "gut feel" or other human biases. I also reject the expectation that data scientists have an obligation (or the ability) to "fix" whatever biases may be revealed by the data. > p-hacking Again, you are making assumptions that data scientists are somehow evil, engaging in shady or illegal tactics to promote discrimination rather than simply doing their job in a straightforward manner. P-hacking is often done by individual researchers looking to get a study published in a scholarly journal (as opposed to months/years or research failing to reach a statistically-valid conclusion), rarely by companies who are looking to make profitable decisions based on what the data is indicating.
    - tanana 8y ago
      It appears that you make the false assumption that the data itself are unbiased and are always factually correct. This is untrue. Data does not appear magically in a dataset. It is the interpretation of the world by humans and may therefore carry the original bias, intended or not, of humans. This is why analysts have to think about what the data means whenever they do their analyses. I would say that is their ethical responsibility.
- chrischen 8y ago
  Big data may conclude that the data shows black people are more likely to commit crimes, but it doesn’t tell you the why, which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life. The AI, based on statistics can tell you the effects, but not the causes. So if i were to go and beat up every person named John, and then machine learning takes a crack at it and tells us that people named John are more likely to get injured, we may end up discriminating against Johns without realizing it was I who just have a thing against people named John. If this happens in a system and creates a feedback loop, It can lead to something becoming a self-fulfilling prophecy when it need not have been. Based on a naive conclusion the solution may be to deny health coverage to people named John, but obviously the real solution is to put me in jail.
  - ekianjo 8y ago
    Regardless of the data used, not knowing the why does not invalidate a strong correlation at all. If you stopped at doing thing only when you know the why there would no progress in science, because there are always new WHYs you uncover as you go.
    - tanana 8y ago
      That's why when studies that make discoveries of new correlations, they have to replicate such findingers in different study designs that actually allow you to say something about causation. In medicine, that would often mean a randomized study between treatment A and a placebo. However, with factors such as race, or sex this is obviously impossible.
      - ekianjo 8y ago
        Intead of looking at race or sex, looking at individual behaviors usually explains things better but it is harder to capture.
  - curiousgal 8y ago
    > which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life Also know as a cofounding effect. All of these problems have been encountered by statisticians decades ago.
gfarah 8y ago
I just finished building a credit risk ML classifier in the company I work for. The model will be used to define if we lend or not money to people/companies. Adding profiling features does give more accurate predictions. However, I pitched not using these features as a competitive advantage to the founders and they (luckily) agreed. We won’t be using them, and we ended up (with more work of course) getting a similar performant model without them. We can and should try to not use those kinds of features and be as fair as possible.
- natalyarostova 8y ago
  It's illegal to use them. Although it's strictly suboptimal for the firm.
- listenallyall 8y ago
  Please explain. If you removed "profiling features" from your analysis, but then through other means ended up with a "similar performant model," then aren't you still profiling, just with less obvious data inputs? If Group A is low credit risk and Group B is high credit risk, won't your new model still make it much harder for Group B to be approved? If not, it's not "similar." If you are able to dissect the Groups and pull individual high/low risks from within the groups, that would be a superior model, which is not what you claimed. So how are you not profiling, and how does it make a difference in terms of who gets approved/rejected?
  - gfarah 8y ago
    We focused on behavioral data. We realized that a solid plan for the the money and how is it lay out, does correlate with a low default rate. We think this is a good way to avoid (or at least minimize) those kinds of proxies you mention.
    - listenallyall 8y ago
      So, people with "a solid plan for the money" (and who is the wise sage that makes THAT decision?) is just your company's replacement for directly using profiling information. You really think you've changed anything?

listenallyall 8y ago

If "big data" informs you that a group of people (race, geographic location, nationality, occupation, gender, etc) is less likely to say, pay back a loan, or successfully complete 4 years at university, or avoid insurance claims... must that be anything-ist? Or just a fact? Or, as your post suggests, are you obligated to throw out the conclusion and just assume the inputs must have been "flawed data"?

mmt 8y ago
I think, perhaps, you misunderstood the OP's objection, particularly with respect to feedback loops. It may not be safe to assume the inputs are biased [1], but, then, it's also not safe merely to assume that they aren't. What's particularly dangerous is to assume unbiased inputs if the subsequent conclusions (the outputs) then influences future inputs [2]. That's the feedback loop that can serve to amplify biases in the data/conclusions with each cycle. > must that be anything-ist? Or just a fact? To answer your question, no, it doesn't have to be anything-ist and is, of course, a fact. However, to characterize it as just a fact ignores that it may well be an anything-ist fact, if it had been based on anything-ist data. Ultimately, this is just a GIGO [3] problem, a concept from early in "computing" (aka "data processing"). [1] which I'm substituting for "flawed", since "unbiased" was the OP's term [2] for example, by setting interest rates which have an influence on loan repayment likelihood [3] https://en.wikipedia.org/wiki/Garbage_in,_garbage_out https://en.wikipedia.org/wiki/Garbage_in,_garbage_out
throwawayjava 8y ago
This topic has been discussed at length on HN and has been the topic of many flame wars. If you're genuinely curious for an answer, in the United States it's absolutely the case that you are legally > obligated to throw out the conclusion when it comes to many of the things you mentioned (e.g., you can't consider race, or often gender and sexual orientation depending on the state, when you evaluate a loan/employment application). The reason for those laws is that, historically, overtly and openly racist people used the loan application process to discriminate in housing. Asking to be able to use any accurate statistical model for any application is an inherently political request, because bad actors in the past have attempted to exclude people based upon skin color. We can't make that discrimination legal whenever someone can dream up a plausible mathematical model justifying an ultimately racial intent. The real point of my post is that a discussion on this topic devolves into a flame war unless everyone agrees that we can't turn p-hacking into a legally justifiable way of allowing racial discrimination in important social processes like housing loan or employment applications.
- listenallyall 8y ago
  > We can't make that discrimination legal > an ultimately racial intent. I reject your assertion that by studying data, data scientists or analysts are somehow complicit in discrimination or any -ism. Their job is to make business decisions based on data, WITHOUT relying on "gut feel" or other human biases. I also reject the expectation that data scientists have an obligation (or the ability) to "fix" whatever biases may be revealed by the data. > p-hacking Again, you are making assumptions that data scientists are somehow evil, engaging in shady or illegal tactics to promote discrimination rather than simply doing their job in a straightforward manner. P-hacking is often done by individual researchers looking to get a study published in a scholarly journal (as opposed to months/years or research failing to reach a statistically-valid conclusion), rarely by companies who are looking to make profitable decisions based on what the data is indicating.
  - tanana 8y ago
    It appears that you make the false assumption that the data itself are unbiased and are always factually correct. This is untrue. Data does not appear magically in a dataset. It is the interpretation of the world by humans and may therefore carry the original bias, intended or not, of humans. This is why analysts have to think about what the data means whenever they do their analyses. I would say that is their ethical responsibility.
chrischen 8y ago
Big data may conclude that the data shows black people are more likely to commit crimes, but it doesn’t tell you the why, which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life. The AI, based on statistics can tell you the effects, but not the causes. So if i were to go and beat up every person named John, and then machine learning takes a crack at it and tells us that people named John are more likely to get injured, we may end up discriminating against Johns without realizing it was I who just have a thing against people named John. If this happens in a system and creates a feedback loop, It can lead to something becoming a self-fulfilling prophecy when it need not have been. Based on a naive conclusion the solution may be to deny health coverage to people named John, but obviously the real solution is to put me in jail.
- ekianjo 8y ago
  Regardless of the data used, not knowing the why does not invalidate a strong correlation at all. If you stopped at doing thing only when you know the why there would no progress in science, because there are always new WHYs you uncover as you go.
  - tanana 8y ago
    That's why when studies that make discoveries of new correlations, they have to replicate such findingers in different study designs that actually allow you to say something about causation. In medicine, that would often mean a randomized study between treatment A and a placebo. However, with factors such as race, or sex this is obviously impossible.
    - ekianjo 8y ago
      Intead of looking at race or sex, looking at individual behaviors usually explains things better but it is harder to capture.
- curiousgal 8y ago
  > which may be that black people are systematically opressed, more likely to come from a poorer background, discriminated upon so as to reduce their options in life Also know as a cofounding effect. All of these problems have been encountered by statisticians decades ago.

gfarah 8y ago

I just finished building a credit risk ML classifier in the company I work for. The model will be used to define if we lend or not money to people/companies. Adding profiling features does give more accurate predictions. However, I pitched not using these features as a competitive advantage to the founders and they (luckily) agreed. We won’t be using them, and we ended up (with more work of course) getting a similar performant model without them. We can and should try to not use those kinds of features and be as fair as possible.

natalyarostova 8y ago
It's illegal to use them. Although it's strictly suboptimal for the firm.
listenallyall 8y ago
Please explain. If you removed "profiling features" from your analysis, but then through other means ended up with a "similar performant model," then aren't you still profiling, just with less obvious data inputs? If Group A is low credit risk and Group B is high credit risk, won't your new model still make it much harder for Group B to be approved? If not, it's not "similar." If you are able to dissect the Groups and pull individual high/low risks from within the groups, that would be a superior model, which is not what you claimed. So how are you not profiling, and how does it make a difference in terms of who gets approved/rejected?
- gfarah 8y ago
  We focused on behavioral data. We realized that a solid plan for the the money and how is it lay out, does correlate with a low default rate. We think this is a good way to avoid (or at least minimize) those kinds of proxies you mention.
  - listenallyall 8y ago
    So, people with "a solid plan for the money" (and who is the wise sage that makes THAT decision?) is just your company's replacement for directly using profiling information. You really think you've changed anything?