51 ms·

Its obvious why this is happening; some form of statistical modeling or AI correlated having a hotmail account with being higher risk. This is a serious proble

by 013a 9y ago

Its obvious why this is happening; some form of statistical modeling or AI correlated having a hotmail account with being higher risk. This is a serious problem with all AI that makes decisions like this. Having a Hotmail account is a symptom of a problem which could lead to higher risk, but it is judging that symptom as if it is a cause which leads to risk. Its exactly similar to the supposition that black people statistically commit more crime, and thus should pre-emptively receive harsher bail or be profiled. Unless you can scientifically prove that ethnicity causes crime to happen, its disgusting. This is obvious to us. In year's past, correlation fallacies might have meant that black people were profiled by police more. Today, it means that we build AIs which make life-ending decisions like determining repeat offender risk, visa status, employment decisions, all of which predetermines an unknown outcome by correlating your known qualities with the known outcome of people with similar qualities to you. We need regulation in place to stop any punitive decision making, public and private, which can be found in court to be based on correlation instead of causation.

ra 9y ago
that's sort of how insurance pricing works though. An actuarial process groups you with others based on similar attributes, and works out an overall risk value for you based on the combined risk profiles of the "groups" to which you belong. Historically, insurance companies only had things like postcode, gender, age, accident history to go off... but they're always interested in other data sets that could be used to modify pricing (or select less risky customers). Clearly someone or something has identified email domain as a factor (presumably based on data).
- tialaramex 9y ago
  They don't want "less risky customers" in car insurance and similar markets they want: More customers (full stop, if you have customers then your competitors don't have them, so you grow and they fail) Customers who pay more money Correctly assessing risk is not irrelevant but it's a small factor. You will not go out of business if you get the risk factors slightly wrong, you will if you can't attract customers or if they don't pay. In markets where you insurer fewer, bigger customers you care even less about accurately computing their individual risk, and more about a problem called aggregate risk - if you ensure all the residential blocks in San Francisco against earthquake damage, the Big One wipes you out. Insure one such block in every US city and you're fine, the Big One means you take a loss in SF but not in New York or Seattle. But if they're all built from Newsteel and then it turns out Newsteel turns into mush after 25 years you're back to big problems. So you are suddenly very interested in understanding what is the Same about things you insure.
- rando444 9y ago
  Even the obvious one, your insurance changes based on the type of car you're insuring.. because it's not all about cost. It's no secret that sports cars are more costly to insure, specifically because people are more likely to drive recklessly in them. If we didn't discriminate against people's car preferences, everyone's insurance would increase overnight.
discoursism 9y ago
I'd say there's a one really important difference between being Black and using Hotmail, and that difference is the same reason why one is a protected class and one isn't. The real concern here isn't discriminating against people with Hotmail accounts. On the surface level, there's nothing wrong with that. This does become an issue if the property of having a Hotmail account is used as an avenue to discriminate against people based on their membership in a protected class -- a sort of actuarial parallel construction, if you will. At that point, we do have a bit of a problem. > We need regulation in place to stop any punitive decision making, public and private, which can be found in court to be based on correlation instead of causation. So, young men could not be charged more for insurance than young women? In fact, young people in general would have to get the same price as everyone else? People in New York would have to pay the same as someone in rural Alaska? Someone who's been driving a year gets charged the same as someone who has been driving fifteen, etc.? Even being in an accident doesn't cause you to get in another accident. It's just correlated with that outcome. Throw out the entire actuarial aspect of insurance? These "punitive" charges are all based on correlation. I don't see this happening. ---- That said, as a non-statistician, I'm interested in what kinds of tests can be run on signals to verify that they're not proxies for selecting and discriminating against protected classes. Is there a way to modify a signal using its correlation with membership in these classes such that you can marginalize the attributes you're not allowed to discriminate based on? For example, it seems reasonable to believe that hotmail use is correlated with age, race, gender, etc. As an insurance company, I would have information on hand to inform me as to the extent of this correlation, at least among my customer base, and I would in turn know the correlations of those factors with risk. Could I somehow remove the racial, gender, and age components of the Hotmail risk signal to obtain a signal that only conveys the portion of the risk correlation that is not based on those classes? If so, what is that statistical technique called?
- lukego 9y ago
  There's a simple technique to "control for" those factors. First you split your population into groups that each have a similar race, gender, and age. Then within those groups you make comparisons of the hotmail users to the others. https://en.wikipedia.org/wiki/Controlling_for_a_variable https://en.wikipedia.org/wiki/Controlling_for_a_variable You would need to keep in mind xkcd #882 (https://xkcd.com/882/ https://xkcd.com/882/).
AFNobody 9y ago
> We need regulation in place to stop any punitive decision making, public and private, which can be found in court to be based on correlation instead of causation. Actuarial analysis for insurance to exercise a privilege (i.e. driving) is not something that needs to be more heavily regulated. This is not health care/insurance, food, housing, or other critical necessities. Correlation is a core part of actuarial science. You'd actually destroy the insurance business if you regulated that into nonexistence as you desire.
imjustsaying 9y ago
>scientifically prove That's not how the scientific method works.
indubitable 9y ago
What would you define as "scientifically proven"? In general, an observation that is observable, repeatable, and predictive is considered the criteria there. Take for instance genetics. The entire field is, with a few exceptions, built on nothing but correlations. And social science is 100% built on correlations. If a geneticist were to state that some gene or another were linked to the unibrow [1] would you consider this scientifically dubious? After all, in that case all they did was look at people who have unibrows and look to find characteristics that were more common to this group than others. What if that genetic indicator did show up disproportionately among all people with unibrows and was relatively less common among those without a unibrow? [1] - https://www.nature.com/articles/ncomms10815 https://www.nature.com/articles/ncomms10815
ggg9990 9y ago
> Unless you can scientifically prove that ethnicity causes crime to happen, its disgusting. If someone was to prove such a fact (say, a gene prevalent in some ethnicity that causes propensity for fraud) would you support decision making based on the ethnicity? How about based on the presence of the gene?
- fluxic 9y ago
  You should watch Gattaca.
  - lmm 9y ago
    And then consider the alternate ending: his heart condition lead him to die halfway through the mission, and without a pilot everyone else onboard died.
    - alien_at_work 9y ago
      I came away from the film with the impression that he didn't have and never had any heart condition. He was assumed to be super unhealthy because he was a natural birth (a kind of superstition, if you will), but that makes him like us. Do we all have heart conditions?
      - lmm 9y ago
        IIRC heart disease is the second most common cause of death, something in the region of 20%. So not all, but a substantial fraction of us have heart conditions.
        alien_at_work 9y ago
        Which might be why they thought that he had it. They seemed to be incorrect about something or he wouldn't routinely outperform his brother in physical sports.
        9y ago
zAy0LfpBZLC8mAC 9y ago
> Having a Hotmail account is a symptom of a problem which could lead to higher risk, but it is judging that symptom as if it is a cause which leads to risk. No, it's not. Insurance is not about establishing causality. Insurance is about assessing risk. If you are correctly assessing risk via proxy signals, you are still correctly predicting the losses that will have to be covered by your insurance, it's completely irrelevant how the losses are causally connected to the signals that you use for the risk assessment. If careless people are more likely to both cause accidents and to have hotmail accounts, then use of a hotmail account does correctly predict an increased risk. > Its exactly similar to the supposition that black people statistically commit more crime, and thus should pre-emptively receive harsher bail or be profiled. No, insurance is not about influencing people, it is about assessing risk. Insurance companies ultimately don't care about how much losses they have to cover, they only care about pricing their premiums such that they do have the money to pay for it. Insurance companies don't raise premiums to incentivize people to reduce risk. Insurance companies raise premiums because there are more losses to cover where the risk is higher. That that might also sometimes incentivize people to reduce risk is a side effect, but not an intention of the insurer.
beagle3 9y ago
> Having a Hotmail account is a symptom of a problem which could lead to higher risk, but it is judging that symptom as if it is a cause which leads to risk. No, it's not judging that symptom as a cause. It doesn't even matter if it's a cause or a symptom, as long as it is statistically sound. Hotmail, like [not] smoking, [not] exercising, [not] drinking alcohol, is a choice. Having chosen one way or the other does, statistically, imply other things. Whether we, as a society, want to allow this kind of statistical inference to meaningfully impact things like insurance costs, prison terms and other important decisions is a completely different issue.
- kome 9y ago
  gosh. no, having a hotmail account correlation it doesn't imply anything at all. is not causation, that's worth repeating 1000 times. > Hotmail, like [not] smoking, [not] exercising, [not] drinking alcohol, is a choice. lol.
  - fpig 9y ago
    Yelling that correlation is not causation 1000 times is ignorant when the effectiveness of a method depends only on correlation, and does not depend on the existence of a causal link. And in the case of filtering, that is the case. Think screening for a virus vs trying to cure the virus. If warts are a symptom of the virus, having warts absolutely matters in screening. However, cutting off the warts will not remove the virus.
  - beagle3 9y ago
    > correlation it doesn't imply anything at all. is not causation, that's worth repeating 1000 times. You might want to understand correlation, causation and dependence, and implication before repeating that 1000 times. I specifically used the term "statistically imply", which is correct, and means "has predictive power", which is true in this case of correlation whether or not causation is involved. Correlation actually implies a lot of things. At the very least, it implies statistical dependence (to the confidence attainable with the data). The converse is NOT true -- lack of correlation does not imply independence. But as independence implies LACK of correlation, correlation necessarily implies dependence. This is very basic probability, and is relevant to estimation of which insurance is an application. Causation is a completely different subject. > lol Care to elaborate? What exactly do you disagree with? Is any of the listed things not a choice?
Libturd 9y ago
Well, you have the numbers too back that up too ;)
fpig 9y ago
I disagree completely. Why does it matter whether it's just correlation or causation? Let's imagine that being black was actually what caused black people to commit more crimes and this was scientifically proven, how does this make it less bad to discriminate against some black guy who may or may not be a criminal? From a business perspective, correlation is a good measure and making decisions based on it is perfectly fine (if the correlation is real). From a moral perspective, it is not OK to discriminate based on correlation or causation. The distinction between correlation and causation matters when you are attempting to affect the outcome. For example you can't cure someone's sickle cell anemia by painting them white.
- smackay 9y ago
  That makes the assumption that you have modelled all the variables that make up the solution space. For the example the real, underlying reason might be poverty. So while for the purposes of the application the solution might be "good enough" but in the broader context (society) there are going to be serious consequences.
  - fpig 9y ago
    No, I do not make any such assumption. And I already covered for which purposes the "underlying cause" matters and for which it doesn't.
    - grive 9y ago
      Let's assume that higher crime rates are caused by poverty and not by the color of your skin. Let's say you accept the answer you've received based on your heuristic that black people will statistically commit more crimes. If a policy was based upon this heuristic, then you would have people living in poverty that would not be subject to the same policy. This is what discrimination is about. Citizens would thus not have the same rights / opportunity. And the only reason you are accepting this heuristic as "good enough", is because you make the assumption that "you have modelled [enough of] the variables that make up the solution space". This is not acceptable for policy makers and should be watched for closely in a democracy.
zardo 9y ago
>Unless you can scientifically prove that ethnicity causes crime to happen, its disgusting. This is obvious to us. Whether or not ethnicity is a causal factor is irrelevant. Even if it is a causal factor, it is still an injustice to act on it. And if you didn't care about justice and just wanted to achieve an optimal prediction, whether or not it's a causal factor doesn't actually matter. Probability theory does not require a causal link. "If it is rainy then it is cloudy." Is a valid inference despite the fact that rain does not cause clouds.
- dragonwriter 9y ago
  > "If it is rainy then it is cloudy." Is a valid deduction It is not, at least locally (because sunshowers happen.)
  - zardo 9y ago
    I thought I could sneak that fix in before anyone would notice. Yes, it's a valid inference, true with a high probability, but not a probability of 1.

by 013a 9y ago

ra 9y ago

that's sort of how insurance pricing works though. An actuarial process groups you with others based on similar attributes, and works out an overall risk value for you based on the combined risk profiles of the "groups" to which you belong. Historically, insurance companies only had things like postcode, gender, age, accident history to go off... but they're always interested in other data sets that could be used to modify pricing (or select less risky customers). Clearly someone or something has identified email domain as a factor (presumably based on data).

tialaramex 9y ago
They don't want "less risky customers" in car insurance and similar markets they want: More customers (full stop, if you have customers then your competitors don't have them, so you grow and they fail) Customers who pay more money Correctly assessing risk is not irrelevant but it's a small factor. You will not go out of business if you get the risk factors slightly wrong, you will if you can't attract customers or if they don't pay. In markets where you insurer fewer, bigger customers you care even less about accurately computing their individual risk, and more about a problem called aggregate risk - if you ensure all the residential blocks in San Francisco against earthquake damage, the Big One wipes you out. Insure one such block in every US city and you're fine, the Big One means you take a loss in SF but not in New York or Seattle. But if they're all built from Newsteel and then it turns out Newsteel turns into mush after 25 years you're back to big problems. So you are suddenly very interested in understanding what is the Same about things you insure.
rando444 9y ago
Even the obvious one, your insurance changes based on the type of car you're insuring.. because it's not all about cost. It's no secret that sports cars are more costly to insure, specifically because people are more likely to drive recklessly in them. If we didn't discriminate against people's car preferences, everyone's insurance would increase overnight.

discoursism 9y ago

I'd say there's a one really important difference between being Black and using Hotmail, and that difference is the same reason why one is a protected class and one isn't. The real concern here isn't discriminating against people with Hotmail accounts. On the surface level, there's nothing wrong with that. This does become an issue if the property of having a Hotmail account is used as an avenue to discriminate against people based on their membership in a protected class -- a sort of actuarial parallel construction, if you will. At that point, we do have a bit of a problem. > We need regulation in place to stop any punitive decision making, public and private, which can be found in court to be based on correlation instead of causation. So, young men could not be charged more for insurance than young women? In fact, young people in general would have to get the same price as everyone else? People in New York would have to pay the same as someone in rural Alaska? Someone who's been driving a year gets charged the same as someone who has been driving fifteen, etc.? Even being in an accident doesn't cause you to get in another accident. It's just correlated with that outcome. Throw out the entire actuarial aspect of insurance? These "punitive" charges are all based on correlation. I don't see this happening. ---- That said, as a non-statistician, I'm interested in what kinds of tests can be run on signals to verify that they're not proxies for selecting and discriminating against protected classes. Is there a way to modify a signal using its correlation with membership in these classes such that you can marginalize the attributes you're not allowed to discriminate based on? For example, it seems reasonable to believe that hotmail use is correlated with age, race, gender, etc. As an insurance company, I would have information on hand to inform me as to the extent of this correlation, at least among my customer base, and I would in turn know the correlations of those factors with risk. Could I somehow remove the racial, gender, and age components of the Hotmail risk signal to obtain a signal that only conveys the portion of the risk correlation that is not based on those classes? If so, what is that statistical technique called?

lukego 9y ago
There's a simple technique to "control for" those factors. First you split your population into groups that each have a similar race, gender, and age. Then within those groups you make comparisons of the hotmail users to the others. https://en.wikipedia.org/wiki/Controlling_for_a_variable https://en.wikipedia.org/wiki/Controlling_for_a_variable You would need to keep in mind xkcd #882 (https://xkcd.com/882/ https://xkcd.com/882/).

AFNobody 9y ago

> We need regulation in place to stop any punitive decision making, public and private, which can be found in court to be based on correlation instead of causation. Actuarial analysis for insurance to exercise a privilege (i.e. driving) is not something that needs to be more heavily regulated. This is not health care/insurance, food, housing, or other critical necessities. Correlation is a core part of actuarial science. You'd actually destroy the insurance business if you regulated that into nonexistence as you desire.

imjustsaying 9y ago

>scientifically prove That's not how the scientific method works.

indubitable 9y ago

What would you define as "scientifically proven"? In general, an observation that is observable, repeatable, and predictive is considered the criteria there. Take for instance genetics. The entire field is, with a few exceptions, built on nothing but correlations. And social science is 100% built on correlations. If a geneticist were to state that some gene or another were linked to the unibrow [1] would you consider this scientifically dubious? After all, in that case all they did was look at people who have unibrows and look to find characteristics that were more common to this group than others. What if that genetic indicator did show up disproportionately among all people with unibrows and was relatively less common among those without a unibrow? [1] - https://www.nature.com/articles/ncomms10815 https://www.nature.com/articles/ncomms10815

ggg9990 9y ago

> Unless you can scientifically prove that ethnicity causes crime to happen, its disgusting. If someone was to prove such a fact (say, a gene prevalent in some ethnicity that causes propensity for fraud) would you support decision making based on the ethnicity? How about based on the presence of the gene?

fluxic 9y ago
You should watch Gattaca.
- lmm 9y ago
  And then consider the alternate ending: his heart condition lead him to die halfway through the mission, and without a pilot everyone else onboard died.
  - alien_at_work 9y ago
    I came away from the film with the impression that he didn't have and never had any heart condition. He was assumed to be super unhealthy because he was a natural birth (a kind of superstition, if you will), but that makes him like us. Do we all have heart conditions?
    - lmm 9y ago
      IIRC heart disease is the second most common cause of death, something in the region of 20%. So not all, but a substantial fraction of us have heart conditions.
      - alien_at_work 9y ago
        Which might be why they thought that he had it. They seemed to be incorrect about something or he wouldn't routinely outperform his brother in physical sports.
        9y ago

zAy0LfpBZLC8mAC 9y ago

> Having a Hotmail account is a symptom of a problem which could lead to higher risk, but it is judging that symptom as if it is a cause which leads to risk. No, it's not. Insurance is not about establishing causality. Insurance is about assessing risk. If you are correctly assessing risk via proxy signals, you are still correctly predicting the losses that will have to be covered by your insurance, it's completely irrelevant how the losses are causally connected to the signals that you use for the risk assessment. If careless people are more likely to both cause accidents and to have hotmail accounts, then use of a hotmail account does correctly predict an increased risk. > Its exactly similar to the supposition that black people statistically commit more crime, and thus should pre-emptively receive harsher bail or be profiled. No, insurance is not about influencing people, it is about assessing risk. Insurance companies ultimately don't care about how much losses they have to cover, they only care about pricing their premiums such that they do have the money to pay for it. Insurance companies don't raise premiums to incentivize people to reduce risk. Insurance companies raise premiums because there are more losses to cover where the risk is higher. That that might also sometimes incentivize people to reduce risk is a side effect, but not an intention of the insurer.

beagle3 9y ago

> Having a Hotmail account is a symptom of a problem which could lead to higher risk, but it is judging that symptom as if it is a cause which leads to risk. No, it's not judging that symptom as a cause. It doesn't even matter if it's a cause or a symptom, as long as it is statistically sound. Hotmail, like [not] smoking, [not] exercising, [not] drinking alcohol, is a choice. Having chosen one way or the other does, statistically, imply other things. Whether we, as a society, want to allow this kind of statistical inference to meaningfully impact things like insurance costs, prison terms and other important decisions is a completely different issue.

kome 9y ago
gosh. no, having a hotmail account correlation it doesn't imply anything at all. is not causation, that's worth repeating 1000 times. > Hotmail, like [not] smoking, [not] exercising, [not] drinking alcohol, is a choice. lol.
- fpig 9y ago
  Yelling that correlation is not causation 1000 times is ignorant when the effectiveness of a method depends only on correlation, and does not depend on the existence of a causal link. And in the case of filtering, that is the case. Think screening for a virus vs trying to cure the virus. If warts are a symptom of the virus, having warts absolutely matters in screening. However, cutting off the warts will not remove the virus.
- beagle3 9y ago
  > correlation it doesn't imply anything at all. is not causation, that's worth repeating 1000 times. You might want to understand correlation, causation and dependence, and implication before repeating that 1000 times. I specifically used the term "statistically imply", which is correct, and means "has predictive power", which is true in this case of correlation whether or not causation is involved. Correlation actually implies a lot of things. At the very least, it implies statistical dependence (to the confidence attainable with the data). The converse is NOT true -- lack of correlation does not imply independence. But as independence implies LACK of correlation, correlation necessarily implies dependence. This is very basic probability, and is relevant to estimation of which insurance is an application. Causation is a completely different subject. > lol Care to elaborate? What exactly do you disagree with? Is any of the listed things not a choice?

Libturd 9y ago

Well, you have the numbers too back that up too ;)

fpig 9y ago

I disagree completely. Why does it matter whether it's just correlation or causation? Let's imagine that being black was actually what caused black people to commit more crimes and this was scientifically proven, how does this make it less bad to discriminate against some black guy who may or may not be a criminal? From a business perspective, correlation is a good measure and making decisions based on it is perfectly fine (if the correlation is real). From a moral perspective, it is not OK to discriminate based on correlation or causation. The distinction between correlation and causation matters when you are attempting to affect the outcome. For example you can't cure someone's sickle cell anemia by painting them white.

smackay 9y ago
That makes the assumption that you have modelled all the variables that make up the solution space. For the example the real, underlying reason might be poverty. So while for the purposes of the application the solution might be "good enough" but in the broader context (society) there are going to be serious consequences.
- fpig 9y ago
  No, I do not make any such assumption. And I already covered for which purposes the "underlying cause" matters and for which it doesn't.
  - grive 9y ago
    Let's assume that higher crime rates are caused by poverty and not by the color of your skin. Let's say you accept the answer you've received based on your heuristic that black people will statistically commit more crimes. If a policy was based upon this heuristic, then you would have people living in poverty that would not be subject to the same policy. This is what discrimination is about. Citizens would thus not have the same rights / opportunity. And the only reason you are accepting this heuristic as "good enough", is because you make the assumption that "you have modelled [enough of] the variables that make up the solution space". This is not acceptable for policy makers and should be watched for closely in a democracy.

zardo 9y ago

>Unless you can scientifically prove that ethnicity causes crime to happen, its disgusting. This is obvious to us. Whether or not ethnicity is a causal factor is irrelevant. Even if it is a causal factor, it is still an injustice to act on it. And if you didn't care about justice and just wanted to achieve an optimal prediction, whether or not it's a causal factor doesn't actually matter. Probability theory does not require a causal link. "If it is rainy then it is cloudy." Is a valid inference despite the fact that rain does not cause clouds.

dragonwriter 9y ago
> "If it is rainy then it is cloudy." Is a valid deduction It is not, at least locally (because sunshowers happen.)
- zardo 9y ago
  I thought I could sneak that fix in before anyone would notice. Yes, it's a valid inference, true with a high probability, but not a probability of 1.