4 ms·

Hit the nail on the nose. Sure, you can build a discriminating classifier or generative model that is the most accurate, correctly identifying / emulating the

by maldeh 8y ago

Hit the nail on the nose. Sure, you can build a discriminating classifier or generative model that is the most accurate, correctly identifying / emulating the reality of our world down to 5 nines; and nobody says you shouldn't be able to identify or quantify all of these "inconvenient" associations. The trouble is always when you intend to make decisions based on your framework -- the decision to offer somebody a loan, the choice of language your chatbot uses, etc. -- and that is where fairness ought to be paramount. And yes, if you manage to build a discriminating model that sneaks in protected classes through indirect causal effects with no attempt to suppress them, it would yield your insurance agency higher returns over time, and just because that's the way the world is, currently. But all this would achieve is perpetuating the current status quo, placing short-term gain ahead of long-term equality. You might be doing right by yourself, but that still makes you morally impugnable. One shouldn't need to wait for an overreaching law prohibiting such indirection, to do the right thing.

haberman 8y ago
If a person is loan-worthy, offering them a loan is good business sense. If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. How far do you expect systems to go to ignore real-world associations? What if it is coming down to personal safety? Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?
- trjordan 8y ago
  One problem is the world has layered effects of bias. Imagine a simple system where people are only evaluated for credit cards and mortgages, and businesses can only offer one of these services. Some people are systematically denied credit cards. Now they have to have more cash for daily needs, which makes them objectively worse as home owners, so the mortgage loaners pick up on this fact and start systematically denying them mortgages. There's no business opportunity here. For either lender, the prospects in this group are worse, only because the other lenders think so. Breaking the barricade is much harder than just starting a new lender or either type.
  - AstralStorm 8y ago
    Also known as self fulfilling prophecies. And then runaway feedback. Three strength of the effect varies but is hard to measure. Not impossible though.
- paganel 8y ago
  > If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. Afaik the black families who were segregated out of their neighborhoods back in the 1940s, 1950s and going into the 1960s were never provided with another comparable "business opportunity" that could have "righted" their situation, they had to settle with living in what turned out to become "ghettos" (for lack of a better word). > Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women? Following the same line of thought, would you be ok with an AI system giving a person named Deion or Jayla or Latisha a higher interest rate on their mortgage (and so, potentially, driving them out of certain markets) compared to the interest rate offered to persons named Chad or Emma or Sophia?
  - PeterisP 8y ago
    If we know nothing else about them but their names (which is a quite big assumption), a person named Deion or Jayla or Latisha is more likely to come from a disadvantaged socioeconomic background than a person named Chad or Emma or Sophia, have a worse job, have a somewhat higher chance of getting arrested in future, and due to these and other factors they causally, objectively have a somewhat higher risk of defaulting on that loan. It's not just bias, prejudice or superstition; it's not a mistake in data or process; it's an indication of some true, factual observation about reality. The correlation isn't huge, but it is there, and it's not a coincidence. If it's your money that you're choosing to lend (or not lend), would you ignore that information or take it into account when making a decision if you should offer that loan and how much the rate should be adjusted to cover the risk of default? What if it's not your money to do with it as you please, but money entrusted to you to invest for maximum results? Sure, if you knew what this particular person is actually like, then you'd base the decision on better data than just their name, and it could well be that this Jayla would get a much better deal than this Sophia; but even if you had a perfectly fair system that correctly estimates the individual risk of default without prejudice causing worthy individuals being dragged down by association with some group, then there would still be a correlation with the name simply because in the current reality the proportion of "risky Jaylas" is larger than the proportion of "risky Sophias".
lopmotr 8y ago
If you don't want unfairness, why are you denying anybody a loan? Maybe I'm bankrupt because of a one-off medical expense and now I'm healthy again, I want a loan. It surely won't happen again but the lender says no. That's unfair, isn't it? So the problem already exists and nobody cares.
- AstralStorm 8y ago
  Fairness is a different concept from justice. It might be fair to deny but still unjust. Also different concept from transparency. Three operational principle of the capitalist business is extracting value from a customer or an externality. That is from the outset unfair. It is just, as long as businesses act independently.

by maldeh 8y ago

haberman 8y ago

If a person is loan-worthy, offering them a loan is good business sense. If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. How far do you expect systems to go to ignore real-world associations? What if it is coming down to personal safety? Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?

trjordan 8y ago
One problem is the world has layered effects of bias. Imagine a simple system where people are only evaluated for credit cards and mortgages, and businesses can only offer one of these services. Some people are systematically denied credit cards. Now they have to have more cash for daily needs, which makes them objectively worse as home owners, so the mortgage loaners pick up on this fact and start systematically denying them mortgages. There's no business opportunity here. For either lender, the prospects in this group are worse, only because the other lenders think so. Breaking the barricade is much harder than just starting a new lender or either type.
- AstralStorm 8y ago
  Also known as self fulfilling prophecies. And then runaway feedback. Three strength of the effect varies but is hard to measure. Not impossible though.
paganel 8y ago
> If others are denying loans for no good reason, that is a business opportunity. It doesn't require benevolence to offer people loans that they are qualified for. Afaik the black families who were segregated out of their neighborhoods back in the 1940s, 1950s and going into the 1960s were never provided with another comparable "business opportunity" that could have "righted" their situation, they had to settle with living in what turned out to become "ghettos" (for lack of a better word). > Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women? Following the same line of thought, would you be ok with an AI system giving a person named Deion or Jayla or Latisha a higher interest rate on their mortgage (and so, potentially, driving them out of certain markets) compared to the interest rate offered to persons named Chad or Emma or Sophia?
- PeterisP 8y ago
  If we know nothing else about them but their names (which is a quite big assumption), a person named Deion or Jayla or Latisha is more likely to come from a disadvantaged socioeconomic background than a person named Chad or Emma or Sophia, have a worse job, have a somewhat higher chance of getting arrested in future, and due to these and other factors they causally, objectively have a somewhat higher risk of defaulting on that loan. It's not just bias, prejudice or superstition; it's not a mistake in data or process; it's an indication of some true, factual observation about reality. The correlation isn't huge, but it is there, and it's not a coincidence. If it's your money that you're choosing to lend (or not lend), would you ignore that information or take it into account when making a decision if you should offer that loan and how much the rate should be adjusted to cover the risk of default? What if it's not your money to do with it as you please, but money entrusted to you to invest for maximum results? Sure, if you knew what this particular person is actually like, then you'd base the decision on better data than just their name, and it could well be that this Jayla would get a much better deal than this Sophia; but even if you had a perfectly fair system that correctly estimates the individual risk of default without prejudice causing worthy individuals being dragged down by association with some group, then there would still be a correlation with the name simply because in the current reality the proportion of "risky Jaylas" is larger than the proportion of "risky Sophias".

lopmotr 8y ago

If you don't want unfairness, why are you denying anybody a loan? Maybe I'm bankrupt because of a one-off medical expense and now I'm healthy again, I want a loan. It surely won't happen again but the lender says no. That's unfair, isn't it? So the problem already exists and nobody cares.

AstralStorm 8y ago
Fairness is a different concept from justice. It might be fair to deny but still unjust. Also different concept from transparency. Three operational principle of the capitalist business is extracting value from a customer or an externality. That is from the outset unfair. It is just, as long as businesses act independently.