15 ms·

EU regulations on algorithmic decision-making and a “right to explanation”

104 points by pmlnr 10y ago

ddebernardy 10y ago
Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
- chriswarbo 10y ago
  I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...
  - PeterisP 10y ago
    Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.
    - fixermark 10y ago
      Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.
      - PeterisP 10y ago
        If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal. However, if you do some reason analysis, then this is useful and gives a lot of interesting information. Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc. Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available. Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability. I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.
        fixermark 10y ago
        This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it. "Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."
    - crististm 10y ago
      Maybe Eric Shmidt is a lower risk customer because he's opinion on "don't be evil" is that is a bad idea? :)
    - matt4077 10y ago
      If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.
      - PeterisP 10y ago
        I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.
        throwawaysocks 10y ago
        Definitely.
- crististm 10y ago
  You are certainly right to question the reasoning behind AI decisions. I'm not sure that neural network based AI can do that though. For reference see an older thread on this very topic: https://news.ycombinator.com/item?id=10388795 https://news.ycombinator.com/item?id=10388795
  - throwawaysocks 10y ago
    > I'm not sure that neural network based AI can do that though. Generating human-readable explanations from a machine learning algorithm or NN is probably an impossible problem to solve in full generality. However, for many particular systems, doing so should be feasible.
    - sapphireblue 10y ago
      Explaining how neural network models generate their output is possible, see http://cs231n.github.io/understanding-cnn/ http://cs231n.github.io/understanding-cnn/ https://auduno.github.io/2016/06/18/peeking-inside-convnets/ https://auduno.github.io/2016/06/18/peeking-inside-convnets/ https://courses.cs.washington.edu/courses/cse590v/14au/cse590v_dec5_DeepVis.pdf https://courses.cs.washington.edu/courses/cse590v/14au/cse59... Also it should be possible to train the NN model to explain its decision, even in plain language. I think the "right to explanation" law is pretty sensible, people should have a right to question the algorithmic judgment that impacts their life. It will be somewhat inconvenient for big companies applying ML to make these decisions, but it won't ruin their business.
vixen99 10y ago
So what effect will this have on risk assessment as used by insurance companies? "Insurance companies use a methodology called risk assessment to calculate premium rates for policyholders. Using software that computes a predetermined algorithm, insurance underwriters gauge the risk that you may file a claim against your policy. These algorithms are based on key indicators about you and then measured against a data set to weigh risk. Insurance underwriters carefully balance the insurance company’s profitability with your potential need to use the policy."
tzs 10y ago
The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reason might be: “your income was too low” or “you haven’t been employed long enough.” An unacceptable reason might be “you didn’t meet our minimum standards.” That information isn’t specific enough. https://www.consumer.ftc.gov/articles/0347-your-equal-credit-opportunity-rights https://www.consumer.ftc.gov/articles/0347-your-equal-credit... (The ECOA is at 15 U.S. Code § 1691, for those who want to go find the gory details) In one of the lectures from Caltech's "Learning from Data" MOOC the professor mentioned that problem came up at a major lender that he had consulted for. He did not say how they solved the problem. I wondered if you could generate a satisfactory rejection reason, at least as far as the law is concerned, by taking rejected applications and running them through the system again but with some of the inputs tweaked until they get an acceptance. Then report that the rejection was due to the un-tweaked parameter value. For instance, if you picked income to tweak, you'd raise the income by, say, $5000 and try again. If that is rejected, raise it another $5000. If it then passes, you can tell the applicant they were rejected because their income was too low, and that an additional $10000 income would be enough to qualify them for a loan. You'd have to put a bit of sophistication into this to make the explanations reasonable. For instance, if a rejected application would be approved with a very large tweak to income or with a small tweak to employment length it would probably be better to give employment length as the reason for rejection.
- PeterisP 10y ago
  A potential problem with this is that if you "tell the applicant they were rejected because their income was too low" then if this is disputed then they will likely be able to point to many other applicants with even lower income that were accepted, because their total combination of factors was better. A simple explanation "factor X is too low" implies that there exists a particular cutoff that is required and sufficient; but an explanation that accurately describes why you were rejected would likely be too complex to be understandable. But yes, probably the intended result could/should be simply identifying the top 1-2 factors that dragged your score down compared to the average accepted candidate (which can be rather easily done automatically even for the most black-box ML methods) and naming them appropriately.
sergioisidoro 10y ago
I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and specially if the models have high impact on people's lives (applying to loans, university, scholarship, insurance), we cannot afford to blindly use existing annotations in black box models. All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. Edit: here is code for the curious https://github.com/sergioisidoro/aequalis https://github.com/sergioisidoro/aequalis
- AnthonyMouse 10y ago
  > Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time travel to tell you with 100.0% accuracy who would pay back their loans, there would be a racial disparity in the result, but the cause of it is not the fault of the algorithm. And you can't fix it there because that's not where the problem is. Suppose you have a group of 800 white middle managers, 100 white sales clerks and 100 black sales clerks. Clearly the algorithm is going to have a racial disparity in outcome if the middle managers are at 500% of the cutoff and the sales clerks are right on the line, because it will accept all of the middle managers and half of the sales clerks which means it will accept >94% of the white applicants and 50% of the black applicants. But the source of the disparity is that black people are underrepresented as middle managers and overrepresented as sales clerks. The algorithm is just telling you that. It can't change it. And inserting bias into the algorithm to "balance" the outcome doesn't actually do that, all you're doing is creating bias against white sales clerks who had previously been on the same footing as black sales clerks. The white middle managers will be unaffected because they're sufficiently far above the cutoff that the change doesn't affect them, even though that they're the source of the imbalance.
mknocker 10y ago
The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?
- matt4077 10y ago
  Nope, because advertisement doesn't "significantly affect" you. At least that's how it's meant and how it will be understood. This is about credit, employment, insurance, medical care etc.
  - Symmetry 10y ago
    Medical care?! Oh God. Well, I suppose the AMA will keep the US from embracing machine learning so it's not like this is actually a chance for the US to reverse its lag in mortality statistics.
  - csn 10y ago
    Although just like the things you list, I can think of a plethora of ways with which advertising can literally, indirectly change my life by convincing me to purchase a product or service.
- denzil_correa 10y ago
  There is some research which shows discrimination in online advertisements. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2208240 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2208240 http://queue.acm.org/detail.cfm?id=2460278 http://queue.acm.org/detail.cfm?id=2460278
- Kristine1975 10y ago
  Just put a link under the ad "why am I seeing this ad?" leading to a page with the explanation.
zoner 10y ago
Finally I get to know why my mortgage was declined :)
- phkahler 10y ago
  >> Finally I get to know why my mortgage was declined :) My first attempt at one was declined due to "problems with my credit report" or some such. I happened to be working at a place where they did credit checks on customers, so I asked someone to pull my credit report. They had all my stuff, but had mixed in a bunch of information from someone else with the same name - different age, there were loans on there from when I was 3. I called the reporting company to complain and they kept getting caught up in "how did you get your report?" as if I had done something wrong in just having access to their incorrect information. Perhaps having someone pull the info instead of the "consumer" getting it through proper channels was against some rule - this was 1995. I think the same is often true today, companies want to collect data on you but don't really want you to know what they've got or how they use it.
  - dublinben 10y ago
    This is exactly why you are legally entitled to receive a copy of your credit report every year. You can challenge anything on it, and have incorrect information removed.
    - phkahler 10y ago
      That had to be made explicit in law precisely because the companies don't want you to have it.
    - jerf 10y ago
      When mentioning this, I find it helpful to include this page: https://www.ftc.gov/faq/consumer-protection/get-my-free-credit-report https://www.ftc.gov/faq/consumer-protection/get-my-free-cred... If you are one of the people just finding out about this today, and you Google "free credit report", you come up with a loooot of bad, scammy links. This is how you get your legally-mandated, really free, annual credit report, starting from an ftc.gov address so you know it's really the right one.
dharma1 10y ago
I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of the data, or offer manual review of the application if rejected (which is something they do) - but it would be very hard to show exact causal reasoning patterns in the automatic scoring.
- dec0dedab0de 10y ago
  why not just have an optional pdf download with the 20000 vectors, and how much they were weighted.
- GFK_of_xmaspast 10y ago
  Sounds like a limitation of their business model.
- kaijush 10y ago
  >> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.
  - JoeAltmaier 10y ago
    Because the training set was covered with x% accuracy. Ok, how does knowing that help?
    - kaijush 10y ago
      OK, so we're assuming supervised learning. In that case, how were the features of the training set chosen? And how was the training set labelled? Most of the time, outside of semi-supervised learning, those things are not in the data, someone has to (painstakingly) decide them. That can very well be explained I believe. More generally, what I mean to say is that you can learn any number of models from the same data, so to a great extent you choose what your model learns by manipulating your algorithm's (hyper)parameters, the training set's features, choosing what goes into the training set itself etc etc. That process leaves room for a lot of regulatory oversight. - in principle. In practice I expect since we're talking about people lending money or selling insurance we're only ever going to get those details out of their cold dead hands, and only with considerable effort at that.
Houshalter 10y ago
Is my reading of this law correct? It sounds like it outlaws all use of algorithms to evaluate people. The right to explanation is just for the rare exceptions where a "member state" authorizes it. But the legalese is difficult to parse, and no one else seems to get this impression. I think this is really bad. That's the majority of uses of machine learning. It also has a lot of economic value to predict things like how likely someone is to pay back a loan, or even who is a spammer. Most importantly, most these applications will go back to human judgement. And humans are far worse. Human predictions about candidates are really bad. We are incredibly biased by things like race, gender, and especially attractiveness. Unattractive people get twice the sentences of attractive people. Not to mention random stuff, like people being judged more harshly when the judge is hungry before lunch. They also rarely give explanations for their decisions, if they are even aware of the true reasons for them. Going back to humans is a huge step backwards and will hurt a lot more people than it helps. I think the same regulations that apply to algorithms, should apply to humans. That would show the absurdity of these laws. But humans are algorithms after all. And particularly bad ones (for this purpose anyway.)
- germanier 10y ago
  > It sounds like it outlaws all use of algorithms to evaluate people. The right to explanation is just for the rare exceptions where a "member state" authorizes it. Who says that this is rare? That's a pretty standard way EU regulations are written to mean "Action A can only be allowed if conditions B are met". If the regulation is approved, member states take a look and change their laws accordingly. They could outlaw algorithmic decision making if they wanted to (but they wouldn't need that regulation to do that). Much more likely is that they impose those conditions on the companies that use such algorithms just like they already impose a lot of conditions (such as "ethnicity is not allowed to be a direct input variable" currently is).
  - ddebernardy 10y ago
    > If the regulation is approved, member states take a look and change their laws accordingly. Small nitpick id I may: you seem to be confusing Regulations with Directives, which must get transcribed within a certain timeframe in national law. Regulations apply directly - no transcription needed. Also: either way, EU law has primacy over national law.
deleted 10y ago
[deleted]
InTheArena 10y ago
Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This is really needed for a number of different reasons. The biggest is that algorithms are increasingly running our world. Even outside of mortgage, have you ever had to fork over the right to pull your credit score for either rent or (in some limited cases) to apply for a job? Not being able to understand: a) Why you were rejected. b) Why you are paying a higher interest rate. c) What you can do to fix that? Housing and employment is fundamentally a question of human rights, not just banking profitability or academic algorithm improvements. Part of the reason why Credit Scoring (which is intrinsically algorithmic decision-making) is because it replaced the "old boys" network that used to dominate. In this world you went to your banker, and the decision to extend credit was dominated by your personal relationship with a banker who might not be the same ethnicity, religion, nationality, etc as you would dominate your ability to purchase housing. The credit score democratized financial access. It can still be used to discriminate. There have been a number of studies that have shown that simply being from a white community tends to increase the number of other people you can lean on in a emergency, and hence make you less likely to default on a loan. From a pure predictability point of view, a bank is at a lower risk of default lending to someone from that community, but that in turn denies financial access to people not in that community, continuing the trend. A big problem in this kind of financial model is that it's relatively easy to accidentally or deliberately find other variables that mirror "protected" vectors. A simplistic example is zip code, where the zip code reflects a area that is predominately ethnic minorities. So it's not cut and dried. It's my PoV that It's not predictability versus red-tape, and the people trying to do unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism.
- deleted 10y ago
  [deleted]
- sergioisidoro 10y ago
  I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)
gersh 10y ago
Can't you just say the decision was based on a random forest trained on the following parameters with historical data. You could get more specific on how the model was trained or even show model if you have to. I'm sure you can create a small-print tree diagram. In terms of human review, does this require more than a spot check? I can imagine look over the p-value and possibly spot check the input data, and then approve it.
Symmetry 10y ago
I wonder machine learning could just handle this the way that humans do. Human decisions processes are of course extremely opaque even if we're good at coming up with post-hoc justifications for our reasoning as needed. "You're income is too low" "But his income is also low and he got a loan!" "Well he has a better credit score." etc It would obviously be impossible to give a complete and accurate explanation of a machine learning decision process the same way it would be impossible for a human learning decision process, but I wonder if it would actually be difficult to come up with particularly salient distinguishing features in general or between certain cases the same way that people do?
Kenji 10y ago
Good luck arguing the 'why' when you have a trained neural network and all you have is the network and weights. On a more serious note, I love transparency but again, this is an overeager regulation (not surprising from the EU). You almost never get the true reasons for being rejected, be it in an interview or for a credit, etc. You have to figure that out yourself from the often very vague rejection letter. Our minds are basically running algorithms to which we have even less insight than our computer algorithms. Therefore, this law only hampers the flourishing of the economy while providing no value whatsoever. Another law that 'solves' a non-issue, brought into being by overpaid career politicians.
- sparkie 10y ago
  If all you have is a network and weights, you have a black box. While it might be magic and make the right decision most of the time, we need to look in and see why those decisions are being made, somehow. This generally basically means seeing how the net was trained, and its initial conditions. If you threw away your training methodology and data set, why should anyone trust that your algorithm can make suitable decisions? Are we to assume that nobody who writes AIs which are making influential decisions has vested interests? If you look for example at common law; we have a system where decisions by judges are binding, and future equivalent cases are required to uphold the previous decisions. The only way this system can work of course, is if someone is keeping a record of the previous decisions. The alternative is we'd have a system of judgement based on heresay, and we can have about as much confidence in it as asking a random person on the street to make the decision. The technical challenge is really about storage and retreival. If we know that discarding training data makes our neural networks "unaccountable for their decisions", then our technical requirement must be that we store the training data in its entirity, such that we can look back and maybe glimpse at why a neural network may behave as it has. For example, I might create an AI which is used to decide whether to give someone a mortgage. I could have millions of samples as training data, but I might chose to sort the training data by race, and begin training a network initially so that non-white people who have been rejected mortgages initially overtrain the network to correlate race with rejection, then use a limited sample of white man, mostly accepted mortgage requests to train it the other way. Of course this is extreme, and an expert who looked at such data set would quickly notice that the AI is unfit for purpose, and blatantly racist. But he needs the training set to even have a chance of concluding that. Without it, he has effectively random numbers which tell him next to nothing. These laws aren't meant to stifle innovation or economic benefits, but only to ensure that fair treatment is practiced in their development. As far as I see it, if you have a neural network, a sound justification of its design and the methodology you used, combined with a complete data set and training set which can be analysed for biases, then there's no reason these regulations should get in your way.
deleted 10y ago
[deleted]
bwgoodman 10y ago
hi, I'm one of the co-authors. This feedback is awesome. I have a longer version in the works. if you're interested please feel free to email me, bryce dot goodman at stx dot ox dot ac dot uk. Thanks! Bryce
zxcvvcxz 10y ago
> We argue that while this law will pose large challenges for industry, it highlights opportunities for machine learning researchers to take the lead in designing algorithms and evaluation frameworks which avoid discrimination. This sounds like a very slippery slope. Let me get this straight, what the EU wants to do here. If I make a business that decides how to, say, make loans to people, and I have software that analyzes how they filled out an application or took a written/logical test or something I may have to: 1) Disclose the findings and methodology of my proprietary algorithm 2) Potentially undo decisions that are deemed discriminatory 3) Thereby potentially be forced to choose who I do and do not do business with Some regulations are good, they help us avoid things like moral hazards. Some regulations, however, act more like power grabs that take away freedom from individuals and businesses. I'm sorry to say but this sort of thing sounds like the latter. There are some uncomfortable facts about people and society that algorithms will uncover, and that people will try using to avoid making mistakes. Here's a poignant example, albeit with much less sophistication: http://www.nytimes.com/2015/10/31/nyregion/hudson-city-bank-settlement.html http://www.nytimes.com/2015/10/31/nyregion/hudson-city-bank-... > Instead, some officials say, some banks have quietly institutionalized bias in their operations, deliberately placing branches, brokers and mortgage services outside minority communities, even as other banks find and serve borrowers in those neighbourhoods. Alright sure. I think we all know it's common sense to not open a jewellery store in inner city Chicago. The local clientele would not be able to afford such products and services, and that's enough of an argument, it's probably business school 101. Just like how you're not going to find a good market for a food delivery startup in Middletown, USA. The economics probably don't work. EDIT - and if you think these businesses are wrong, and they should be in these communities, let their competitors eat their lunch! But it's viewed as discriminatory. Back to the paper, it's discussing essentially the same thing: it's the natural evolution of "don't start a jewellery store in inner city Chicago" turning into "don't start a jewellery store in neighbourhood where access to urban center is less than X, available nearby 3-lane highways is less than Y, percent of women from 18-29 is less than Z, proportion of people from culture W who are in the bottom quartile of people who buy jewellery is more than V, ...". The solution here isn't to regulate and force people to open jewellery stores in inner city Chicago. The solution, if we really want to bring diamonds to Englewood, is to drastically reduce the poverty and crime rates so that businesses will want to sell there (https://en.wikipedia.org/wiki/Englewood,_Chicago#Socioeconomics https://en.wikipedia.org/wiki/Englewood,_Chicago#Socioeconom...). And if you think having access to a local business is a "right", then you must concede that such a public good should be provided by a central government of some type. And then you have to ask yourself, do you want the government managing your bank loans, or would you rather have private companies compete on prices and service? If you want the latter, then you need to accept that certain things are just going to arise (or not arise) naturally.

104 points by pmlnr 10y ago

Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm... In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?

chriswarbo 10y ago
I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...
- PeterisP 10y ago
  Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.
  - fixermark 10y ago
    Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.
    - PeterisP 10y ago
      If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal. However, if you do some reason analysis, then this is useful and gives a lot of interesting information. Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc. Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available. Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability. I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.
      - fixermark 10y ago
        This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it. "Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."
  - crististm 10y ago
    Maybe Eric Shmidt is a lower risk customer because he's opinion on "don't be evil" is that is a bad idea? :)
  - matt4077 10y ago
    If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.
    - PeterisP 10y ago
      I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.
      - throwawaysocks 10y ago
        Definitely.
crististm 10y ago
You are certainly right to question the reasoning behind AI decisions. I'm not sure that neural network based AI can do that though. For reference see an older thread on this very topic: https://news.ycombinator.com/item?id=10388795 https://news.ycombinator.com/item?id=10388795
- throwawaysocks 10y ago
  > I'm not sure that neural network based AI can do that though. Generating human-readable explanations from a machine learning algorithm or NN is probably an impossible problem to solve in full generality. However, for many particular systems, doing so should be feasible.
  - sapphireblue 10y ago
    Explaining how neural network models generate their output is possible, see http://cs231n.github.io/understanding-cnn/ http://cs231n.github.io/understanding-cnn/ https://auduno.github.io/2016/06/18/peeking-inside-convnets/ https://auduno.github.io/2016/06/18/peeking-inside-convnets/ https://courses.cs.washington.edu/courses/cse590v/14au/cse590v_dec5_DeepVis.pdf https://courses.cs.washington.edu/courses/cse590v/14au/cse59... Also it should be possible to train the NN model to explain its decision, even in plain language. I think the "right to explanation" law is pretty sensible, people should have a right to question the algorithmic judgment that impacts their life. It will be somewhat inconvenient for big companies applying ML to make these decisions, but it won't ruin their business.

vixen99 10y ago

So what effect will this have on risk assessment as used by insurance companies? "Insurance companies use a methodology called risk assessment to calculate premium rates for policyholders. Using software that computes a predetermined algorithm, insurance underwriters gauge the risk that you may file a claim against your policy. These algorithms are based on key indicators about you and then measured against a data set to weigh risk. Insurance underwriters carefully balance the insurance company’s profitability with your potential need to use the policy."

tzs 10y ago

The United States does not have anything as sweeping as this, but in the limited area of credit the Equal Credit Opportunity Act (ECOA) requires that lenders that turn down your credit application give you an explanation. From the Federal Trade Commission's site: The creditor must tell you the specific reason for the rejection or that you are entitled to learn the reason if you ask within 60 days. An acceptable reason might be: “your income was too low” or “you haven’t been employed long enough.” An unacceptable reason might be “you didn’t meet our minimum standards.” That information isn’t specific enough. https://www.consumer.ftc.gov/articles/0347-your-equal-credit-opportunity-rights https://www.consumer.ftc.gov/articles/0347-your-equal-credit... (The ECOA is at 15 U.S. Code § 1691, for those who want to go find the gory details) In one of the lectures from Caltech's "Learning from Data" MOOC the professor mentioned that problem came up at a major lender that he had consulted for. He did not say how they solved the problem. I wondered if you could generate a satisfactory rejection reason, at least as far as the law is concerned, by taking rejected applications and running them through the system again but with some of the inputs tweaked until they get an acceptance. Then report that the rejection was due to the un-tweaked parameter value. For instance, if you picked income to tweak, you'd raise the income by, say, $5000 and try again. If that is rejected, raise it another $5000. If it then passes, you can tell the applicant they were rejected because their income was too low, and that an additional $10000 income would be enough to qualify them for a loan. You'd have to put a bit of sophistication into this to make the explanations reasonable. For instance, if a rejected application would be approved with a very large tweak to income or with a small tweak to employment length it would probably be better to give employment length as the reason for rejection.

PeterisP 10y ago
A potential problem with this is that if you "tell the applicant they were rejected because their income was too low" then if this is disputed then they will likely be able to point to many other applicants with even lower income that were accepted, because their total combination of factors was better. A simple explanation "factor X is too low" implies that there exists a particular cutoff that is required and sufficient; but an explanation that accurately describes why you were rejected would likely be too complex to be understandable. But yes, probably the intended result could/should be simply identifying the top 1-2 factors that dragged your score down compared to the average accepted candidate (which can be rather easily done automatically even for the most black-box ML methods) and naming them appropriately.

sergioisidoro 10y ago

I made a couple of experiments on discrimination free machine learning models with naive Bayes, and I changed my perspective on data science. Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Removing sensitive variables (gender, race, etc) doesn't always help, and specially if the models have high impact on people's lives (applying to loans, university, scholarship, insurance), we cannot afford to blindly use existing annotations in black box models. All of this excludes the fact that companies will probably maximise to profit, and will use "algorithm" as an excuse to turn down people disregarding ethics. Edit: here is code for the curious https://github.com/sergioisidoro/aequalis https://github.com/sergioisidoro/aequalis

AnthonyMouse 10y ago
> Usually people are concerned about maximising prediction accuracy, and never stop to think about what correlations is the model finding down below, and the human biases present in the data annotations. Because maximizing prediction accuracy is inherently unbiased. Bias is when the predictions made are inaccurate to the detriment of a group of people. If you had a prediction algorithm that functioned using time travel to tell you with 100.0% accuracy who would pay back their loans, there would be a racial disparity in the result, but the cause of it is not the fault of the algorithm. And you can't fix it there because that's not where the problem is. Suppose you have a group of 800 white middle managers, 100 white sales clerks and 100 black sales clerks. Clearly the algorithm is going to have a racial disparity in outcome if the middle managers are at 500% of the cutoff and the sales clerks are right on the line, because it will accept all of the middle managers and half of the sales clerks which means it will accept >94% of the white applicants and 50% of the black applicants. But the source of the disparity is that black people are underrepresented as middle managers and overrepresented as sales clerks. The algorithm is just telling you that. It can't change it. And inserting bias into the algorithm to "balance" the outcome doesn't actually do that, all you're doing is creating bias against white sales clerks who had previously been on the same footing as black sales clerks. The white middle managers will be unaffected because they're sufficiently far above the cutoff that the change doesn't affect them, even though that they're the source of the imbalance.

mknocker 10y ago

The first thing that came to my mind was... what about advertising on the web ? This is an area where machine learning and algorithmic decision-making use discrimination to either show or not an ad to a particular user. There are millions decision made everyday. Under that regulation, could you ask for the reason an ad was shown to you ?

matt4077 10y ago
Nope, because advertisement doesn't "significantly affect" you. At least that's how it's meant and how it will be understood. This is about credit, employment, insurance, medical care etc.
- Symmetry 10y ago
  Medical care?! Oh God. Well, I suppose the AMA will keep the US from embracing machine learning so it's not like this is actually a chance for the US to reverse its lag in mortality statistics.
- csn 10y ago
  Although just like the things you list, I can think of a plethora of ways with which advertising can literally, indirectly change my life by convincing me to purchase a product or service.
denzil_correa 10y ago
There is some research which shows discrimination in online advertisements. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2208240 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2208240 http://queue.acm.org/detail.cfm?id=2460278 http://queue.acm.org/detail.cfm?id=2460278
Kristine1975 10y ago
Just put a link under the ad "why am I seeing this ad?" leading to a page with the explanation.

zoner 10y ago

Finally I get to know why my mortgage was declined :)

phkahler 10y ago
>> Finally I get to know why my mortgage was declined :) My first attempt at one was declined due to "problems with my credit report" or some such. I happened to be working at a place where they did credit checks on customers, so I asked someone to pull my credit report. They had all my stuff, but had mixed in a bunch of information from someone else with the same name - different age, there were loans on there from when I was 3. I called the reporting company to complain and they kept getting caught up in "how did you get your report?" as if I had done something wrong in just having access to their incorrect information. Perhaps having someone pull the info instead of the "consumer" getting it through proper channels was against some rule - this was 1995. I think the same is often true today, companies want to collect data on you but don't really want you to know what they've got or how they use it.
- dublinben 10y ago
  This is exactly why you are legally entitled to receive a copy of your credit report every year. You can challenge anything on it, and have incorrect information removed.
  - phkahler 10y ago
    That had to be made explicit in law precisely because the companies don't want you to have it.
  - jerf 10y ago
    When mentioning this, I find it helpful to include this page: https://www.ftc.gov/faq/consumer-protection/get-my-free-credit-report https://www.ftc.gov/faq/consumer-protection/get-my-free-cred... If you are one of the people just finding out about this today, and you Google "free credit report", you come up with a loooot of bad, scammy links. This is how you get your legally-mandated, really free, annual credit report, starting from an ftc.gov address so you know it's really the right one.

dharma1 10y ago

I think it will be very hard to implement/enforce this regulation. I attended the London AI summit last week, and they had a speaker from a German lender called Kreditech. There was a question of black box credit scoring, and the speaker made a fair point - their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? You can list the sources of the data, or offer manual review of the application if rejected (which is something they do) - but it would be very hard to show exact causal reasoning patterns in the automatic scoring.

dec0dedab0de 10y ago
why not just have an optional pdf download with the 20000 vectors, and how much they were weighted.
GFK_of_xmaspast 10y ago
Sounds like a limitation of their business model.
kaijush 10y ago
>> their models have 20,000 vectors in determining credit worthiness. How would you begin to break that down to something explainable? Well, somehow they decided that their 20k-parameter model is accurate. They should at least be able to explain why they took that decision, even if the model itself is too complex.
- JoeAltmaier 10y ago
  Because the training set was covered with x% accuracy. Ok, how does knowing that help?
  - kaijush 10y ago
    OK, so we're assuming supervised learning. In that case, how were the features of the training set chosen? And how was the training set labelled? Most of the time, outside of semi-supervised learning, those things are not in the data, someone has to (painstakingly) decide them. That can very well be explained I believe. More generally, what I mean to say is that you can learn any number of models from the same data, so to a great extent you choose what your model learns by manipulating your algorithm's (hyper)parameters, the training set's features, choosing what goes into the training set itself etc etc. That process leaves room for a lot of regulatory oversight. - in principle. In practice I expect since we're talking about people lending money or selling insurance we're only ever going to get those details out of their cold dead hands, and only with considerable effort at that.

Houshalter 10y ago

Is my reading of this law correct? It sounds like it outlaws all use of algorithms to evaluate people. The right to explanation is just for the rare exceptions where a "member state" authorizes it. But the legalese is difficult to parse, and no one else seems to get this impression. I think this is really bad. That's the majority of uses of machine learning. It also has a lot of economic value to predict things like how likely someone is to pay back a loan, or even who is a spammer. Most importantly, most these applications will go back to human judgement. And humans are far worse. Human predictions about candidates are really bad. We are incredibly biased by things like race, gender, and especially attractiveness. Unattractive people get twice the sentences of attractive people. Not to mention random stuff, like people being judged more harshly when the judge is hungry before lunch. They also rarely give explanations for their decisions, if they are even aware of the true reasons for them. Going back to humans is a huge step backwards and will hurt a lot more people than it helps. I think the same regulations that apply to algorithms, should apply to humans. That would show the absurdity of these laws. But humans are algorithms after all. And particularly bad ones (for this purpose anyway.)

germanier 10y ago
> It sounds like it outlaws all use of algorithms to evaluate people. The right to explanation is just for the rare exceptions where a "member state" authorizes it. Who says that this is rare? That's a pretty standard way EU regulations are written to mean "Action A can only be allowed if conditions B are met". If the regulation is approved, member states take a look and change their laws accordingly. They could outlaw algorithmic decision making if they wanted to (but they wouldn't need that regulation to do that). Much more likely is that they impose those conditions on the companies that use such algorithms just like they already impose a lot of conditions (such as "ethnicity is not allowed to be a direct input variable" currently is).
- ddebernardy 10y ago
  > If the regulation is approved, member states take a look and change their laws accordingly. Small nitpick id I may: you seem to be confusing Regulations with Directives, which must get transcribed within a certain timeframe in national law. Regulations apply directly - no transcription needed. Also: either way, EU law has primacy over national law.

deleted 10y ago

[deleted]

InTheArena 10y ago

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This is really needed for a number of different reasons. The biggest is that algorithms are increasingly running our world. Even outside of mortgage, have you ever had to fork over the right to pull your credit score for either rent or (in some limited cases) to apply for a job? Not being able to understand: a) Why you were rejected. b) Why you are paying a higher interest rate. c) What you can do to fix that? Housing and employment is fundamentally a question of human rights, not just banking profitability or academic algorithm improvements. Part of the reason why Credit Scoring (which is intrinsically algorithmic decision-making) is because it replaced the "old boys" network that used to dominate. In this world you went to your banker, and the decision to extend credit was dominated by your personal relationship with a banker who might not be the same ethnicity, religion, nationality, etc as you would dominate your ability to purchase housing. The credit score democratized financial access. It can still be used to discriminate. There have been a number of studies that have shown that simply being from a white community tends to increase the number of other people you can lean on in a emergency, and hence make you less likely to default on a loan. From a pure predictability point of view, a bank is at a lower risk of default lending to someone from that community, but that in turn denies financial access to people not in that community, continuing the trend. A big problem in this kind of financial model is that it's relatively easy to accidentally or deliberately find other variables that mirror "protected" vectors. A simplistic example is zip code, where the zip code reflects a area that is predominately ethnic minorities. So it's not cut and dried. It's my PoV that It's not predictability versus red-tape, and the people trying to do unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism.

deleted 10y ago
[deleted]
sergioisidoro 10y ago
I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)

gersh 10y ago

Can't you just say the decision was based on a random forest trained on the following parameters with historical data. You could get more specific on how the model was trained or even show model if you have to. I'm sure you can create a small-print tree diagram. In terms of human review, does this require more than a spot check? I can imagine look over the p-value and possibly spot check the input data, and then approve it.

Symmetry 10y ago

I wonder machine learning could just handle this the way that humans do. Human decisions processes are of course extremely opaque even if we're good at coming up with post-hoc justifications for our reasoning as needed. "You're income is too low" "But his income is also low and he got a loan!" "Well he has a better credit score." etc It would obviously be impossible to give a complete and accurate explanation of a machine learning decision process the same way it would be impossible for a human learning decision process, but I wonder if it would actually be difficult to come up with particularly salient distinguishing features in general or between certain cases the same way that people do?

Kenji 10y ago

Good luck arguing the 'why' when you have a trained neural network and all you have is the network and weights. On a more serious note, I love transparency but again, this is an overeager regulation (not surprising from the EU). You almost never get the true reasons for being rejected, be it in an interview or for a credit, etc. You have to figure that out yourself from the often very vague rejection letter. Our minds are basically running algorithms to which we have even less insight than our computer algorithms. Therefore, this law only hampers the flourishing of the economy while providing no value whatsoever. Another law that 'solves' a non-issue, brought into being by overpaid career politicians.

sparkie 10y ago
If all you have is a network and weights, you have a black box. While it might be magic and make the right decision most of the time, we need to look in and see why those decisions are being made, somehow. This generally basically means seeing how the net was trained, and its initial conditions. If you threw away your training methodology and data set, why should anyone trust that your algorithm can make suitable decisions? Are we to assume that nobody who writes AIs which are making influential decisions has vested interests? If you look for example at common law; we have a system where decisions by judges are binding, and future equivalent cases are required to uphold the previous decisions. The only way this system can work of course, is if someone is keeping a record of the previous decisions. The alternative is we'd have a system of judgement based on heresay, and we can have about as much confidence in it as asking a random person on the street to make the decision. The technical challenge is really about storage and retreival. If we know that discarding training data makes our neural networks "unaccountable for their decisions", then our technical requirement must be that we store the training data in its entirity, such that we can look back and maybe glimpse at why a neural network may behave as it has. For example, I might create an AI which is used to decide whether to give someone a mortgage. I could have millions of samples as training data, but I might chose to sort the training data by race, and begin training a network initially so that non-white people who have been rejected mortgages initially overtrain the network to correlate race with rejection, then use a limited sample of white man, mostly accepted mortgage requests to train it the other way. Of course this is extreme, and an expert who looked at such data set would quickly notice that the AI is unfit for purpose, and blatantly racist. But he needs the training set to even have a chance of concluding that. Without it, he has effectively random numbers which tell him next to nothing. These laws aren't meant to stifle innovation or economic benefits, but only to ensure that fair treatment is practiced in their development. As far as I see it, if you have a neural network, a sound justification of its design and the methodology you used, combined with a complete data set and training set which can be analysed for biases, then there's no reason these regulations should get in your way.

deleted 10y ago

[deleted]

bwgoodman 10y ago

hi, I'm one of the co-authors. This feedback is awesome. I have a longer version in the works. if you're interested please feel free to email me, bryce dot goodman at stx dot ox dot ac dot uk. Thanks! Bryce

zxcvvcxz 10y ago

> We argue that while this law will pose large challenges for industry, it highlights opportunities for machine learning researchers to take the lead in designing algorithms and evaluation frameworks which avoid discrimination. This sounds like a very slippery slope. Let me get this straight, what the EU wants to do here. If I make a business that decides how to, say, make loans to people, and I have software that analyzes how they filled out an application or took a written/logical test or something I may have to: 1) Disclose the findings and methodology of my proprietary algorithm 2) Potentially undo decisions that are deemed discriminatory 3) Thereby potentially be forced to choose who I do and do not do business with Some regulations are good, they help us avoid things like moral hazards. Some regulations, however, act more like power grabs that take away freedom from individuals and businesses. I'm sorry to say but this sort of thing sounds like the latter. There are some uncomfortable facts about people and society that algorithms will uncover, and that people will try using to avoid making mistakes. Here's a poignant example, albeit with much less sophistication: http://www.nytimes.com/2015/10/31/nyregion/hudson-city-bank-settlement.html http://www.nytimes.com/2015/10/31/nyregion/hudson-city-bank-... > Instead, some officials say, some banks have quietly institutionalized bias in their operations, deliberately placing branches, brokers and mortgage services outside minority communities, even as other banks find and serve borrowers in those neighbourhoods. Alright sure. I think we all know it's common sense to not open a jewellery store in inner city Chicago. The local clientele would not be able to afford such products and services, and that's enough of an argument, it's probably business school 101. Just like how you're not going to find a good market for a food delivery startup in Middletown, USA. The economics probably don't work. EDIT - and if you think these businesses are wrong, and they should be in these communities, let their competitors eat their lunch! But it's viewed as discriminatory. Back to the paper, it's discussing essentially the same thing: it's the natural evolution of "don't start a jewellery store in inner city Chicago" turning into "don't start a jewellery store in neighbourhood where access to urban center is less than X, available nearby 3-lane highways is less than Y, percent of women from 18-29 is less than Z, proportion of people from culture W who are in the bottom quartile of people who buy jewellery is more than V, ...". The solution here isn't to regulate and force people to open jewellery stores in inner city Chicago. The solution, if we really want to bring diamonds to Englewood, is to drastically reduce the poverty and crime rates so that businesses will want to sell there (https://en.wikipedia.org/wiki/Englewood,_Chicago#Socioeconomics https://en.wikipedia.org/wiki/Englewood,_Chicago#Socioeconom...). And if you think having access to a local business is a "right", then you must concede that such a public good should be provided by a central government of some type. And then you have to ask yourself, do you want the government managing your bank loans, or would you rather have private companies compete on prices and service? If you want the latter, then you need to accept that certain things are just going to arise (or not arise) naturally.