10 ms·

Will GDPR Make Machine Learning Illegal?

111 points by Vaslo 9y ago

stochastic_monk 9y ago
No. This is an absurd victim roleplay. People’s rights to control what happens to their information is not going to make machine learning illegal. It’ll just give all users a little more privacy and make selling their data a little less lucrative. Besides, there’s been plenty of work into improving interpretibility. Stop pushing the myth that we don’t understand anything about neural networks and they’re the biggest, blackest box since Schwarzschild.
- gnud 9y ago
  GDPR might make certain uses of machine learning more difficult. A hypothetical insurance company that uses ML to decide who gets offered which rate, might run into trouble when a customer asked for "meaningful information" about the logic behind the decision. But for the same insurance company, using ML to flag possible fraud cases, and then having a human review them, seems not to pose a problem. All decisions were made by humans, the machines just showed where a decision was needed. Of course, IANAL.
  - disgruntledphd2 9y ago
    Insurance companies are already prevented by law from using methods which cannot be explained, so they will be fine. I actually think that most ML will be fine, as the outcomes are trivial. The particular set of results that Google shows you on Google.com are not a decision about you, and I can't see those needing to be interpreted. Anything more consequential (i.e. job hiring) probably will, but also has laws that prevent the use of indiscriminate data already. So, I don't see this an issue.
    - maksimum 9y ago
      > The particular set of results that Google shows you on Google.com are not a decision about you I think that's easily demonstrated as not true: when I search for something like `luigi` in a regular window, I get the github link 1st. When I search for it in an anonymous window the github link is 3rd. Try it for yourself with a technical term that also has a common non-technical meaning. I believe that search personalization is useful, but it may be a privacy issue.
      - disgruntledphd2 9y ago
        See I get you, but I disagree. My example is python, whereby biologists and such-like people get snakes, people hanging out here get the programming language and so on. I personally believe (with no evidence) that the number of potential SERPs is much lower than the number of users, as I reckon they use some kind of dimension reduction technique and pick vectors of results "close" to yours. But I completely accept that I could be entirely wrong on this (i still think such a decision isn't consequential enough to trigger any GDPR provisions, but again I could also be wrong on that :) )
    - gnud 9y ago
      The insurance example was just the first thing I could think of. Replace with phone company, utilities company, what have you.
  - bitL 9y ago
    Would publishing a deep learning model on GitHub satisfy that requirement? You want to know how it makes decisions? Here you go!
hn_throwaway_99 9y ago
What's that saying about "The answer to any headline that is posed as a yes-or-no question is 'no'."
- simcop2387 9y ago
  https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- mitchty 9y ago
  Betteridge's law of headlines I believe. https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- deleted 9y ago
  [deleted]
reificator 9y ago
As per tradition: No. https://news.ycombinator.com/item?id=9232419 https://news.ycombinator.com/item?id=9232419
matthewmacleod 9y ago
No. It may make it illegal, via the aggregate "right to explanation", to make decisions about people solely on the basis of machine learning models. That is, "we fed some data into a computer and it said 'no', so too bad" is not going to be a valid explanation. That sounds like a fucking amazing outcome to me.
- John_KZ 9y ago
  But unfortunately it's very easy to find a way around it: Make all decisions based on ML models, and only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. It's definitely a whole lot better than not having that law, but we're not quite there yet.
  - protomyth 9y ago
    > only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. That doesn't sound exactly legal as the actual reason had to exist at the time of the decision and an after the fact reason would not be the truth. Don't the ML models log why they made decisions in the first place?
    - narrator 9y ago
      The log is we multiplied an 80x25x3 matrix of your planet's photograph by 30 40x40 matrices of other numbers and then fed it through a neural net which consisted of adding and multiplying a few hundred other numbers and we got this number, which when run through a tanh function, classified your planet as uninhabitable space garbage. That's why the Vogon constructor fleet arrived and cleared it to make room for an intergalactic highway.
    - dogecoinbase 9y ago
      The point of interpretability in ML is that the "reasons" that model arrived at a decision/classification/etc are incomprehensible (and, given e.g. the existence of adversarial examples in image classification, this should be expected -- no reasonable answer exists for "why" the model thought this static was a cat).
norachichi 9y ago
My name is Nora Chichi, i want to tell the world about how i got cured from HIV Disease by Dr. Ozala. i was Infected with HIV Disease for so many years and i was tired of life because of the sufferings i was going through, so one faithful day a friend of mine gave me a contact of a herbal doctor who cured her cousin from HIV Disease and immediately i contacted the Doctor on his email address: (drozalahomeremedy@gmail.com) and his whatsapp: +2349074008745, and immediately the Doctor sent me a herbal medicine and told me to take the medicine only at bed time with is at night and i did and immediately the HIV Virus DISAPPEARED and i was cured from HIV Disease, the name of this Doctor is Dr Ozala.. so if you are infected with HIV Disease or any other dangerous and uncurable Disease then Dr. Ozala is the right man to be contacted for the cure. so kindly contact him now on his email: (drozalahomeremedy@gmail.com) or His Whatsapp number: +2349074008745… once again, Dr. Ozala thank you very much for what you did for me am very grateful… He can also cure Hepatitis Virus and Herpes virus..
akshatpradhan 9y ago
It's in your best interest to reduce your risk by going through a De-identification process for data collected: >De-identification is adopted as one of the main approaches of data privacy protection. It is commonly used in fields of communications, multimedia, biometrics, big data, cloud computing, data mining, internet, social networks and audio–video surveillance. https://en.wikipedia.org/wiki/De-identification https://en.wikipedia.org/wiki/De-identification
polskibus 9y ago
I have a related question - can EU citizen under GDPR demand that a company "unlearns" all of their machine learning models from his/her data? I'm not asking if this is technically feasible (you could do that by removing old model, removing user data from training set and then reruning learning). I'm asking whether GDPR will give citizens power to demadn that.
- amarkov 9y ago
  GDPR only requires erasure of personal data on request, where "personal data" is defined as information relating to some identified or identifiable person. Machine learning models trained on personal data aren't going to be personal data themselves.
  - yxhuvud 9y ago
    Well, unless they can be used to identify the person.
    - emilfihlman 9y ago
      But we are (reminds me of black mirror...) entitled to having knowledge and memories. Corporate entities will start to claim that you cannot ask us to remove our memories (for example, identifying people in images).
  - Asdfbla 9y ago
    Machine learning models leaking private information is a real problem though, unless mechanisms to ensure something like differential privacy are applied. Those approaches may still be too theoretical and hard to reason about for many machine learning practitioners. Will be interesting how that plays out.
gaius 9y ago
Serious question: is there a web based application of ML that isn’t about exploiting or deceiving customers/users? ML has its legit uses sure, but it always seems to be used to outwit people in practice
- amarkov 9y ago
  Well, I mean, what counts as exploitation? If I learn a lot about what shoes people like in my job as a shoe salesman, and I use that experience when new people come in to make recommendations, I think everyone would call that just good customer service. But when Amazon sets up an ML model to do the same thing, many people argue they're exploiting your personal data for the sake of profit. Off the top of my head, I can't think of a widespread web-based ML application that some people don't consider exploitative. But is that a property of the ML or the people?
  - gaius 9y ago
    I’d argue that “people who bought X also bought Y” is too simplistic to be considered ML. But say, charging a little extra for X to people who previously bought Y would be exploitative.
- cimmanom 9y ago
  Sure. How about the basic e-commerce use case of "based on your interest in product X, you may also be interested in product Y"?
- stupidcar 9y ago
  Catching fake / spamming / abusive users on social media. People are (rightfully) upset by the amount of spam and harassment happening on platforms like Twitter. Machine learning is a good way to tackle this problem at scale, in an automated way, by training models to spot and auto-moderate bad posts. Under GDPR, anyone can demand an explanation for why they were marked as spam / abusive. This will allow trolls and spammers to improve their workarounds, by demanding an explanation for why their posts were blocked, then tailoring their later posts to avoid these signifiers. It will also let them impose a tax on platforms that try to block them by flooding them with demands for explanations under GDPR.
singron 9y ago
Regardless of the whatever the GDPR actually says and even if ML is a complete black box, I would be very happy at least knowing my input features to ML algorithms. e.g. If I'm denied a loan, I want to learn the various "facts" about me that led to the ML coming up with the denial. Then I can fact-check it (e.g. "actually I don't already have $X of debt") and come up with an appeal.
- maxxxxx 9y ago
  Agreed. We are building up this level of faceless bureaucracy the more automated algorithms with multiple inputs we are using. I know someone who had to deal with identity theft and it's a sheer nightmare to figure out why you get denied a rental car or airplane boarding. The people facing the customer know that the person is blocked but don't know why and have no way to figure out why.
ThePhysicist 9y ago
Personally, I think many people and organizations currently exaggerate the possible changes and risks (for companies) brought on by the GDPR, often with the goal to instill fear (and generate business) or get press coverage. The relevant article (https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/) clearly states that automated decision making is allowed under the GDPR, it just gives the data subject the right to demand a manual assessment of that decision. In practice, this means that you're absolutely free to use any kind of automated algorithm (AI-based or not) to make decisions on individuals, but if people complain and demand an explanation you need to review the decision manually. Addendum: If you want to learn more about the motivations behind article 22 as well as articles 13 & 14 (that concern the transparency requirements), I highly recommend the official document of the Article 29 working party: http://ec.europa.eu/newsroom/article29/item-detail.cfm?item_id=612053 http://ec.europa.eu/newsroom/article29/item-detail.cfm?item_...
- dcd0101 9y ago
  It sounds like they're suggesting that, assuming those rights exist in these situations (they claim there's ambiguity there), that there are big questions around what qualifies as an acceptable explanation. Did I misunderstand?
  - ThePhysicist 9y ago
    Yes, but reading the official guidelines should clarify this ([1], p. 14 ff.): > The controller should find simple ways to tell the data subject about the rationale behind, or the criteria relied on in reaching the decision without necessarily always attempting a complex explanation of the algorithms used or disclosure of the full algorithm. The information provided should, however, be meaningful to the data subject. There's also a detailed example in the document that should make it clearer what kind of explanation is required (and what is not required). Again, personally I understand that it's not clear what exactly will be required here, but titling an article "Will GDPR Make Machine Learning Illegal?" is just an attempt to garner attention by instilling (unfounded) fear. [1]: http://ec.europa.eu/newsroom/just/document.cfm?doc_id=47963 http://ec.europa.eu/newsroom/just/document.cfm?doc_id=47963
tzs 9y ago
In the case of things like being rejected by a machine learning algorithm for a mortgage, it should be pretty easy to automate coming up with an explanation for the applicant. Run the application again with $5k higher income. Keep raising income by $5k until you find an income level that results in approval. If that income level is reasonable for people in general who were approved for similar mortgage amounts and whose other inputs are in the same ballpark as the rejected application, tell the applicant the rejection was for income too low. You can do a similar thing with other parameters, such as credit rating, length of employment, and whatever else is input to your algorithm that is under the applicant's control. In some cases you might not be able to find a good single parameter tweak for approval and will have to resort to combinations. There will usually be many different ways then to tweak for approval. Use comparisons to similar applications that were approved to pick a reasonable combination of tweaks to base the rejection explanation on.
- paulddraper 9y ago
  The only reason I could see this wouldn't work is your explanation had to be "good" one. E.g. if you lived 10 miles to the west, had the same income, had a mortgage before 2005, and didn't apply for credit between Sept 2010 and June 2011, then you would have been accepted. (There are other situations that would have resulted in acceptance; this is just one.)
petercooper 9y ago
Rather than explaining how the machine learning system works as an "explanation", perhaps auditing could be added to machine learning algorithms so that you very much could produce a very long but accurate description of the process, a la pages full of "X was compared to Y, X was larger, and thus we move on to step 261". A bit like disassembling machine code. Of course, in machine learning, the datasets backing up the comparisons could not be shared as they contain variations of confidential and personal data, but you might still end up with a legally tolerable record of the algorithmic steps involved in the decision making process even if they're not useful to see.
- piqalq 9y ago
  I see what you’re getting at, but this would be a very lengthy and arduous process. Not to mention, many ML and DL algorithms are incredibly mathematically complex that describing them with literal step-by-step detail sounds like hell.
  - petercooper 9y ago
    It would be if you had to be involved, but I'm suggesting algorithms could have some sort of instrumentation so such "explanations" could be automatically generated and thrown into a data warehouse for possible future use. (This is all a cynical attempt to meet legal requirements rather than anything actually useful for the user, of course.)
JepZ 9y ago
Nobody else has a problem with the mix up of 'Machine Learning' and 'Deep Learning' here? Last time I heard of those terms, 'Machine Learning' was a much broader category than 'Deep Learning'. Its like someone says 'Next years Porsches will be illegal' and the next person calls 'Next year cars will be illegal'?!?
nmca 9y ago
For those interested in the cutting edge of interpretability, check out https://github.com/marcotcr/anchor https://github.com/marcotcr/anchor I'm pretty sure this sort of approach will be how most businesses handle the problem.
emilfihlman 9y ago
I find it extremely concerning that people, especially on this platform, don't care about what the law says and choose to just hand wave it as "it doesn't apply to us / they don't enforce it like that". The letter of the law is even more important than the spirit of it because it allows governments, corporations and other entities to simply say "this is what the law says" and because you are small you don't have the money/power to challenge this rendition if the spirit of the law disagrees with them. Not to mention that laws are _laws_, they are not meant to be really interpreted (it is a byproduct of having to do so to have a functioning society). It's an extremely slippery slope to not care what the law says. If we don't we'll have some dystopian laws that allow for anything quite soon.

111 points by Vaslo 9y ago

No. This is an absurd victim roleplay. People’s rights to control what happens to their information is not going to make machine learning illegal. It’ll just give all users a little more privacy and make selling their data a little less lucrative. Besides, there’s been plenty of work into improving interpretibility. Stop pushing the myth that we don’t understand anything about neural networks and they’re the biggest, blackest box since Schwarzschild.

gnud 9y ago
GDPR might make certain uses of machine learning more difficult. A hypothetical insurance company that uses ML to decide who gets offered which rate, might run into trouble when a customer asked for "meaningful information" about the logic behind the decision. But for the same insurance company, using ML to flag possible fraud cases, and then having a human review them, seems not to pose a problem. All decisions were made by humans, the machines just showed where a decision was needed. Of course, IANAL.
- disgruntledphd2 9y ago
  Insurance companies are already prevented by law from using methods which cannot be explained, so they will be fine. I actually think that most ML will be fine, as the outcomes are trivial. The particular set of results that Google shows you on Google.com are not a decision about you, and I can't see those needing to be interpreted. Anything more consequential (i.e. job hiring) probably will, but also has laws that prevent the use of indiscriminate data already. So, I don't see this an issue.
  - maksimum 9y ago
    > The particular set of results that Google shows you on Google.com are not a decision about you I think that's easily demonstrated as not true: when I search for something like `luigi` in a regular window, I get the github link 1st. When I search for it in an anonymous window the github link is 3rd. Try it for yourself with a technical term that also has a common non-technical meaning. I believe that search personalization is useful, but it may be a privacy issue.
    - disgruntledphd2 9y ago
      See I get you, but I disagree. My example is python, whereby biologists and such-like people get snakes, people hanging out here get the programming language and so on. I personally believe (with no evidence) that the number of potential SERPs is much lower than the number of users, as I reckon they use some kind of dimension reduction technique and pick vectors of results "close" to yours. But I completely accept that I could be entirely wrong on this (i still think such a decision isn't consequential enough to trigger any GDPR provisions, but again I could also be wrong on that :) )
  - gnud 9y ago
    The insurance example was just the first thing I could think of. Replace with phone company, utilities company, what have you.
- bitL 9y ago
  Would publishing a deep learning model on GitHub satisfy that requirement? You want to know how it makes decisions? Here you go!

hn_throwaway_99 9y ago

What's that saying about "The answer to any headline that is posed as a yes-or-no question is 'no'."

simcop2387 9y ago
https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
mitchty 9y ago
Betteridge's law of headlines I believe. https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
deleted 9y ago
[deleted]

reificator 9y ago

As per tradition: No. https://news.ycombinator.com/item?id=9232419 https://news.ycombinator.com/item?id=9232419

matthewmacleod 9y ago

No. It may make it illegal, via the aggregate "right to explanation", to make decisions about people solely on the basis of machine learning models. That is, "we fed some data into a computer and it said 'no', so too bad" is not going to be a valid explanation. That sounds like a fucking amazing outcome to me.

John_KZ 9y ago
But unfortunately it's very easy to find a way around it: Make all decisions based on ML models, and only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. It's definitely a whole lot better than not having that law, but we're not quite there yet.
- protomyth 9y ago
  > only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. That doesn't sound exactly legal as the actual reason had to exist at the time of the decision and an after the fact reason would not be the truth. Don't the ML models log why they made decisions in the first place?
  - narrator 9y ago
    The log is we multiplied an 80x25x3 matrix of your planet's photograph by 30 40x40 matrices of other numbers and then fed it through a neural net which consisted of adding and multiplying a few hundred other numbers and we got this number, which when run through a tanh function, classified your planet as uninhabitable space garbage. That's why the Vogon constructor fleet arrived and cleared it to make room for an intergalactic highway.
  - dogecoinbase 9y ago
    The point of interpretability in ML is that the "reasons" that model arrived at a decision/classification/etc are incomprehensible (and, given e.g. the existence of adversarial examples in image classification, this should be expected -- no reasonable answer exists for "why" the model thought this static was a cat).

norachichi 9y ago

My name is Nora Chichi, i want to tell the world about how i got cured from HIV Disease by Dr. Ozala. i was Infected with HIV Disease for so many years and i was tired of life because of the sufferings i was going through, so one faithful day a friend of mine gave me a contact of a herbal doctor who cured her cousin from HIV Disease and immediately i contacted the Doctor on his email address: (drozalahomeremedy@gmail.com) and his whatsapp: +2349074008745, and immediately the Doctor sent me a herbal medicine and told me to take the medicine only at bed time with is at night and i did and immediately the HIV Virus DISAPPEARED and i was cured from HIV Disease, the name of this Doctor is Dr Ozala.. so if you are infected with HIV Disease or any other dangerous and uncurable Disease then Dr. Ozala is the right man to be contacted for the cure. so kindly contact him now on his email: (drozalahomeremedy@gmail.com) or His Whatsapp number: +2349074008745… once again, Dr. Ozala thank you very much for what you did for me am very grateful… He can also cure Hepatitis Virus and Herpes virus..

akshatpradhan 9y ago

It's in your best interest to reduce your risk by going through a De-identification process for data collected: >De-identification is adopted as one of the main approaches of data privacy protection. It is commonly used in fields of communications, multimedia, biometrics, big data, cloud computing, data mining, internet, social networks and audio–video surveillance. https://en.wikipedia.org/wiki/De-identification https://en.wikipedia.org/wiki/De-identification

polskibus 9y ago

I have a related question - can EU citizen under GDPR demand that a company "unlearns" all of their machine learning models from his/her data? I'm not asking if this is technically feasible (you could do that by removing old model, removing user data from training set and then reruning learning). I'm asking whether GDPR will give citizens power to demadn that.

amarkov 9y ago
GDPR only requires erasure of personal data on request, where "personal data" is defined as information relating to some identified or identifiable person. Machine learning models trained on personal data aren't going to be personal data themselves.
- yxhuvud 9y ago
  Well, unless they can be used to identify the person.
  - emilfihlman 9y ago
    But we are (reminds me of black mirror...) entitled to having knowledge and memories. Corporate entities will start to claim that you cannot ask us to remove our memories (for example, identifying people in images).
- Asdfbla 9y ago
  Machine learning models leaking private information is a real problem though, unless mechanisms to ensure something like differential privacy are applied. Those approaches may still be too theoretical and hard to reason about for many machine learning practitioners. Will be interesting how that plays out.

gaius 9y ago

Serious question: is there a web based application of ML that isn’t about exploiting or deceiving customers/users? ML has its legit uses sure, but it always seems to be used to outwit people in practice

amarkov 9y ago
Well, I mean, what counts as exploitation? If I learn a lot about what shoes people like in my job as a shoe salesman, and I use that experience when new people come in to make recommendations, I think everyone would call that just good customer service. But when Amazon sets up an ML model to do the same thing, many people argue they're exploiting your personal data for the sake of profit. Off the top of my head, I can't think of a widespread web-based ML application that some people don't consider exploitative. But is that a property of the ML or the people?
- gaius 9y ago
  I’d argue that “people who bought X also bought Y” is too simplistic to be considered ML. But say, charging a little extra for X to people who previously bought Y would be exploitative.
cimmanom 9y ago
Sure. How about the basic e-commerce use case of "based on your interest in product X, you may also be interested in product Y"?
stupidcar 9y ago
Catching fake / spamming / abusive users on social media. People are (rightfully) upset by the amount of spam and harassment happening on platforms like Twitter. Machine learning is a good way to tackle this problem at scale, in an automated way, by training models to spot and auto-moderate bad posts. Under GDPR, anyone can demand an explanation for why they were marked as spam / abusive. This will allow trolls and spammers to improve their workarounds, by demanding an explanation for why their posts were blocked, then tailoring their later posts to avoid these signifiers. It will also let them impose a tax on platforms that try to block them by flooding them with demands for explanations under GDPR.

singron 9y ago

Regardless of the whatever the GDPR actually says and even if ML is a complete black box, I would be very happy at least knowing my input features to ML algorithms. e.g. If I'm denied a loan, I want to learn the various "facts" about me that led to the ML coming up with the denial. Then I can fact-check it (e.g. "actually I don't already have $X of debt") and come up with an appeal.

maxxxxx 9y ago
Agreed. We are building up this level of faceless bureaucracy the more automated algorithms with multiple inputs we are using. I know someone who had to deal with identity theft and it's a sheer nightmare to figure out why you get denied a rental car or airplane boarding. The people facing the customer know that the person is blocked but don't know why and have no way to figure out why.

ThePhysicist 9y ago

Personally, I think many people and organizations currently exaggerate the possible changes and risks (for companies) brought on by the GDPR, often with the goal to instill fear (and generate business) or get press coverage. The relevant article (https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/) clearly states that automated decision making is allowed under the GDPR, it just gives the data subject the right to demand a manual assessment of that decision. In practice, this means that you're absolutely free to use any kind of automated algorithm (AI-based or not) to make decisions on individuals, but if people complain and demand an explanation you need to review the decision manually. Addendum: If you want to learn more about the motivations behind article 22 as well as articles 13 & 14 (that concern the transparency requirements), I highly recommend the official document of the Article 29 working party: http://ec.europa.eu/newsroom/article29/item-detail.cfm?item_id=612053 http://ec.europa.eu/newsroom/article29/item-detail.cfm?item_...

dcd0101 9y ago
It sounds like they're suggesting that, assuming those rights exist in these situations (they claim there's ambiguity there), that there are big questions around what qualifies as an acceptable explanation. Did I misunderstand?
- ThePhysicist 9y ago
  Yes, but reading the official guidelines should clarify this ([1], p. 14 ff.): > The controller should find simple ways to tell the data subject about the rationale behind, or the criteria relied on in reaching the decision without necessarily always attempting a complex explanation of the algorithms used or disclosure of the full algorithm. The information provided should, however, be meaningful to the data subject. There's also a detailed example in the document that should make it clearer what kind of explanation is required (and what is not required). Again, personally I understand that it's not clear what exactly will be required here, but titling an article "Will GDPR Make Machine Learning Illegal?" is just an attempt to garner attention by instilling (unfounded) fear. [1]: http://ec.europa.eu/newsroom/just/document.cfm?doc_id=47963 http://ec.europa.eu/newsroom/just/document.cfm?doc_id=47963

tzs 9y ago

In the case of things like being rejected by a machine learning algorithm for a mortgage, it should be pretty easy to automate coming up with an explanation for the applicant. Run the application again with $5k higher income. Keep raising income by $5k until you find an income level that results in approval. If that income level is reasonable for people in general who were approved for similar mortgage amounts and whose other inputs are in the same ballpark as the rejected application, tell the applicant the rejection was for income too low. You can do a similar thing with other parameters, such as credit rating, length of employment, and whatever else is input to your algorithm that is under the applicant's control. In some cases you might not be able to find a good single parameter tweak for approval and will have to resort to combinations. There will usually be many different ways then to tweak for approval. Use comparisons to similar applications that were approved to pick a reasonable combination of tweaks to base the rejection explanation on.

paulddraper 9y ago
The only reason I could see this wouldn't work is your explanation had to be "good" one. E.g. if you lived 10 miles to the west, had the same income, had a mortgage before 2005, and didn't apply for credit between Sept 2010 and June 2011, then you would have been accepted. (There are other situations that would have resulted in acceptance; this is just one.)

petercooper 9y ago

Rather than explaining how the machine learning system works as an "explanation", perhaps auditing could be added to machine learning algorithms so that you very much could produce a very long but accurate description of the process, a la pages full of "X was compared to Y, X was larger, and thus we move on to step 261". A bit like disassembling machine code. Of course, in machine learning, the datasets backing up the comparisons could not be shared as they contain variations of confidential and personal data, but you might still end up with a legally tolerable record of the algorithmic steps involved in the decision making process even if they're not useful to see.

piqalq 9y ago
I see what you’re getting at, but this would be a very lengthy and arduous process. Not to mention, many ML and DL algorithms are incredibly mathematically complex that describing them with literal step-by-step detail sounds like hell.
- petercooper 9y ago
  It would be if you had to be involved, but I'm suggesting algorithms could have some sort of instrumentation so such "explanations" could be automatically generated and thrown into a data warehouse for possible future use. (This is all a cynical attempt to meet legal requirements rather than anything actually useful for the user, of course.)

JepZ 9y ago

Nobody else has a problem with the mix up of 'Machine Learning' and 'Deep Learning' here? Last time I heard of those terms, 'Machine Learning' was a much broader category than 'Deep Learning'. Its like someone says 'Next years Porsches will be illegal' and the next person calls 'Next year cars will be illegal'?!?

nmca 9y ago

For those interested in the cutting edge of interpretability, check out https://github.com/marcotcr/anchor https://github.com/marcotcr/anchor I'm pretty sure this sort of approach will be how most businesses handle the problem.

emilfihlman 9y ago

I find it extremely concerning that people, especially on this platform, don't care about what the law says and choose to just hand wave it as "it doesn't apply to us / they don't enforce it like that". The letter of the law is even more important than the spirit of it because it allows governments, corporations and other entities to simply say "this is what the law says" and because you are small you don't have the money/power to challenge this rendition if the spirit of the law disagrees with them. Not to mention that laws are _laws_, they are not meant to be really interpreted (it is a byproduct of having to do so to have a functioning society). It's an extremely slippery slope to not care what the law says. If we don't we'll have some dystopian laws that allow for anything quite soon.