4 ms·
My favourite is still Art. 22, p. 1: "The data subject shall have the right not to be subject to a decision based solely on automated processing, including pro
by Tade0 6y ago
My favourite is still Art. 22, p. 1:
"The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her."
Which means that if e.g. a bank declines your mortgage application based on a decision from a fully automated system you have a right to have a human being review it.
Interestingly this made banks reluctant to use AI and helped with efforts in development of so-called "explainable AI".
- CalRobert 6y agoJust did exactly that with Avant. Could see where I was initially approved but then their manager said I was "still on contract" and quashed my application.
- em-bee 6y agodoesn't sound like this worked in your favor, or did i get that wrong?
- TeMPOraL 6y agoI read this as you do, but it's still a big win. If you make a machine learning model explain itself to humans, you open the possibility to challenge the model's conclusion. It's not just the question of whether the model does its math right - it also means we, as society, can say, "I see your math and it works out, but because $POLICY, you're not allowed to take these inputs into account". It means adding control points to a previously opaque process.
- em-bee 6y agoyou are right, and i absolutely agree. basing any decisions that substantially affect someone on machine learning alone should be illegal (even if it is the right decision)
- CalRobert 6y agoI was denied, but now I know why instead of just guessing. I could see that my "willingness to pay" (unsure what that means) etc and income were all fantastic, their manager just didn't like that I was on contract instead of a permanent employee.
- em-bee 6y agoso arguably, in this case the machine learning algorithm was possibly right, and the human was wrong. on the face of it, not using machine learning here would not have changed the outcome. what i am wondering is though if this can't be used to everyones benefit. apart from neding a human to verify a decision, how about, if the human and machine learning decision disagree, then the consumer gets the right to appeal for a second human review?
- CalRobert 6y agoFunny thing is that I specifically opted out of automated decision making, having been declined before. It wasn't a big deal, really. EU lenders are far warier of contract jobs than American ones.
- em-bee 6y agooh there was an opt-out. that's interesting. how does that work? you obviously opted out before, but could you also opt out after they tell you what the automated decision would be? if so that would effectively be that kind of appeal that i suggested.
- CalRobert 6y agoI applied for a loan a year ago, didn't opt out got denied, and wondered why. Then I applied again ~4 months ago, opted out of automated decision making, and the subject access request showed the conversation.
- oytis 6y agoBut also banks don't have to explain you why they have taken this or that decision.
- afiori 6y agoIt is about whose mistake or choice it is, if a bank denies loans to $ProtectedCategory because of a bug in its software it is one thing, if it is because employees have biases and discriminate applicants it is another.
- johnchristopher 6y agoWhat about a human reviewing the AI review and just nodding all along ? Does that fly ?
- number6 6y agoYes, the Nodder has to have proforma authority to override the AIs decision.
- antris 6y agoIn that case there is a human being who is legally responsible for the decision that is made. I have no problem with that. If the person is "just nodding", with possibly illegal consequences, that person can be sued.
- dorgo 6y agoWhy not suing the bank? It always works, AI or no AI.
- antris 6y agoWell, if all the decisions are given to the AI, it's harder to establish mens rea. They can just claim they never intended to discriminate anyone but that it was only an "unfortunate glitch" in the system. Of course, this defense might not hold, but AI is a decent legal cover as it's more difficult point to any single person knowingly doing anything illegal.
- BlueTemplar 6y agoSome countries have pushed it further : for administrative decisions where a program was used (whether alone or helping a decision), all the steps of the algorithm resulting in the decision must be provided in clear and simple language (if a request is made).