4 ms·

> Notably, Recital 12 says the definition "should not cover systems that are based on the rules defined solely by natural persons to automatically execute opera

> Notably, Recital 12 says the definition "should not cover systems that are based on the rules defined solely by natural persons to automatically execute operations." That's every AI system. It follows the rules defined solely by the programmers (who I suppose might sometimes stretch the definition of natural persons) who made pytorch or whatever framework.

freeone3000 2y ago
It means a decision tree where every decision node is entered by humans is not an AI system, but an unsupervised random forest is. It’s not difficult to see the correct interpretation.
TeMPOraL 2y ago
In strict mathematical reading, maybe - depends on how you define "rules", "defined" and "solely" :P. Fortunately, legal language is more straightforward like than that. The obvious straightforward read is along the lines of: imagine you make some software, which then does something bad, and you end up in court defending yourself with an argument along the lines of, "I didn't explicitly make it do it, this behavior was a possible outcome (i.e. not a bug) but wasn't something we intended or could've reasonably predicted" -- if that argument has a chance of holding water, then the system in question does not fall under the exception your quoted. The overall point seems to be to make sure systems that can cause harm always have humans that can be held accountable. Software where it's possible to trace the bad outcome back to specific decisions made by specific people who should've known better is OK. Software that's adaptive to the point it can do harm "on its own" and leaves no one but "the system" to blame is not allowed in those applications.
Muromec 2y ago
If the thinking machine rejects my mortgage application, it should be possible to point out which exact rule triggered the rejection. With rules explicitly set by an operator it's possible. It's also possible to say that the rules in place comply with the law and stay compliant during the operation, for example it doesn't unintentionally guess that I'm having another citizenship based on my surname or postal code.
- tzs 2y ago
  If the mortgage application evaluation system is deterministic so that the same input always produces the same output then it is easy to answer "Why was my application rejected?". Just rerun the application with higher income until you get a pass. Then tell the person their application was rejected because income was not at least whatever that passing income amount was. Maybe also vary some other inputs to see if it is possible to get a pass without raising income as much, and add to the explanation that they could lower the income needed by say getting a higher credit score or lowering your outstanding debt or not changing jobs as often or whatever.
  - mjburgess 2y ago
    That tells you how sensitive the model is along its decision boundary to permutations in the input -- but it isnt a relevant kind of reason why the application was rejected, since this decision boundary wasnt crafted by any person. We're here looking for programs which express prior normative reasoning (eg., "you should get a loan if...") -- whereas this decision boundary expresses no prior normative reason. It is simply that, eg., "on some historical dataset, this boundary most relibaly predicted default" -- but this confers no normative reason to accept or reject any individual application (cf. the ecological fallacy). And so, in a very literal way, there is no normative reason the operator of this model has in accepting/rejecting any individual application.
  - xvokcarts 2y ago
    > If the mortgage application evaluation system is deterministic so that the same input always produces the same output then it is easy to answer "Why was my application rejected?". But banks, at least in my country (central EU), don't have to explain their reasons for rejecting a mortgage application. So why would their automated systems have to?
    - tzs 2y ago
      In the US they do have to explain. It's a requirement of the Equal Credit Opportunity Act of 1974 [1]. Here is an article with more detail on what is required in the explanation [2]. [1] https://en.wikipedia.org/wiki/Equal_Credit_Opportunity_Act#Requirements https://en.wikipedia.org/wiki/Equal_Credit_Opportunity_Act#R... [2] https://www.nolo.com/legal-encyclopedia/do-lenders-have-to-tell-you-why-you-are-denied-credit.html https://www.nolo.com/legal-encyclopedia/do-lenders-have-to-t...

freeone3000 2y ago

It means a decision tree where every decision node is entered by humans is not an AI system, but an unsupervised random forest is. It’s not difficult to see the correct interpretation.

TeMPOraL 2y ago

In strict mathematical reading, maybe - depends on how you define "rules", "defined" and "solely" :P. Fortunately, legal language is more straightforward like than that. The obvious straightforward read is along the lines of: imagine you make some software, which then does something bad, and you end up in court defending yourself with an argument along the lines of, "I didn't explicitly make it do it, this behavior was a possible outcome (i.e. not a bug) but wasn't something we intended or could've reasonably predicted" -- if that argument has a chance of holding water, then the system in question does not fall under the exception your quoted. The overall point seems to be to make sure systems that can cause harm always have humans that can be held accountable. Software where it's possible to trace the bad outcome back to specific decisions made by specific people who should've known better is OK. Software that's adaptive to the point it can do harm "on its own" and leaves no one but "the system" to blame is not allowed in those applications.

Muromec 2y ago

If the thinking machine rejects my mortgage application, it should be possible to point out which exact rule triggered the rejection. With rules explicitly set by an operator it's possible. It's also possible to say that the rules in place comply with the law and stay compliant during the operation, for example it doesn't unintentionally guess that I'm having another citizenship based on my surname or postal code.

tzs 2y ago
If the mortgage application evaluation system is deterministic so that the same input always produces the same output then it is easy to answer "Why was my application rejected?". Just rerun the application with higher income until you get a pass. Then tell the person their application was rejected because income was not at least whatever that passing income amount was. Maybe also vary some other inputs to see if it is possible to get a pass without raising income as much, and add to the explanation that they could lower the income needed by say getting a higher credit score or lowering your outstanding debt or not changing jobs as often or whatever.
- mjburgess 2y ago
  That tells you how sensitive the model is along its decision boundary to permutations in the input -- but it isnt a relevant kind of reason why the application was rejected, since this decision boundary wasnt crafted by any person. We're here looking for programs which express prior normative reasoning (eg., "you should get a loan if...") -- whereas this decision boundary expresses no prior normative reason. It is simply that, eg., "on some historical dataset, this boundary most relibaly predicted default" -- but this confers no normative reason to accept or reject any individual application (cf. the ecological fallacy). And so, in a very literal way, there is no normative reason the operator of this model has in accepting/rejecting any individual application.
- xvokcarts 2y ago
  > If the mortgage application evaluation system is deterministic so that the same input always produces the same output then it is easy to answer "Why was my application rejected?". But banks, at least in my country (central EU), don't have to explain their reasons for rejecting a mortgage application. So why would their automated systems have to?
  - tzs 2y ago
    In the US they do have to explain. It's a requirement of the Equal Credit Opportunity Act of 1974 [1]. Here is an article with more detail on what is required in the explanation [2]. [1] https://en.wikipedia.org/wiki/Equal_Credit_Opportunity_Act#Requirements https://en.wikipedia.org/wiki/Equal_Credit_Opportunity_Act#R... [2] https://www.nolo.com/legal-encyclopedia/do-lenders-have-to-tell-you-why-you-are-denied-credit.html https://www.nolo.com/legal-encyclopedia/do-lenders-have-to-t...