6 ms·

But unfortunately it's very easy to find a way around it: Make all decisions based on ML models, and only when a customer manages to go through the bureaucrat

But unfortunately it's very easy to find a way around it: Make all decisions based on ML models, and only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. It's definitely a whole lot better than not having that law, but we're not quite there yet.

protomyth 9y ago
> only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. That doesn't sound exactly legal as the actual reason had to exist at the time of the decision and an after the fact reason would not be the truth. Don't the ML models log why they made decisions in the first place?
- narrator 9y ago
  The log is we multiplied an 80x25x3 matrix of your planet's photograph by 30 40x40 matrices of other numbers and then fed it through a neural net which consisted of adding and multiplying a few hundred other numbers and we got this number, which when run through a tanh function, classified your planet as uninhabitable space garbage. That's why the Vogon constructor fleet arrived and cleared it to make room for an intergalactic highway.
- dogecoinbase 9y ago
  The point of interpretability in ML is that the "reasons" that model arrived at a decision/classification/etc are incomprehensible (and, given e.g. the existence of adversarial examples in image classification, this should be expected -- no reasonable answer exists for "why" the model thought this static was a cat).
  - shawn-butler 9y ago
    It's not magic. It amazes me how many "data scientists" can't explain the basic mathematics behind their modelling. If the outcome of algorithm/model can't be explained by its regulators then it shouldn't be making any decisions of any significance or importance to human lives. When engineers fail to address cracking in a bridge and people die when it falls, humans are held to account and explain why this decision was so. If machine learning statistical models can't do the same, they better not be used to scan any bridges for structural cracking where I live and vote.
  - philipodonnell 9y ago
    I have heard approaches described that train a simpler ML algorithm to predict why a complex ML algorithm made a decision. Decision-tree models predicting neural net models. Paypal or Square I believe.
    - vinn124 9y ago
      how does this solve anything? if a simple decision tree could predict the outputs of more complex deep nets, why not use the decision tree in the first place? also, what do you do when a decision tree isnt powerful enough, as in the case of many interesting problems such as speech, computer vision, etc.
      - philipodonnell 9y ago
        The uses cases for this were around fraud detection systems where you need to provide a reason for the flag. I don't have to know the complexities of how all the factors are interacting within a neural net to be able to tell someone that if they increased their credit score by 100 points, they'd be approved.
  - protomyth 9y ago
    Hold up, data scientists don't know why their ML is deciding things and companies are using it to decide important things in people's lives? Sounds like it should be banned until companies can actually explain the decisions and some data scientists need to take an ethics class.
    - bobcostas55 9y ago
      >until companies can actually explain the decisions That's almost certainly never going to happen. Our most accurate models are unintelligible, and our most intelligible models are inaccurate. There's a trade-off here and without some sort of magic I don't see us transcending that trade-off.
      - sgt101 9y ago
        What's the uplift between your super accurate model and 1-r? What's the uplift vs a 50 node decision tree? Also accuracy is hard to talk about if you don't have the whole distribution you are estimating from; which generally we don't... Many highly accurate models (in the sense of working on the test set and n-fold xvalidation) underperform in production.
      - protomyth 9y ago
        This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate? Going further, how in the heck do they justify using something like this on credit applications or anything else that can adversely affect their fellow humans? The ethics of such a lack of knowledge should give pause and frankly I hope they get sued if this is the truth of the matter. That is monstrous.
        Aeolos 9y ago
        > This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate? They understand that by evaluating the model on a "test dataset" that is, hopefully, representative of the real-world. This does allow you can explain why a given model makes the decisions it makes - it only allows you to understand how well it performs on the data you feed it. Using an inaccurate or biased model to detect assign credit ratings is indeed immoral. This is orthogonal to interpretability: you can have a model that is both inaccurate and interpretable (i.e. it's wrong and you can say why it's wrong); you can also have a model that is highly accurate but not interpretable (i.e. it works, but you cannot explain why).
gpm 9y ago
That sounds a lot like fraud to me... are you sure this would be at all legal? (Alternatively is your argument "but companies might blatantly and intentionally break the law"?)
- lokedhs 9y ago
  I can't speak for the OP, but doesn't companies blatantly and intentionally break the law all the time?
  - gpm 9y ago
    No? Well I mean in aggregate yes, but that's just because there are so damn many of them, that's not different from saying "people murder eachother all the time". Presumably you mean normalized to the number of companies/people involved/something. Also yes if you mean "misinterpret civil law in their favor", or just "not quite comply with some regulation - in the regulators opinion". But not in a case like this where it would appear to be fairly unambiguous and criminal law which you could go to jail for. When that happens it's newsworthy. Besides which, even if that were true, the right response isn't to create more laws on top of this GDPR thing, it's to create it and then enforce it, and fraud. Once you send a few hundred people to jail for fraud I imagine that the rest would fall in line.
maksimum 9y ago
You're right, additional properties are desirable. The ACM has recently published a [set of guidelines](https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf https://www.acm.org/binaries/content/assets/public-policy/20...) for algorithmic decision making. "Access" and "Auditability" are properties that mean it shouldn't be difficult to get the explanation, and that it shouldn't be possible to 'bullshit' an explanation afterwards.

protomyth 9y ago

> only when a customer manages to go through the bureaucratic, expensive process of requesting an explanation, have a human review the case and come up with a plausible reason. That doesn't sound exactly legal as the actual reason had to exist at the time of the decision and an after the fact reason would not be the truth. Don't the ML models log why they made decisions in the first place?

narrator 9y ago
The log is we multiplied an 80x25x3 matrix of your planet's photograph by 30 40x40 matrices of other numbers and then fed it through a neural net which consisted of adding and multiplying a few hundred other numbers and we got this number, which when run through a tanh function, classified your planet as uninhabitable space garbage. That's why the Vogon constructor fleet arrived and cleared it to make room for an intergalactic highway.
dogecoinbase 9y ago
The point of interpretability in ML is that the "reasons" that model arrived at a decision/classification/etc are incomprehensible (and, given e.g. the existence of adversarial examples in image classification, this should be expected -- no reasonable answer exists for "why" the model thought this static was a cat).
- shawn-butler 9y ago
  It's not magic. It amazes me how many "data scientists" can't explain the basic mathematics behind their modelling. If the outcome of algorithm/model can't be explained by its regulators then it shouldn't be making any decisions of any significance or importance to human lives. When engineers fail to address cracking in a bridge and people die when it falls, humans are held to account and explain why this decision was so. If machine learning statistical models can't do the same, they better not be used to scan any bridges for structural cracking where I live and vote.
- philipodonnell 9y ago
  I have heard approaches described that train a simpler ML algorithm to predict why a complex ML algorithm made a decision. Decision-tree models predicting neural net models. Paypal or Square I believe.
  - vinn124 9y ago
    how does this solve anything? if a simple decision tree could predict the outputs of more complex deep nets, why not use the decision tree in the first place? also, what do you do when a decision tree isnt powerful enough, as in the case of many interesting problems such as speech, computer vision, etc.
    - philipodonnell 9y ago
      The uses cases for this were around fraud detection systems where you need to provide a reason for the flag. I don't have to know the complexities of how all the factors are interacting within a neural net to be able to tell someone that if they increased their credit score by 100 points, they'd be approved.
- protomyth 9y ago
  Hold up, data scientists don't know why their ML is deciding things and companies are using it to decide important things in people's lives? Sounds like it should be banned until companies can actually explain the decisions and some data scientists need to take an ethics class.
  - bobcostas55 9y ago
    >until companies can actually explain the decisions That's almost certainly never going to happen. Our most accurate models are unintelligible, and our most intelligible models are inaccurate. There's a trade-off here and without some sort of magic I don't see us transcending that trade-off.
    - sgt101 9y ago
      What's the uplift between your super accurate model and 1-r? What's the uplift vs a 50 node decision tree? Also accuracy is hard to talk about if you don't have the whole distribution you are estimating from; which generally we don't... Many highly accurate models (in the sense of working on the test set and n-fold xvalidation) underperform in production.
    - protomyth 9y ago
      This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate? Going further, how in the heck do they justify using something like this on credit applications or anything else that can adversely affect their fellow humans? The ethics of such a lack of knowledge should give pause and frankly I hope they get sued if this is the truth of the matter. That is monstrous.
      - Aeolos 9y ago
        > This is where I don't understand. If the data scientists don't know the why of the decision making and the software cannot explain itself then how do they understand that it is accurate? They understand that by evaluating the model on a "test dataset" that is, hopefully, representative of the real-world. This does allow you can explain why a given model makes the decisions it makes - it only allows you to understand how well it performs on the data you feed it. Using an inaccurate or biased model to detect assign credit ratings is indeed immoral. This is orthogonal to interpretability: you can have a model that is both inaccurate and interpretable (i.e. it's wrong and you can say why it's wrong); you can also have a model that is highly accurate but not interpretable (i.e. it works, but you cannot explain why).

gpm 9y ago

That sounds a lot like fraud to me... are you sure this would be at all legal? (Alternatively is your argument "but companies might blatantly and intentionally break the law"?)

lokedhs 9y ago
I can't speak for the OP, but doesn't companies blatantly and intentionally break the law all the time?
- gpm 9y ago
  No? Well I mean in aggregate yes, but that's just because there are so damn many of them, that's not different from saying "people murder eachother all the time". Presumably you mean normalized to the number of companies/people involved/something. Also yes if you mean "misinterpret civil law in their favor", or just "not quite comply with some regulation - in the regulators opinion". But not in a case like this where it would appear to be fairly unambiguous and criminal law which you could go to jail for. When that happens it's newsworthy. Besides which, even if that were true, the right response isn't to create more laws on top of this GDPR thing, it's to create it and then enforce it, and fraud. Once you send a few hundred people to jail for fraud I imagine that the rest would fall in line.

maksimum 9y ago

You're right, additional properties are desirable. The ACM has recently published a [set of guidelines](https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf https://www.acm.org/binaries/content/assets/public-policy/20...) for algorithmic decision making. "Access" and "Auditability" are properties that mean it shouldn't be difficult to get the explanation, and that it shouldn't be possible to 'bullshit' an explanation afterwards.