4 ms·

I think the main reason is that creating a so called "Black Box" model is faster and easier to some degree. Its very similar to the argument kids make "Why do

I think the main reason is that creating a so called "Black Box" model is faster and easier to some degree. Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?" I tend to agree with the articles premise that any model being used in critical decision making should at the very least have a list of parameters and the weight given to each, no matter how long and complicated such a list might be. I think eventually this will be the end result of most critical implementations of machine learning applications as they make their way through the courts, as I can't imagine a judge accepting the argument "The machine made a mistake, we are not sure why or the reason why is proprietary, but we are not responsible because the machine made the error".

sigmaprimus 7y ago
Ofcourse the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. As much as I hate regulations I think this may be a good place to legislate in order to protect those wrongly affected. It makes me think about the mess the derivatives markets made in 2008
- taneq 7y ago
  It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint. Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.
  - BoiledCabbage 7y ago
    No, they were impenetrable from the customer perspective, but either the human making the decision or the software being used can answer. With a black box model, nobody can answer.
    - taneq 7y ago
      From the post I responded to: > the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. I was pointing out that it’s already like this. Whether it’s a black box ML model or a capricious bank employee makes no difference if there’s no transparency either way.
  - _jal 7y ago
    Bank managers, however, can at least theoretically be hauled in to court and questioned.
    - taneq 7y ago
      That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.
      - _jal 7y ago
        Yes, I'm (somewhat) aware of the research. I'm unaware of any legal jurisdiction that recognizes it, let alone tries to address it somehow. There's reality, there's legal reality, and then there's what legal reality does to reality.
rightbyte 7y ago
The killer feature with "black box" models is that you can do illegal discrimination with regards to eg. race and gender and then just blame the computer since you can't see the weight as easily as in component analysis.
klyrs 7y ago
> Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?" Exactly. We included mathematicians, computer scientists and engineers in the design of those calculators. They use algorithms backed by rigorous proofs, run upon circuits with rigorous tolerances such that the answer is correct (to a reasonable amount of precision). Kids with their calculators are intellectually lazy -- we should expect that, but continue to teach otherwise. To see this from full-grown adults in academia is an embarrassment. But, god, would I enjoy a ML salary...
dijksterhuis 7y ago
A little off topic, but also not: I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot). At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot. I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs). The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it. And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers. But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).

sigmaprimus 7y ago

Ofcourse the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. As much as I hate regulations I think this may be a good place to legislate in order to protect those wrongly affected. It makes me think about the mess the derivatives markets made in 2008

taneq 7y ago
It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint. Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.
- BoiledCabbage 7y ago
  No, they were impenetrable from the customer perspective, but either the human making the decision or the software being used can answer. With a black box model, nobody can answer.
  - taneq 7y ago
    From the post I responded to: > the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. I was pointing out that it’s already like this. Whether it’s a black box ML model or a capricious bank employee makes no difference if there’s no transparency either way.
- _jal 7y ago
  Bank managers, however, can at least theoretically be hauled in to court and questioned.
  - taneq 7y ago
    That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.
    - _jal 7y ago
      Yes, I'm (somewhat) aware of the research. I'm unaware of any legal jurisdiction that recognizes it, let alone tries to address it somehow. There's reality, there's legal reality, and then there's what legal reality does to reality.

rightbyte 7y ago

The killer feature with "black box" models is that you can do illegal discrimination with regards to eg. race and gender and then just blame the computer since you can't see the weight as easily as in component analysis.

klyrs 7y ago

> Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?" Exactly. We included mathematicians, computer scientists and engineers in the design of those calculators. They use algorithms backed by rigorous proofs, run upon circuits with rigorous tolerances such that the answer is correct (to a reasonable amount of precision). Kids with their calculators are intellectually lazy -- we should expect that, but continue to teach otherwise. To see this from full-grown adults in academia is an embarrassment. But, god, would I enjoy a ML salary...

dijksterhuis 7y ago

A little off topic, but also not: I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot). At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot. I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs). The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it. And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers. But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).