16 ms·

Why are we using black box models in AI when we don’t need to? (2019)

• sigmaprimus 7y ago
  I think the main reason is that creating a so called "Black Box" model is faster and easier to some degree. Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?" I tend to agree with the articles premise that any model being used in critical decision making should at the very least have a list of parameters and the weight given to each, no matter how long and complicated such a list might be. I think eventually this will be the end result of most critical implementations of machine learning applications as they make their way through the courts, as I can't imagine a judge accepting the argument "The machine made a mistake, we are not sure why or the reason why is proprietary, but we are not responsible because the machine made the error".
  • sigmaprimus 7y ago
    Ofcourse the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. As much as I hate regulations I think this may be a good place to legislate in order to protect those wrongly affected. It makes me think about the mess the derivatives markets made in 2008
    • taneq 7y ago
      It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint. Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.
      • BoiledCabbage 7y ago
        No, they were impenetrable from the customer perspective, but either the human making the decision or the software being used can answer. With a black box model, nobody can answer.
        • taneq 7y ago
          From the post I responded to: > the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. I was pointing out that it’s already like this. Whether it’s a black box ML model or a capricious bank employee makes no difference if there’s no transparency either way.
      • _jal 7y ago
        Bank managers, however, can at least theoretically be hauled in to court and questioned.
        • taneq 7y ago
          That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.
          • _jal 7y ago
            Yes, I'm (somewhat) aware of the research. I'm unaware of any legal jurisdiction that recognizes it, let alone tries to address it somehow. There's reality, there's legal reality, and then there's what legal reality does to reality.
  • rightbyte 7y ago
    The killer feature with "black box" models is that you can do illegal discrimination with regards to eg. race and gender and then just blame the computer since you can't see the weight as easily as in component analysis.
  • klyrs 7y ago
    > Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?" Exactly. We included mathematicians, computer scientists and engineers in the design of those calculators. They use algorithms backed by rigorous proofs, run upon circuits with rigorous tolerances such that the answer is correct (to a reasonable amount of precision). Kids with their calculators are intellectually lazy -- we should expect that, but continue to teach otherwise. To see this from full-grown adults in academia is an embarrassment. But, god, would I enjoy a ML salary...
  • dijksterhuis 7y ago
    A little off topic, but also not: I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot). At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot. I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs). The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it. And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers. But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).
• owenshen24 7y ago
  My understanding is that interpretable models, especially for neural networks, are very far and away from state of the art performance. Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models [1]. Even in computer vision, which is where I think they've been most successful, the visualization techniques used seem more suggestive, then explanatory [2]. [1] http://www.shallowmind.co/jekyll/pixyll/2017/12/30/tree-regularization/ http://www.shallowmind.co/jekyll/pixyll/2017/12/30/tree-regu... [2] https://distill.pub/2019/activation-atlas/ https://distill.pub/2019/activation-atlas/
  • throwawayjava 7y ago
    I think it's a real open question whether the interpretable models are actually worse, or merely worse in competition/benchmark problem sets. The more deep models I build, the more I'm convinced that behind every inscrutable parameter hides a certain amount of overfitting, with maybe a few notable exceptions. E.g., can you build a decision tree that's not obviously overfit but is also susceptible to adversarial perturbations?
    • owenshen24 7y ago
      I agree that there's probably a good question about to what extent looking at benchmark sets is biasing our judgment. But it's also unclear to me how to get a decision tree to perform as well on image recognition tasks the same way that a CNN does. (Of course, as you mention, the CNN will likely face adversarial examples.)
    • loopz 7y ago
      If it can be explained in terms of precise rules, that's called programming.
      • throwawayjava 7y ago
        The distinction between ML and programming is mostly propaganda, in the sense that it's not flat out wrong but is mostly used to win money/power. It's not actually a helpful way of understanding... anything.
• dchichkov 7y ago
  I've happen to be in that workshop room and had chosen the 2% robot, not the 15% surgeon. If I remember correctly, the point of the question was determining willingness of society to use black box models, not challenging the need for models interpretability. Interpretability obviously doesn't hurt accuracy. But it is costly to engineer. And not always possible to make. Not always possible, because human capacity (and willingness to put in the effort) into understanding the explanation is limited.
  • 1_over_n 7y ago
    >> Interpretability obviously doesn't hurt accuracy. Why do you say this? From what i have seen it certainly can and does. For some industries finding a trade off where the magic is. https://link.springer.com/article/10.1007/s10664-018-9638-1 https://link.springer.com/article/10.1007/s10664-018-9638-1 https://www.oreilly.com/ideas/predictive-modeling-striking-a-balance-between-accuracy-and-interpretability https://www.oreilly.com/ideas/predictive-modeling-striking-a...
    • dchichkov 7y ago
      You can take nearly any black box model and add some interpretability into it, without modifying this model. No modifications to the model, so accuracy stays the same.
• zevets 7y ago
  I think this argument fails to understand that purpose of black-box models is to be impossible to understand, as it shields the operator from pre-emptive accountability. Then, since they are supposedly the 'only way' to provide whatever function the market is 'demanding', then clearly we must abandon any notion of accountability, as this need 'must' be met. This logic is perhaps most evident in google/facebooks content moderation dilemmas, where both companies refuse to define any sort of non-vague actionable standards about what they censor (or fail to promote) as 1) they can't provide them and 2) they don't want to be held responsible for what they actually are. Then, as outrage has grown over facebook/google's terrible content moderation and censorship policies, and the need for accountability has grown, both companies have been forced to hire ever more moderators and censors, because the technologies as claimed just don't work well enough.
  • dmix 7y ago
    Sounds more like a useful side effect than anything someone would design on purpose. But that's an interesting point none-the-less. Occam's razors the easier way to explain the actions of the big guys and their own internal responses to cultural pressures to not allow things they don't like from happening or being tolerated in the world.
  • taneq 7y ago
    Well maybe also because black box models can be applied to a wider range of systems and can help to understand general properties of a class of algorithms. I’m sure it’s mostly all a conspiracy, though.
    • James-primitive 7y ago
      > I’m sure it’s mostly all a conspiracy, though. Yeah, mostly
  • James-primitive 7y ago
    >the purpose of black-box models is to be impossible to understand nothing is impossible to understand
  • 7y ago
• mkl 7y ago
  They never got to the results of the competition and how the interpretable model did! From https://www.fico.com/en/newsroom/fico-announces-winners-of-inaugural-xml-challenge https://www.fico.com/en/newsroom/fico-announces-winners-of-i...: "The team representing Duke University, which included Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang and Tong Wang, received the FICO Recognition Award acknowledging their submission for going above and beyond expectations with a fully transparent global model and a user-friendly dashboard to allow users to explore the global model and its explanations. The Duke team took home $3,000." Cynthia Rudin is one of the article's authors.
  • RSchaeffer 7y ago
    But how did their model compare against others? The article only mentions how their interpretable model compared against their own ML attempts
    • jph00 7y ago
      Their model didn't win. IBM's model won, based on actual metrics around useful insights.
      • Mathnerd314 7y ago
        The IBM team got $5,000 and the second place/honorable mention NYU got $2,000. So going by prize amounts, the Duke model was still pretty good. IBM turned the model/paper into a toolkit: https://www.ibm.com/blogs/research/2019/08/ai-explainability-360/ https://www.ibm.com/blogs/research/2019/08/ai-explainability... Their model seems to be a variant of decision trees that has a knob controlling how complicated the trees are. And the evaluation was completely subjective, so there's not any meaning to the Duke people losing besides that the judges didn't like them.
        • A_No_Name_Mouse 7y ago
          > And the evaluation was completely subjective, so there's not any meaning to the Duke people losing besides that the judges didn't like them. That's what you get if you use a black box for judging :-)
• anfilt 7y ago
  Like I have pretty much always thought this when I read about ML or uses of it. The process and end result is just math so it has to be explainable in some way. I feel like not enough effort is given to understand the important variables to ML models or how are results derived. When I see uses of it it's more like the following: "Oh, it works, that's good enough." For-instance when a model starts being over fitted, one can say it's pretty much converging to something akin to a LUT (Look up table). While over fitting is generally undesirable it still might be interesting. Principally, if someone could figure out how basically it is indexing the data. Perhaps you could create a simpler hash function or find some rules for creating useful hash functions for that type of data.
  • psv1 7y ago
    The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that works for 100s or 1000s of dimensions.
    • otnobot 7y ago
      Yes, the model should work with any "new or old" data and also any "type". Data from any products, services, commodities or measurable force / energy. The model will need "adjustments" from time to time but the core will stay intact indefinitly.
    • fallous 7y ago
      Unless there is a self-modifying feedback loop you can indeed replace any ML model with a hash function. The problem is one of size rather than any fundamental limitation of a look-up table.
    • shakna 7y ago
      Shouldn't this fall into the "trust by verify" category of things? Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour. You have to trust it on new data. You can always verify why it made the choices it did on old data. That seems like something incredibly helpful, to me.
• OrgNet 7y ago
  why include 2019 in the title of this post? was it automatically added (is that a bug)?
  • Gaelan 7y ago
    There's a (human-enforced) HN rule to do that for old articles, so they're not mistaken for news. I'm not sure what the standard for "old" is, but it isn't 2019 anymore, so I guess it's technically correct.
    • OrgNet 7y ago
      2 months old is usually not considered old news on this site... I think that their algo is buggy on year-change
      • Gaelan 7y ago
        As far as I know, there's no algorithm doing it. People put it in the title themselves, or mods do if they forgot.
• Lucadg 7y ago
  The convenience to be able to say: "we didn't take the decision, the algorithm did" is the holy grail of non accountability. I wouldn't be surprised if someone, especially in business, is actively opposing transparent AI.
• jayd16 7y ago
  Can you train a white box solution with only a tagged data set? It seems obvious to me why we have black box solutions even if we could have spent man hours engineering an algorithm.
  • swiftcoder 7y ago
    You actually can - for example https://arxiv.org/abs/1806.10574 https://arxiv.org/abs/1806.10574
• ummonk 7y ago
  I think there is a lot of scope for interpretation in black box models. E.g. when doing transfer learning with a frozen pretrained model it should explain the new classifications in terms of pretrained classes.
• 1e-9 7y ago
  The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you should not even be considering a blackbox approach (at least not without transparent safeguards). There is a large gray area between these two extremes that requires good engineering judgement to choose well. I think we definitely need more R&D dedicated to creating easier-to-use, lower-cost approaches to transparent, explainable ML. There is way too much effort devoted to blackbox R&D today. Ultimately, transparent, explainable ML should almost always beat blackbox ML due to a better ability to find and resolve hidden problems, such as dataset biases, that may be holding back performance.
  • 1_over_n 7y ago
    I would also add in that blackbox ML solutions are deceptively "simple" approaches. Like a miraj in the desert for software developers: No domain expertise neccessary - or even the counter narriatives may come in... "domain expertise holds you back - you need to be an outsider with a new method", where as many problems require domain expertise for a realistic understanding of the complexity of the problem-solution. Other narriatives that can creep in "once we get enough data..." which for some problems may as well be never. Also, for some systems pertubing the system changes the problem meaning your dataset and model may not reflect changes in a dyanmic / complex system. Also, blackbox ML approaches struggle to combine data from multiple modalities which is often relevant to many real world problems (at least with the majority of algorithms which are realistically implementable off the shelf).
  • autokad 7y ago
    > The main advantage of a blackbox ML solution is shorter development time to a useful performance level. I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16 GPU instances over a couple weeks of training ) meanwhile, I consider gradient boosting fairly interpretable and they can get pretty close results with a lot less tweaking and training time. If you want to go full non-black box, logistic regression with l1 penalization and only a little bit of feature engineering often does really well, probably a lot less development time / cost compared to those high cost PHD research scientists.
• m0zg 7y ago
  Human cognition itself is a black box model. We routinely make snap decisions first and then justify them after the fact. Anything that requires serious consideration is darn near insurmountable for a human mind in the presence of incomplete information, emotions, and conflicting motivations. So while I'd like neural nets to be more interpretable, to me it'd take a distant second place. The first would be to get models that actually work better than humans for practical tasks, even under limited circumstances.
  • Rapzid 7y ago
    I share this sentiment. It occurs to me that human decisions making can be either explained a very high level, or a very low level(neurons firing). But the magic in between is too complicated to draw direct lines between stimuli and decisions. AI seems to be the same way. We have statistical models that explain how most everything works at a low level. That a simple math formula can approximate just about anything. How we arrive at weights that separate data sets in different dimensions and etc. For convolution networks we also understand how layer "decisions" flow through to other layers. But, it's too complicated to look at an image and explain exactly how the input pixels will result in a classification output. I'm not sure how much I should care about that.. Seems like a problem for mathematicians/statisticians. Having the doctor explain himself gives me no more true insight into how his brain works than having somebody explain at a high level how the robot was trained and reacts in different situations.
• andlima 7y ago
  It's not that accuracy will always be sacrificed if one wants an explainable model. The point is: if interpretability is an important constraint, it could prevent improvements on accuracy. Sometimes, the best interpretable model is as good as a black box, and that's great. When this is not the case, the trade-off is that one should see what's more important for the actual problem. Perhaps interpretability is not a big deal. Another solution is to try to extract interpretability from the more accurate black box model with something like SHAP.
  • 1_over_n 7y ago
    This is a great point. There is a general lack of understanding about what it means for models to be interpretable & explainable. These words get thrown around often by people who don't understand the definition, and also the trade off with accuracy. Some papers i found interesting on the subject: https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1707.03886 https://arxiv.org/abs/1707.03886 https://arxiv.org/abs/1806.07552 https://arxiv.org/abs/1806.07552 https://arxiv.org/abs/1702.08608 https://arxiv.org/abs/1702.08608 (i found this was a good sumary of the issues)
• rubyn00bie 7y ago
  > [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer. If your data is off, your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science (e.g. racist AI making racist predictions). A model cannot be accurate if it doesn't have enough information (like predicting crime, or the stock market). There are an insane amount of statistical tests to detect bullshit, and yet we're pushing those and hypothesis testing right out the window when we create models we don't understand. Like I just don't get how some folks say "look what my AI knows" and assume it's even remotely right without understanding the underlying system of equations and dataset. You honestly don't even know if the answer you're receiving back is to the question you're asking, it might just be strongly correlated bullshit that's undetectable to you the ignorant wizard. I find it pretty hard to believe we can model the physical forces inside a fucking neutron star (holla at strange matter) but literally no one in the world could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could, it would just feel unrewarding to invest that much time into doing it correctly. I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.
  • StuffedParrot 7y ago
    > So uhhh, isn't this like not science? Very little of technology has anything to do with validating hypotheses. > meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions. The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model. This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.
• StuffedParrot 7y ago
  Absolution of liability. Why else use the term "AI" to describe your heuristic?
• crazygringo 7y ago
  I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction. Black box doesn't necessarily mean bad. I think the relevant question is: how do we ensure machine learning is trained and verified in ways that don't encode bias, and only used with "intended" inputs, so a model isn't being trusted to make predictions on things it wasn't trained for? And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?
  • fhennig 7y ago
    > because despite being less accurate, they can have designers who can be held legally accountable and liable Are you implying you cannot be held accountable for black box models?
    • darkerside 7y ago
      I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?
      • danShumway 7y ago
        I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain. AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for using the system. You can still hold manufacturers and businesses accountable for promising a degree of accuracy that their system couldn't meet. In the same way, if my dog bites someone, and I get sued and argue, "animal motivation is really hard, we don't know why the dog chose to do that", I'm not going to win the case. The reason it happened is because I didn't leash my dog. The reason the black box hurt someone is because an accountable human improperly used it or inaccurately advertised it.
• mlthoughts2018 7y ago
  “Explainability” is a subjective idea that allows people to act as gatekeepers about the use of a model through arbitrary politics. “The Mythos of Model Interpretability” is good reading on this. https://arxiv.org/pdf/1606.03490.pdf https://arxiv.org/pdf/1606.03490.pdf
• jph00 7y ago
  This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example: > "For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depended on race, age, and criminal history, that COMPAS itself must depend on race. However, when one approximates COMPAS using a nonlinear model, the explicit dependence on race vanishes, leaving dependence on race only through age and criminal history. This is an example of how an incorrect explanation of a black box can spiral out of control." The concern about the strong relationship between race and COMPAS predictions is not largely based on a concern about whether there is an explicit dependence in the model. The concern is whether there's a relationship either explicitly or implicitly. And in particular, whether such a relationship results in unfair outcomes. The findings of the ProPublica study (https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm https://www.propublica.org/article/how-we-analyzed-the-compa...) strongly suggested this was the case: "- Black defendants were often predicted to be at a higher risk of recidivism than they actually were. Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent). - White defendants were often predicted to be less risky than they were. Our analysis found that white defendants who re-offended within the next two years were mistakenly labeled low risk almost twice as often as black re-offenders (48 percent vs. 28 percent). - The analysis also showed that even when controlling for prior crimes, future recidivism, age, and gender, black defendants were 45 percent more likely to be assigned higher risk scores than white defendants." I understand the desire of the MIT researchers to promote the value of their work, but in this case they appear to be doing so in a potentially damaging way.
  • makomk 7y ago
    No, they're not misrepresenting this at all. ProPublica's article https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... was pushing the claim that somehow, the COMPAS black box was implicitly deducing defendants' race from the "137 questions" input to it and labelling them likely reoffenders based on it in a way that was indepdendent of the key factors known to affect reoffending rates, such as age and gender. The paper in question seems to demonstrate the exact opposite is true: ProPublica were inadvertently using race as an imperfect proxy for age at sentencing, which is what the COMPAS algorithm really cared about, because their attempt to control for age at sentencing didn't work. After controlling for the actual age factor, COMPAS results didn't have any relationship to race anymore. (It's not even a weird weighting factor: predicted reoffending risk falls off rapidly with increasing age at first, then more slowly in a smooth fashion. It's just not linear.) Now, it's of course possible to argue that judging reoffending risk based on age is in fact unfair and racist because it has disproportionate impact on certain racial groups, even though it's strongly predictive across all racial groups. That's not the argument ProPublica made, though. Their argument was about the supposed perils of black boxes, and they kind of acknowledged that age probably wasn't a racist criteria - or at least that it would be a lot harder to justify calling it one - by attempting to strip out its effects in the first place. It's also a different kind of argument entirely, one that revolves not around whether the algorithm is somehow treating people differently based on their detected race - because it isn't - but around what it means for a decision like this to be fair in the first place.
• computerex 7y ago
  This is a pointless article. Obviously everyone would prefer fully interpretable models if it was possible to get comparable performance with the leading black box/deep learning approaches. No one chooses to make black box, non-interpretable models, it's simply the fact that the models that give the best performance tend to be black box. There is a lot of work happening on trying to interpret deep learning models, because no one prefers non-interpretable models and being unable to pin point and explain why some prediction was wrong.
  • erostrate 7y ago
    The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models. They also give other examples of cases where people have used black box models unnecessarily.
    • computerex 7y ago
      > The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models. That's awesome, but this was a single model for a single application being measured against presumably by a limited test criteria/benchmarks. The point still stands that the recent narrow AI "renaissance" is largely due to deep learning, which is inherently black box. There is a lot of work going on in making deep learning more interpretable precisely because it's so prominent today and because its lack of interpretability is a huge con. Machine vision for example has come a long way due to deep learning. A lot of autonomous car companies are relying on it for perception. All of these companies hate the fact that there is no way to tell when the classifier will fail or why it'd fail. When using deep learning in finance setting, its lack of interpretability is a huge downside. Despite this deep learning is being used because it provides an appreciable improvement in performance over the alternatives. Certainly there could be alternatives to deep learning that may perform just as well or even better. But finding those alternatives for the vast number of applications deep learning is being used for today is much easier said than done.
• rdlecler1 7y ago
  For god sakes. Enough of this black-box—we don’t understand how this works narrative. Neural networks are JUST computational circuits. Unfortunately there’s been a lot of laziness in cleaning those circuits up (W_ijk=O) to better reveal the topological circuitry.
• hyperpallium 7y ago
  It's true our own cognition is also a black box, e.g. how do we recognize? It's as much a mystery to introspection as digestion. But the goal of science is to understand. Whereas with DL, we're just chimps in the dirt playing with sticks.
• boltzmannbrain 7y ago
  A couple salient quotes from "Interpreting AI Is More Than Black And White" in Forbes [1]: "No matter the definition, developing an AI system to be interpretable is typically challenging and ambiguous. It is often the case that a model or algorithm is too complex to understand or describe because its purpose is to model a complex hypothesis or navigate a high-dimensional space, a catch-22. Not to mention what is interpretable in one application may be useless in another." "Underspecified and misaligned notions of interpretation impede progress towards the rigorous development of understandable, transparent, trusted AI systems." [1] https://www.forbes.com/sites/alexanderlavin/2019/06/17/beyond-black-box-ai https://www.forbes.com/sites/alexanderlavin/2019/06/17/beyon...
• i000 7y ago
  While I can definitely see the value of black box models in engineering and commercial applications (including clinical ones), I am quite troubled by the new hype of applying to deep learning to so many basic scientific problems. The 'deep' models do not tend to result in deeper insights - which idealistically ought to be the goal.
• somurzakov 7y ago
  When I want to predict something, I need the highest accuracy, and I am not interested in interpretability of an individual parameter. and interpretability is only a half measure. If i really want to understand the model, I will conduct causal analysis with randomized controlled trial and will find the root cause and impact of the factor of my interest.
• szczepano 7y ago
  What a dumb question is it? The same logic is behind sports and nobel price awards. It's easier to say to people who make financial decision that this is great artificial intelligence that will solve all your problems then explain mathematical equations behind it that 90% people won't understand. The more complicated the stuff is the higher probability it will be sold. That's the standard in IT world from quite some time.
• bitL 7y ago
  I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of their interpretable models, then acting as arbiters and judges of the work of others, preventing future advancements of the field.
  • lonelappde 7y ago
    In your view, it's better to answer the wrong questions optimally than the right questions suboptimally? That's not the kind of AI want putting people in jail or denying them housing or healthcare or shooting missiles at them.
    • bitL 7y ago
      There are two parts to my opinion: 1) I would prefer something that works most of the time (blackbox) than something interpretable that doesn't 2) if my DL converged to e.g. solving some complex partial differential equation, how does it help me that I could "interpret that" if 99.9999% human population has no clue what is going on regardless? "Your loan was rejected, because the solution to this obscure PDR with these starting conditions said so; these Nobel prize economy winners created this model with these simplifications and we deem it good enough to keep you where you are."
  • progval 7y ago
    > Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage While this is probably true (and IMO, possibly the right choice), this: > faking results of their interpretable models is taking a huge leap. Why are they more likely to fake their results than companies selling a black box model?
  • throwawayjava 7y ago
    > IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. I'm also pretty wary of interpretability/explainability research in AI. Work on robustness and safety tends to be a bit better (those communities at least mathematically characterize their goals and contributions, and propose reasonable benchmarks). But I'm also skeptical of a lot of modern deep learning research in general. In particular, your critique goes both directions. If I had a penny for every dissertation in the past few years that boiled down to "I built an absurdly over-fit/wrongly-fit model in domain D and claimed it beats SoTA in that domain. Unfortunately, I never took a course about D and ignored or wildly misused that domain's competitions/benchmarks. No one in that community took my amazing work seriously, so I submitted to NeurIPS/AAAI/ICML/IJCAI/... instead. On the Nth resubmission I got some reviewers who don't know anything about D but lose their minds over anything with the word deep (conv, residual, variational, adversarial, ... depending on the year) in the title. So, now I have a PhD in 'AI for D' but everyone doing research in D rolls their eyes at my work." > Those same people will likely at some point call for a strict regulation of AI... The most effectual calls for regulation of the software industry will not come from technologists. The call will come from politicians in the vein of, e.g., Josh Hawley or Elizabeth Warren. Those politicians have very specific goals and motivations which do not align with those of researchers doing interpretability/explainability research. If the tech industry is regulated, it's extremely unlikely that those regulations will be based upon proposals from STEM PhDs. At least in the USA. > faking results of their interpretable models Jumping from "this work is probably not valuable" to "this entire research community are a bunch of fraudsters" is a pretty big jump. Do you have any evidence of this happening?
• hooande 7y ago
  I don't know how many of you are familiar with Kaggle competitions, but the winning scores on the leaderboards can come down to 0.0001 points of accuracy or less. Winning something like that with an explainable model is like having one hand tied behind your back. If tiny fractions of a point matter (the rare case), then making the model explainable is adding a ton of complexity for little practical gain. It sounds nice, but the numbers are unforgiving
• lalaland1125 7y ago
  Interpretability is probably one of the most misunderstood topics in AI. Interpretability is fundamentally not a math or statistics challenge, it's a Human Computer Interaction (HCI) problem and needs to be studied in a similar manner, with user studies. The fundemental goal of interpretability is to help users achieve some concrete goal such as detecting bias, identifying bad models, and debugging. This is similar how we can better design keyboards to help users achieve faster typing speed. Unfortunately, most interpretability studies don't actually bother testing if their methods actually work and I am highly skeptical that things like "white box models" or salience maps or attention weights or whatever latest methods can actually achieve the aims that interpretability proponents claim they do. I would love to see a simple randomized user study that would emperically show a concrete improvement in something like bias detection or error detection.
  • anthony_doan 7y ago
    > nterpretability is fundamentally not a math or statistics challenge No, I keep on seeing AI/ML/DS people keep on downplaying statistic. Statistic interpret things. The majority of the models out there have a one to one, predictor to response, holding all other predictor constant (linear regression, logistic regression, arima, anova, etc..). Statistic inference is a thing. Inference is interpreting. Descriptive statistic is interpreting. Parsimonious is a thing. Experimental design is a thing. Degree of freedom is a thing in statistic. If you want interpretability do statistic. One of it's tenant is to quantify and live with uncertainty not fitting a curve and lots of coefficient to just predict. Not just classification. It's a reason why biostat or econometric is a thing. Statistic. Even the blog cited statistic papers even though it barely mention statistic models in it. ~~And Rudin is a statistician and contributed a lot in statistic.~~ Wrong person (I'm thinking of Rubin for casuality and missingness) This is not a tribal fight between statistic and ML. This is pointing out that ignoring statistic is a detriment to AI/ML/DS as a field. I predict that 2020 to 2030 statistic will be coming to AI/ML much more so regardless how much people downplay statistic. ~~Seeing on Dr. Rudin is coming over.~~ I've seen other statistician too. Dr. Loh works took decision tree and added ANOVA and Chisqaure to build parsimonious decision tree.
• amitpaka 7y ago
  Interpretable models are an exciting new entry to the ML toolkit. While they are likely sufficient for everyday ML tasks, they might not be expressive enough vs deep networks to tackle complex tasks like fraud and anti-money laundering classification. As with any tool, interpretable vs black models have optimal target applications that data scientist can apply them for. As interpretable models get wider tooling for training, expect them to play a larger role in simpler ML tasks. Black box models are however here to stay and will always have their place with their versatility.
• nurettin 7y ago
  We have all these tools to build neural networks and we use them to peform feats where a simple C4.5 would have sufficed. To continue innovation in the algorithmic field, I think we need a branch of scientific research or a competition where researchers are constrained to 100MHz CPU clocks and no GPU cheats when actually running their algorithms (not when loading and parsing data)
• tw1010 7y ago
  I feel like we're ignoring the obvious answer: because most people in AI right now are overwhelmed by the necessary mathematical background and comfort necessary to produce first-principles generative innovative algorithms and models, so all they're left with is following the latest trend, which is locked on deep learning right now.
• netcan 7y ago
  Ultimately, a lot comes down to the accessibility of tools and know how. The more widespread and general a technique or tool, the more widespread and general it becomes... because people already have it and know how to apply it. OTOH, these authors (and old school heavyweights like David Ferucci) are not wrong. Interpretability, explainability & interoperability with human intelligence is not something to just give up on. I like "challenges" as ways of exploring these areas. Good luck to the authors. Female team, btw.
• tdhttt 7y ago
  When using ML models to deal with natural science such as physics, working well+being interpretable actually implies new discovery. Working well only, on the other hand, cannot win you a Nobel Prize.
• sytelus 7y ago
  There are probably dozens of holes one can drill into this sensationally titled click-baity study. It's surprising what gets through Nature's review system these days.
  • hesselink 7y ago
    And yet you provide none. What do you base this on?
• z3rgl1ng 7y ago
  ITT: people espousing opinions who clearly didn’t read the paper.
• colinprince 7y ago
  Off topic, apologies, but why are we putting (2019) in the title when this article is barely two months old? Makes is seem like it's an out-of-date article. Is it?
• Ericson2314 7y ago
  The interest in black-box models is partly for the reasons given, that it is presumed that their inexplanability makes them more powerful, but also because business people are sick of dealing with programmers and analysts as a class, people with arcane knowledge limited replacability, and a penchant for blowing deadlines. The dream of machine learning is to fire the programmer and break up the last trade (baring doctors and lawyers). If it was just business people being alienated, I'd say fuck it they deserve it. But the fact is everyone, and many of the programmers too, are alienated too when much of the code makes sense to no one. I would rather fix programming to be comprehensible to more people even though it will take down the walls between programmer and non-programmer.
• tripzilch 7y ago
  > there are actually a lot of applications where people do not try to construct an interpretable model, because they might believe that for a complex data set, an interpretable model could not possibly be as accurate as a black box. Or perhaps they want to preserve the model as proprietary. I think they missed an important reason here. It's why I decided to study AI and Machine Learning: The ability to "just throw some data at it and see if it figures it out" is just so exciting, mysterious and intriguing. I remember actually being a bit disappointed when I learned how (classic, 3 layer, early 2000s) neural networks worked. That it was just something super simple with derivatives, and resulted in something that seemed a lot like a more complex form of statistical regression. Kind of took the magic out of it for me, a little (didn't stop me of course, it was and still is an exciting field of research). I know it's not a good reason, but surely I'm not the only one who thinks it's extremely cool that one is even able to build a black box that does useful things with what you feed it, but you don't know how it works, and yet you built it. It's just something mystifying, and I believe there is also a tinge of fear for that disappointment, to figure it out and find out it's not really doing as clever things as you hoped it might.