12 ms·

Sometimes, requiring explanability is counterproductive

148 points by metahost 7y ago

krishnagade 7y ago
It depends, while Yann Lecun has an interesting point of view. The argument for explainability depends on the risk of harm from an AI model decision. Explaining why airplanes fly is moderately interesting but why did the 737-max crash is much more interesting and needed. While the former is probably only needed for people studying aerodynamics, the latter is meant for passengers, regulators, airlines, governments, etc. Here is a tweet thread we posted in the past: https://twitter.com/krishnagade/status/1182317508667133952 https://twitter.com/krishnagade/status/1182317508667133952
RookyNumbas 7y ago
I'm surprised he made the drug analogy. It takes decades and hundreds of millions dollars to be able to test a drug on humans. I'm curious if he thinks we need a similar safeguard for all AI-human interactions.
- PaulRobinson 7y ago
  It takes decades and hundreds of millions of dollars in order to build a statistical argument that explains likely effects, side-effects and risks of prescribing that drug to the population at large. It takes that long and costs that much because there is no safe way to do this with human physiology other than to carefully, slowly and expensively try it and measure the consequences. We can take any AI system and model its behaviour with statistical certainty much more cheaply and quickly, and be more confident in its future behaviour. Remember, AI and ML are fancy words for some form of a regression to a mean more often than not. If we run hundreds of millions of tests of such a system in simulation for a very wide range of contexts/inputs (which is cheap to do), we can have a much higher degree of confidence in a short space of time around behaviour of that system than we will for any drug test, even if we still don't fully understand the causality.
  - RookyNumbas 7y ago
    In the case of a go-playing AI, and maybe even things like self driving cars, then yes I agree. But we are quickly moving beyond the types of applications where you can simply test in simulation. Self driving cars are probably right at the edge of this. It's what comes next that worries me. And I don't pretend to know what it will be.
  - naresh_xai 7y ago
    If you were ever involved in the drug discovery process, then you would know that statistical evidence through clinical trials is only for the last ounce of drug testing. Out of 10,000+ drug molecule candidates only ~ 5 molecules get selected for clinical trials. A lot of molecules are removed as candidates for toxic components/ solubility issues/ chemical stability issues etc. So even when we do not understand the precise impact of drugs on humans and there is no safer mechanism to test, we leave only 0.5% of candidate molecules to empirical/statistical evidence in the form of clinical trials. On the other hand if drug discovery was treated as a pure AI problem, we would have thousands of unverified and unsafe molecules in clinical trials. Causal principles get us to 99.5% of the way in drug discovery. Unfortunately not so in AI.
    - PaulRobinson 7y ago
      I'm not involved in the drug discovery process, but I can imagine that manufacturers would be interested in helping to use AI to deal with the funnel much more effectively to get to those ~5 molecules much more efficiently. You're still left with double-blind trials and having to get large sample groups to try those molecules though. And it's for that reason that drug discovery is always likely to be quite slow, complex and expensive - the efficiency gains will be pushed towards the top of the funnel to make new ideas reasonable to explore, I would imagine. My point was that when you're not dealing with human physiology and instead dealing with problems that are more tractable through AI - i.e. using regression to tune algorithms through patterns in data - you are going to get quicker and more impactful returns without the same complexity. And - critically - it's OK to often trust the AI solution you have without understanding causality. If you later find it's doing something odd that is undesirable, you can use that data to help tune the algorithm again without having to understand the causal relationship. Put another way, you can teach an AI to get better without necessarily understanding the subject completely yourself.
heyitsguay 7y ago
I think the type of explainability Yann LeCun is describing in this post is not the same as what people really want when they talk about explainable AI. To use his example, I think most people would be fine with an AI that said "we're doing X because that's what heuristic Y prescribes for the situation", and getting from where we are now to something like that is the challenge. FWIW i see a similar dialogue in biomedical contexts quite a bit, where there's some cool but hard-to-trust work being done in data-driven image restoration (https://www.biorxiv.org/content/10.1101/236463v5.abstract https://www.biorxiv.org/content/10.1101/236463v5.abstract). There's still a lot that could be done short of programs providing a priori mathematical proof for all models.
- buboard 7y ago
  > because that's what heuristic Y prescribes for the situation people will keep asking "why" and keep escalating, but that's a flawed approach to evaluating a model. every answer is literally constructed and thus can't be trusted on its own. The best that an evaluator can do is to evaluate the training dataset
pesenti 7y ago
For those not on FB: Link: https://www.scientificamerican.com/article/no-one-can-explain-why-planes-stay-in-the-air/?fbclid=IwAR0-S3nl5P0-vKUsr0oN5eeXmTVPTHMRZ9L_5bDqQStD-7uEHdukXC5JR6o https://www.scientificamerican.com/article/no-one-can-explai... "We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. Yes, that can be useful. Or at least reassuring. But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation. In many cases of physical phenomena, "explanations" contain causal loops where A causes B and B causes A. A good example is how a wing causes lift. The computational fluid dynamics model, based on Navier-Stokes equations, works just fine. But there is no completely-accurate intuitive "explanation" of why airplanes fly. Is it because of Bernoulli principle? Because a wing deflects the air downwards? Because the air above the wing want to keep going straight but by doing so creates a low-pressure region above the wing that forces the flow downwards sucks the wing upwards? All of the above, but none of the above by itself. Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence. You know what other life-critical phenomena we don't have good causal explanations for? The mechanism of action of many drugs (if not most of them). An example? How does lithium treat bipolar disorder? We do have considerable empirical evidence provided by extensive clinical studies. This is not to say that causality is not an important area of research for AI. It is. But sometimes, requiring explanability is counterproductive."
- deleted 7y ago
  [deleted]
krnsll 7y ago
Would be interested to see what Judea Pearl says about this. Not on twitter any more but back when I was, I recall him regularly tweeting about causality and AI's shortcomings in that regard.
pesenti 7y ago
Einstein had a great quote: "The most incomprehensible thing about the world is that it is comprehensible." Well, turns out, a lot of the world is not comprehensible. And that's what made Einstein irrelevant for the latter part of his life. Our overestimation of the comprehensibility of the world may very well be some version of the Drunkard's search principle. We are much more likely to know about what's comprehensible than what's not.
- buboard 7y ago
  that's a misstatement of einstein's work. he didn't believe the world is incomprehensible, rather that it's not inherently random.
- drcode 7y ago
  As far as I know, a lot of the criticism Einstein leveled against the foundations of quantum mechanics later in his career are more recently being revisited... instead of being irrelevant, some of his later work may simply have been too far ahead of its time.
buboard 7y ago
he s probably correct. A lot of what we call "human explanation" is in reality a rationalized version of a hunch, but not necessarily the best representation of it. At most situations, some part of the brain crosses a threshold and causes a decision or action, but it's rather rare that humans can explain that action correctly. It's also rare that causation can be established from unambiguous temporal order (e.g. clouds/rain); in most cases our rationalizations are post-hoc. We get a glimpse of it from recent language models: It's become rather easy to start blurting out language that is convincibly and comfortingly coherent, and it can be nudged to point to one direction or another. that doesnt mean it's true
gatherhunterer 7y ago
Hopefully a mod can change the link. > Please submit the original source. If a post reports on something found on another site, submit the latter.
- albertzeyer 7y ago
  This is less about the article linked by Yann, but more about what Yann writes. He basically uses the argument made in the article, and adopts it to make a similar argument about AI. And that this argument about AI is the interesting content here (although the linked article is also interesting).
Antwan 7y ago
This guy is a total genius. Shame he doesn't care more about who he is working for and what his researches are used for.
bordercases 7y ago
This is true for the strict Pearl-style form of causation. The concept of what counts as a causative explanation can be more expansive, and it varies between disciplines. See the work of Nancy Cartwright. https://www.researchgate.net/publication/30527010_What_Is_Wrong_With_Bayes_Nets https://www.researchgate.net/publication/30527010_What_Is_Wr... TL;DR we've been explaining "causes" without Bayes Nets for awhile, Bayes Nets unsubtly disregards the common-sense logic scientists use for their practice, including the way that explanations tend to be qualified by context.
t_serpico 7y ago
As a somewhat outsider to deep learning, intuitively it seems to be true that if you were able to demystify the black box, then it would be easier to improve your models (as you understand where it succeeds and fails and why). From this perspective, explainability would be incredibly productive.
- b3kart 7y ago
  I am not convinced the explanations we’d get from these models would be interpretable enough to drive their conceptual improvement. Deep nets just don’t learn nice interpretable features, nothing in the training objective makes them do so. E.g. they make use of really dubious features [1], and might not actually learn “hierarchies” of features as we thought they were [2]. [1] https://arxiv.org/abs/1905.02175 https://arxiv.org/abs/1905.02175 [2] https://arxiv.org/abs/1904.00760 https://arxiv.org/abs/1904.00760
  - naresh_xai 7y ago
    Or you can actually visualize and quantify impact of aggregates of meaningful features within networks using the methodology described in NetDissect or in TCAV. But of course, that casts doubt into a lot of claimed mechanisms and tons of ML research as I have found out.
carlosdp 7y ago
Another more basic example is gravity. We have a statistical intuition of gravity, we depend on it being there and acting in a consistent way for almost everything in physics. But we don't have an accurate understanding of how gravity works or why it's there. We have theories, but nothing proven. But it's so easy to prove gravity's properties (on Earth) by consistent results with experimentation, that it's literally a science project in every elementary school.
arcanus 7y ago
> Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence This is argument is complete nonsense. Navier Stokes has a rigorous derivation based on extremely high-fidelity assumptions, such as conservation of mass, momentum, and energy. We understand these assumptions, and we understand regimes in which using N-S would result in catastrophe (such as rarefied gases, relativistic velocities, etc.) Neural networks require data. The Navier stokes does not need to be 'trained'. Deep networks have very little a-priori knowledge baked-in (from a baysian perspective there are priors such as translation invariance that are intrinsic). They are admittedly extremely useful, because they are high dimensional (and so are universal approximators) and can be trained efficiently. Furthermore, you can develop an intuitive approach to many fluid flows. I can provide a much better estimate of the drag profile for a given wing geometry than an untrained person. No such analog is possible with deep nets, which as significantly more opaque in terms of dynamics and non-linear response. The only way his comments make any sense is if you assume he isn't talking about physical models, like Navier stokes, and instead consider turbulence models, such as RANS or LES. These are parameterized models, and are used for turbulence modeling. They also have little physical intuition. However, this is not the same as saying we do not have high confidence in physical-based systems such as Navier stokes. Source: I have a PhD in CFD and several ML publications.
- ericjang 7y ago
  Yann was not making a statement about deep nets' ability (or lack thereof) to fly a plane in a way that matches expert design. He's making the point that the ML field's obsession with causal inference (and causal discovery) is overrated, precisely because our gold standards of interpretable, safety-critical systems (airplane flight) are based on Navier Stokes/CFD. Planes were made to fly based on empirical validation of these models, long before we gained a more detailed understanding of how causality (the equations and models themselves are time reversible, implying that they contain imperfect knowledge of causality)
  - ssivark 7y ago
opless 7y ago
I challenged LeCun about explain-ability of neural nets a few years ago, he seemed to dismissed the need for it. As I recall his explanation was that the weights & configuration of the neural net is enough to give you an equation which is explanation enough. I'd link the post but it seems to have disappeared. LeCun is also heavily biased against Bayes.
- deleted 7y ago
  [deleted]
- lars 7y ago
  He's not wrong though. We know precisely how any given neural network works, it's just not understandable to us. The point is though, the functions we're trying to learn, like image classifiers, have no responsibility to us to be understandable. In fact, the brightest minds of several generations have worked hard on trying to write down rules to do image classification, and they never came up with anything that works. There is a huge space of functions that are beyond what a human can understand, where we can't write down rules to express the function. This is precisely where we need to use machine learning to find the functions. Lack of explainability is not a bug, the entire point of a neural nets is to find functions that are beyond what we can understand.
  - opless 7y ago
    Lack of explainability is not a bug, it's a wholly missing feature. There are a whole set of mathematical functions that are beyond the range of a typical human mind to understand. But there are proofs that exist that explain how they are correct. I don't think it's a big ask for a new model to be able to justify its own determinations. Of course, that makes things slightly more difficult for all those in research and those selling snake oil and everyone in between. But that's what the difference between science and alchemy is isn't it?
    - lars 7y ago
      If I show you a picture and ask you if its a dog or a cat, you can answer the question, but you can't tell me why. You can make up a story to justify why it's a dog or a cat, but if someone tries to implement the procedure you give, it won't work. If it did, we could do computer vision by implementing your justification. You are a black box AI. I can nevertheless trust you to classify dogs vs cats.
xaryk 7y ago
Not all explanations are causal. The explanation literature in the philosophy of science goes pretty far back, but here are some of the highlights: The Deductive Nomological Model (Hempel and Oppenheim, 1948) tries to explain a phenomenon using a deductive argument where the premises include particular facts and a general lawlike statement (like a law of nature) and the conclusion is the thing to be explained.[1] The Statistical Relevance Model (Wesley Salmon) attempts to fix some shortcomings in the DN model that allowed explanations using particular facts and general laws that were not at all relevant to the phenomenon being explained. The idea is that you can explain why X hasn't become pregnant by saying that X has taken birth control, and people who take birth control do not become pregnant, and that would fit the DN model, but this explanation is not statistically relevant if X is male.[2] Unificationist accounts (Philip Kitcher) seek to unify scientific explanations under a common umbrella as was done with, e.g. electromagnetism. If it is possible to have a unified theory of something, each element becomes more explainable based on its position within that unified theory [3] pragmatic and psychological accounts tend to fit more closely with the kinds of rationalizations that we've seen as some explanations of AI. They can be fictional, but they don't have to be [4] IMO we don't currently have an adequate account of explanation within the philosophy of science that works for deep neural networks. This is what my dissertation research focuses on. [1] https://en.wikipedia.org/wiki/Deductive-nomological_model https://en.wikipedia.org/wiki/Deductive-nomological_model [2] https://plato.stanford.edu/entries/scientific-explanation/#SRMod https://plato.stanford.edu/entries/scientific-explanation/#S... [3] https://plato.stanford.edu/entries/scientific-explanation/#UniAccExp https://plato.stanford.edu/entries/scientific-explanation/#U... [4] https://plato.stanford.edu/entries/scientific-explanation/#PraTheExp https://plato.stanford.edu/entries/scientific-explanation/#P...
gambler 7y ago
>Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. I nominate this as the worst analogy of the year. Airplanes are rigorously tested under the same conditions they will operate in. AI by definition is tested under conditions that are different from the environment it will operate in, because that's the whole point of AI - we want algorithms that adapt themselves to novel information.
wnoise 7y ago
I hate the comparison to how airfoils work. It's a single question that can be answered at a variety of levels, but the basics of how they work is simple and cannot be gainsaid: that they deflect air down. All the complications come in the exact details of how they deflect air down. How much is lower than ambient pressure above the wing redirecting slipstreams vs higher than ambient pressure below the wing doesn't fundamentally change the answer, though those details certainly matter (especially when designing a wing). In contrast, even for a single "AI", how it responds differently to different input is unlikely to be even remotely explainable by the same high level principles, and it's not clear that the details don't matter.
- jsmith45 7y ago
  Hmm... I'm certainly rusty on physics principles, but as far as I remember the (flawed) classic bernoulli effect explanation of airfoil (assume equal travel time ... pressure differential causes lift) does not really directly talk about moving air down at all, and would hypothetically still work if all downward air movement from the top of the wing were magically canceled out, (and airfoil had an attitude of 0 degrees, so no downward air movement from the bottom). Now granted even with that magic, there would be downward air movement relative to the airfoil, due to the airfoil rising, but that would be an effect of the lift, not a cause. Now I could well be overlooking something here, and having invoked magic to violate conservation of momentum, this thought experiment is not rigorous, and is discounting the fact that the equal time hypothesis is unfounded, and even wrong. (But in reality the air above moves faster, so we would see a larger effect than predicted, even before accounting for redirected airflow.) Of course there is no magic canceling out any downward momentum of the air from the top of the wing, and planes normally have a nose up attitude, so the bottom of the wing also directly reflects air downward, so downward directed airflow is definately also a cause of lift.
kiterunner2346 7y ago
leCun hit big when neural networks got a boost from Moore's Law and he was one of the few remaining working on them. Now he's the toast of the town. Is LeCon a genius, a drudge or does he just lack inventiveness? My guess is he's the man with a broken watch: right once b/c he was standing still. In any case le Con is unequivocally and unabashedly a one-trick (deep neural nets) pony: https://www.technologyreview.com/s/608911/is-ai-riding-a-one-trick-pony/ https://www.technologyreview.com/s/608911/is-ai-riding-a-one... Does le Con really believe that NN are the be-all and end-all of AI? How many here seriously believe that NNs will be the last, or even most important, hurdle towards AI?
omarhaneef 7y ago
Interesting exchange in the comments: Gaurav Mukherjee: Well argued! Yes, it is true that we don’t have causal relationships for a number of phenomena. But the absence of evidence is not the evidence of absence. So corner cases of failure may exist in all these phenomena which can cost lives. Does this mean that we halt the fast paced progress of AI research or any other scientific pursuit? No! But leveraging empirical evidence that can’t be fully explained in situations where lives are at stake, should require a very high bar of regulation. Responsible scientists and engineers agree that causality is important to understand and they do all they can to understand how systems work. However there are likely many among us who do not employ similar standards to the application of ill understood techniques. When it comes to regulation, we must pay heed to the worst in us. Yann LeCun: Actually, there is no clear definition of causality in classical and quantum physics, simply because the elementary equations of motions are time reversible (if you also reverse charge and parity). For every phenomenon that can occur, the same phenomenon can occur backwards in time (with corresponding anti-particles). Take a Feynman diagram where an electron and a positron annihilate to produce a gamma ray photon. It can be interpreted as a gamma ray photon spontaneously creating an electron-positron pair. It's the same diagram where time goes right to left instead of left to right. How can one possibly say that A causes B, when B could very have caused A in a time-reversed view point? Even worse, most physical phenomena have loopy causal graphs. Motion "causes" friction. But friction limits the velocity of motion. Most differential equations have coupled terms with loopy interactions in which quantity x affects quantity y and vice versa. You rarely have y(t+dt)=f(x(t)) in physics. More often than not, you have coupled equations y(t+dt)=f(x(t)) and x(t+dt)=g(y(t)). In these (frequent) cases, x cause y, but y also causes x. There is something like that going on in fluid dynamics, which is why it's difficult to come up with "simple" causal explanations. Only when collective phenomena are considered does the "arrow of time" appear to have a definite direction (with the 2nd law of thermodynamics). Edit: para spacing
- omarhaneef 7y ago
  Placing this in a comment in case someone else is interested in some other aspect, but I am confused by this time-reversal point. Yes, the equation can go in either direction but time itself -- as we understand it -- goes in one direction, basically divided by our memory. So why not just stipulate that the equation is one directional, and there is another (inverse, but otherwise identical) equation for the reverse process?
6gvONxR4sf7o 7y ago
Okay, I feel like a crazy person arguing against a luminary of the field, but this is so misleading as to be disingenuous and make me trust the guy less. He's throwing causality and explainability into the same bucket and arguing against the need for the latter. >We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. >Yes, that can be useful. Or at least reassuring. >But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation. It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships. >Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. >But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence. The physical models we have are causal ones. The intuitive abstractions like bernoulli's principle may not work, but analysis based on navier stokes sure does. You plug your shape (cause) into the equations and see what forces (effect) occur. That's causation. >You know what other life-critical phenomena we don't have good causal explanations for? > The mechanism of action of many drugs (if not most of them). Using an industry that's nearly synonymous with a randomized controlled trial as a refutation for the need of a causal relationship is crazy talk. The mechanism may be missing, but the causal explanation is that via a series of RCTs it's established that the drug causes the effects. I get that half of this is trying to go against a percieved need for intuitive explanations, but it weirdly lumps causation in there.
- nabla9 7y ago
  > The physical models we have are causal ones. The question is do you need a detailed causal explanation you can understand. In the case of lift, you can simulate aircraft model and calculate the change of vertical momentum in particles going above the wing and below the wing. Usually 80% of the lift comes from above the wing. If you are modelling a brick, it's the other way around. But do aircraft designers need to know that or are they just satisfied with less?
  - 6gvONxR4sf7o 7y ago
    No I don't think that is the question. The questions (plural) are whether you need detailed explanations or intuitively understandable explanations or explanations at all, as well as whether you have to build a detailed causal model or an intuitive causal model or establish causation at all. Aircraft designers are happy to test something in a wind tunnel and establish that the change in shape caused a change in performance. But they knew to try that particular new shape that because of all the physical (causal) models at various levels of intuitive understandability.
tomrod 7y ago
For those interested in a great read on why causality really does matter, give "The Book of Why" a read. I'm not convinced Judea Pearl's modeling approach is the most rigorous, but it does a clear and convincing job putting causality (esp. w/ data fusion!) at the heart of modern systems (including ML). For those on a more mathematical bend, check out "Causality" by the same author or "Causal Inference for Statistics, Social, and Biomedical Sciences" by Imbens and Rubin
huyegn 7y ago
I agree with Yann here ... I think the demand for explainability is like a person asking for a "faster horse" when what they really need is a car. When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates . Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it. For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know! I've been doing a review of techniques that are becoming more popular in this area: https://blog.everyhue.me/posts/why-uncertainty-matters/ https://blog.everyhue.me/posts/why-uncertainty-matters/
- wnoise 7y ago
  I think that unlike Yann, you're making a good point.
- opless 7y ago
  I disagree with LeCun but you have a good point. Telling you how much uncertainty there is in a result is a useful metric. But it is only part of what is required (by my definition of explainability). My counterpoint is when a model trained on dog breeds misidentifies a breed with a high certainty factor, the model should be able to justify (explain) why it made that judgement.
  - huyegn 7y ago
    That's an interesting idea and I get the motivation. I often wonder if we're already able explain predictions. There are things like gradcam to determine which inputs caused which activations. It's just that the explanations aren't very understandable or satisfying from a human perspective.
    - naresh_xai 7y ago
      You should definitely look at advances in the field then. There a lot more promising work beyond grad cam. And there are a few techniques for human oriented explanations - namely TCAV (human oriented explanations) and PatternNet/PatternAttribution as well as contrastive LRP (in terms of much more informative heatmapping techniques which are dependent on intermediate layers as well.)
halayli 7y ago
I wonder where/if incompleteness theorem fits in this picture. https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
joe_the_user 7y ago
From article: "How does lithium treat bipolar disorder?" The thing is, that this isn't a question that's insignificant. There has been quite a bit of debate whether various drugs reverse the basic process happening with a given psychiatric disorder or whether they provide a different change which allows a person function. The phrase "chemical imbalance" has been based on the supposition that various drugs that change brain chemical distributions "cure" various conditions - but the question these drug are directly reversing a condition or whether they are adding something more seems important, even if we assume the drugs are broadly useful for helping people function in society.
screye 7y ago
Yann is undoubtedly a pillar of the deep learning era of AI. But, I strongly disagree with what he is saying. Firstly, the Navier-stokes equations existed before mechanized flight was discovered. It was not explicitly invented to reason about flight, in that way that some of the deep learning "theory" is being projected. It also worked perfectly (in sub-sonic speeds) for almost every situation involving flight. ML has time and again proven to not be internally consistent, with contradictions on nearly every corner. Batch norm was "theoretically" thought to be an essential part of neural networks, until suddenly it wasn't. Drop out was considered essential to avoid overfitting, until it wasn't. We still do not know if an information bottle neck is good or bad. ADAM's whole formula was incorrect and no one realized that for 6 years. > The mechanism of action of many drugs (if not most of them). > An example? How does lithium treat bipolar disorder? This example does more to disapprove his point that approve of it. Medical and nutritional sciences are among the least understood fields out there, with some of their "fundamentals" flipping on their head over the last 50 years. The only reason these "sciences" continue being used is because medicine is essential. A reasonably effective solution with side effects is still better than dead people. It is a begrudging compromise, not a example to be emulated. AI is not essential. AI imposes on your life without consent. AI will soon be ingrained into every single aspect of your life. Yann seems to be conflating explainability with causality. Explanaibility can also mean fully observed correlation. Explainbility can mean predictable and reproducible behavior of ML models given a hypothesis. Explanability can mean the ability to ascertain if the change in model performance was because of the hypothesis or a way to exploit an unintended aspect of the data/model architecture. Explainability fundamentally allows ML researchers to make strides in the field in a meaningful way, and not blindly throw different computational structures at thousands of GPUs and let luck of the draw decide what works and when. Looking back at ideas such as Transformers and ResNets, there was literally no way for the authors to guess that this new computational structure would revolutionize the field. It could easily have been an idea someone else tried or rotted in someone's TO-TRY list. Explainbility and some theoretical logic around NN development would allow for a systematic way to go about research in the ML community as whole. That's unlikely to happen, but I would rather see people strive for it than not.
naresh_xai 7y ago
When a person’s claim to fame and research is dependent on ignoring explainability and causality in research, he will ignore it to the best of his means. To him, all the precursors to clinical trials (selection of a molecule from restricted set of molecules which satisfy certain causal criteria) then followed by extensive multi million dollar experiments to safeguard the last 1-2% of uncertainty is equivalent to a barely preprocessed neural network. I mean let’s ignore the basic principles of pharmacology and medicine and just run every possible molecule through humans since that is a valid approach according to him.
haecceity 7y ago
In re airplanes, deflecting air down is a perfectly intuitive explanation. Change in momentum of air means change in momentum of wing in opposite direction by the third law.
leereeves 7y ago
Yann LeCun and others debated this for an hour at NeurIPS 2017. https://www.youtube.com/watch?v=93Xv8vJ2acI https://www.youtube.com/watch?v=93Xv8vJ2acI There's a lot of good points in there from both sides, but what really stuck with me is that given the choice between an explainable model and a black box model that works better (more accurate predictions), most people choose the black box model.
- opless 7y ago
  I think your insight is a perfect description of humans authority bias. People love to be told what to do. Even if in the face of overwhelming real world evidence people blindly follow their GPS, and to ignore it can cause a lot of stress.
- naresh_xai 7y ago
  Which is strongly misleading tradeoff. For a ton of tasks, deep learning methods are no better than white box regression or tree ensemble methods. And there is no reason to expect that a deep learning model has to be unexplainable. He’s putting up a mystical interpretability vs accuracy tradeoff which does not exist. There is a ton of research out there making deep neural networks (slightly to significantly interpretable)
anthony_doan 7y ago
Yeah AI world have a loose definition of explanability and interpretablity. I also see this very dogmatic mindset that Deep Learning will do prediction and interpretability. What is stopping you from building two models? a regression statistic model to do interpretability/explanability and another deep learning for prediction? Like each coefficient in a regression have a t-test for significant in correlation with response. You don't have something like that in deep learning. Also I've seen many MLer use logistic regression as a classifier and ignoring the probability aspect like the Titanic dataset highlight the different mindset between statistician and ML. ML often will see this as a classify problem dead or not dead. Statistician will phrase to "What's the probability of this person dying with these covariates?" You know why this matter? It really matter in health/medical/social science. Often time inference is what they want and they need to know what affect your health not just shoving in tons of data and covariates/features. Not only that you many not even have enough data for these data hungry ML models. Another example is biostatistician figure out threshold between the benefit of taking an invasive procedure versus not taking it. We figure it out but giving a percentage and the doctor and experts will tell you where the threshold is, 20%, 40%? It's certainly not 50% that many MLer do. > We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. Yes, that can be useful. Or at least reassuring. To me this just an excuse to not learn statistic. He should really look into Propensity modeling under Rubin-Neymer causality model. This is what statistic is going into after regression for observational data. With all the criticism I have for ML. I think it's just the mind set. I think the ML algorithms have a place and they're very good in certain domain such as NLP and computer vision. But to act as if they're the end all be all when statistic models have been there and use extensive in biostatistic and econometric fields is just hubris and ignorance. While ML is making excuses for causality. Econometric and statistician are working to build causality model. IIRC econometric is doing structure equation while statistician are going for Rubin-Neyman model. There is debate on which model is better but that's ongoing we'll wait and see from all the research papers.
datastoat 7y ago
If Facebook lawyers understood the implications of LeCun's argument, they wouldn't be happy! There are two types of explanations here: (1) why did the data come to be as it is, (2) why did my ML make the prediction it did. Science looks for the answer to (1), and causal models are a great way to think about it. Science and engineering, when they go hand in hand, build a machine by saying "Here is data, let me do science to understand nature's underlying laws, and my machine shall be based on those laws". The machine is inherently explainable because it's based on scientific laws. In the ML world, we can bypass the "learn scientific laws" part, and jump straight to "build a machine based on data". So the best answer to (2) has got to be "my ML made the prediction it did because of its training data". As Pearl said, ML is just curve fitting, so the only way to "explain" a ML prediction is to say "here are the points that the curve was fitted to". Prediction is just reading a value off the curve. Think the machine is biased? Look for bias in the training dataset! Think the machine is inaccurate? Look for sparsity or conflict in the training dataset! So the consequence of LeCun's distinction is that, when the GDPR calls for explainability of ML decision making, it is really calling for sharing of the training data. Facebook, watch out!
Rumudiez 7y ago
The word the author was looking for when they wrote "explainability" is actually "explicable," as in something that can be explained, and is converse to the more commonly used word "inexplicable."
- thanatropism 7y ago
  Good point. Compare with efficiency vs efficacy.
monadic2 7y ago
How do you validate the model without understanding where you might find problematic results? That doesn’t require a narrative, maybe, but it does require not using black-box-only testing.

148 points by metahost 7y ago

It depends, while Yann Lecun has an interesting point of view. The argument for explainability depends on the risk of harm from an AI model decision. Explaining why airplanes fly is moderately interesting but why did the 737-max crash is much more interesting and needed. While the former is probably only needed for people studying aerodynamics, the latter is meant for passengers, regulators, airlines, governments, etc. Here is a tweet thread we posted in the past: https://twitter.com/krishnagade/status/1182317508667133952 https://twitter.com/krishnagade/status/1182317508667133952

RookyNumbas 7y ago

I'm surprised he made the drug analogy. It takes decades and hundreds of millions dollars to be able to test a drug on humans. I'm curious if he thinks we need a similar safeguard for all AI-human interactions.

PaulRobinson 7y ago
It takes decades and hundreds of millions of dollars in order to build a statistical argument that explains likely effects, side-effects and risks of prescribing that drug to the population at large. It takes that long and costs that much because there is no safe way to do this with human physiology other than to carefully, slowly and expensively try it and measure the consequences. We can take any AI system and model its behaviour with statistical certainty much more cheaply and quickly, and be more confident in its future behaviour. Remember, AI and ML are fancy words for some form of a regression to a mean more often than not. If we run hundreds of millions of tests of such a system in simulation for a very wide range of contexts/inputs (which is cheap to do), we can have a much higher degree of confidence in a short space of time around behaviour of that system than we will for any drug test, even if we still don't fully understand the causality.
- RookyNumbas 7y ago
  In the case of a go-playing AI, and maybe even things like self driving cars, then yes I agree. But we are quickly moving beyond the types of applications where you can simply test in simulation. Self driving cars are probably right at the edge of this. It's what comes next that worries me. And I don't pretend to know what it will be.
- naresh_xai 7y ago
  If you were ever involved in the drug discovery process, then you would know that statistical evidence through clinical trials is only for the last ounce of drug testing. Out of 10,000+ drug molecule candidates only ~ 5 molecules get selected for clinical trials. A lot of molecules are removed as candidates for toxic components/ solubility issues/ chemical stability issues etc. So even when we do not understand the precise impact of drugs on humans and there is no safer mechanism to test, we leave only 0.5% of candidate molecules to empirical/statistical evidence in the form of clinical trials. On the other hand if drug discovery was treated as a pure AI problem, we would have thousands of unverified and unsafe molecules in clinical trials. Causal principles get us to 99.5% of the way in drug discovery. Unfortunately not so in AI.
  - PaulRobinson 7y ago
    I'm not involved in the drug discovery process, but I can imagine that manufacturers would be interested in helping to use AI to deal with the funnel much more effectively to get to those ~5 molecules much more efficiently. You're still left with double-blind trials and having to get large sample groups to try those molecules though. And it's for that reason that drug discovery is always likely to be quite slow, complex and expensive - the efficiency gains will be pushed towards the top of the funnel to make new ideas reasonable to explore, I would imagine. My point was that when you're not dealing with human physiology and instead dealing with problems that are more tractable through AI - i.e. using regression to tune algorithms through patterns in data - you are going to get quicker and more impactful returns without the same complexity. And - critically - it's OK to often trust the AI solution you have without understanding causality. If you later find it's doing something odd that is undesirable, you can use that data to help tune the algorithm again without having to understand the causal relationship. Put another way, you can teach an AI to get better without necessarily understanding the subject completely yourself.

heyitsguay 7y ago

I think the type of explainability Yann LeCun is describing in this post is not the same as what people really want when they talk about explainable AI. To use his example, I think most people would be fine with an AI that said "we're doing X because that's what heuristic Y prescribes for the situation", and getting from where we are now to something like that is the challenge. FWIW i see a similar dialogue in biomedical contexts quite a bit, where there's some cool but hard-to-trust work being done in data-driven image restoration (https://www.biorxiv.org/content/10.1101/236463v5.abstract https://www.biorxiv.org/content/10.1101/236463v5.abstract). There's still a lot that could be done short of programs providing a priori mathematical proof for all models.

buboard 7y ago
> because that's what heuristic Y prescribes for the situation people will keep asking "why" and keep escalating, but that's a flawed approach to evaluating a model. every answer is literally constructed and thus can't be trusted on its own. The best that an evaluator can do is to evaluate the training dataset

pesenti 7y ago

For those not on FB: Link: https://www.scientificamerican.com/article/no-one-can-explain-why-planes-stay-in-the-air/?fbclid=IwAR0-S3nl5P0-vKUsr0oN5eeXmTVPTHMRZ9L_5bDqQStD-7uEHdukXC5JR6o https://www.scientificamerican.com/article/no-one-can-explai... "We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. Yes, that can be useful. Or at least reassuring. But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation. In many cases of physical phenomena, "explanations" contain causal loops where A causes B and B causes A. A good example is how a wing causes lift. The computational fluid dynamics model, based on Navier-Stokes equations, works just fine. But there is no completely-accurate intuitive "explanation" of why airplanes fly. Is it because of Bernoulli principle? Because a wing deflects the air downwards? Because the air above the wing want to keep going straight but by doing so creates a low-pressure region above the wing that forces the flow downwards sucks the wing upwards? All of the above, but none of the above by itself. Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence. You know what other life-critical phenomena we don't have good causal explanations for? The mechanism of action of many drugs (if not most of them). An example? How does lithium treat bipolar disorder? We do have considerable empirical evidence provided by extensive clinical studies. This is not to say that causality is not an important area of research for AI. It is. But sometimes, requiring explanability is counterproductive."

deleted 7y ago
[deleted]

krnsll 7y ago

Would be interested to see what Judea Pearl says about this. Not on twitter any more but back when I was, I recall him regularly tweeting about causality and AI's shortcomings in that regard.

pesenti 7y ago

Einstein had a great quote: "The most incomprehensible thing about the world is that it is comprehensible." Well, turns out, a lot of the world is not comprehensible. And that's what made Einstein irrelevant for the latter part of his life. Our overestimation of the comprehensibility of the world may very well be some version of the Drunkard's search principle. We are much more likely to know about what's comprehensible than what's not.

buboard 7y ago
that's a misstatement of einstein's work. he didn't believe the world is incomprehensible, rather that it's not inherently random.
drcode 7y ago
As far as I know, a lot of the criticism Einstein leveled against the foundations of quantum mechanics later in his career are more recently being revisited... instead of being irrelevant, some of his later work may simply have been too far ahead of its time.

buboard 7y ago

he s probably correct. A lot of what we call "human explanation" is in reality a rationalized version of a hunch, but not necessarily the best representation of it. At most situations, some part of the brain crosses a threshold and causes a decision or action, but it's rather rare that humans can explain that action correctly. It's also rare that causation can be established from unambiguous temporal order (e.g. clouds/rain); in most cases our rationalizations are post-hoc. We get a glimpse of it from recent language models: It's become rather easy to start blurting out language that is convincibly and comfortingly coherent, and it can be nudged to point to one direction or another. that doesnt mean it's true

gatherhunterer 7y ago

Hopefully a mod can change the link. > Please submit the original source. If a post reports on something found on another site, submit the latter.

albertzeyer 7y ago
This is less about the article linked by Yann, but more about what Yann writes. He basically uses the argument made in the article, and adopts it to make a similar argument about AI. And that this argument about AI is the interesting content here (although the linked article is also interesting).

Antwan 7y ago

This guy is a total genius. Shame he doesn't care more about who he is working for and what his researches are used for.

bordercases 7y ago

This is true for the strict Pearl-style form of causation. The concept of what counts as a causative explanation can be more expansive, and it varies between disciplines. See the work of Nancy Cartwright. https://www.researchgate.net/publication/30527010_What_Is_Wrong_With_Bayes_Nets https://www.researchgate.net/publication/30527010_What_Is_Wr... TL;DR we've been explaining "causes" without Bayes Nets for awhile, Bayes Nets unsubtly disregards the common-sense logic scientists use for their practice, including the way that explanations tend to be qualified by context.

t_serpico 7y ago

As a somewhat outsider to deep learning, intuitively it seems to be true that if you were able to demystify the black box, then it would be easier to improve your models (as you understand where it succeeds and fails and why). From this perspective, explainability would be incredibly productive.

b3kart 7y ago
I am not convinced the explanations we’d get from these models would be interpretable enough to drive their conceptual improvement. Deep nets just don’t learn nice interpretable features, nothing in the training objective makes them do so. E.g. they make use of really dubious features [1], and might not actually learn “hierarchies” of features as we thought they were [2]. [1] https://arxiv.org/abs/1905.02175 https://arxiv.org/abs/1905.02175 [2] https://arxiv.org/abs/1904.00760 https://arxiv.org/abs/1904.00760
- naresh_xai 7y ago
  Or you can actually visualize and quantify impact of aggregates of meaningful features within networks using the methodology described in NetDissect or in TCAV. But of course, that casts doubt into a lot of claimed mechanisms and tons of ML research as I have found out.

carlosdp 7y ago

Another more basic example is gravity. We have a statistical intuition of gravity, we depend on it being there and acting in a consistent way for almost everything in physics. But we don't have an accurate understanding of how gravity works or why it's there. We have theories, but nothing proven. But it's so easy to prove gravity's properties (on Earth) by consistent results with experimentation, that it's literally a science project in every elementary school.

arcanus 7y ago

> Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence This is argument is complete nonsense. Navier Stokes has a rigorous derivation based on extremely high-fidelity assumptions, such as conservation of mass, momentum, and energy. We understand these assumptions, and we understand regimes in which using N-S would result in catastrophe (such as rarefied gases, relativistic velocities, etc.) Neural networks require data. The Navier stokes does not need to be 'trained'. Deep networks have very little a-priori knowledge baked-in (from a baysian perspective there are priors such as translation invariance that are intrinsic). They are admittedly extremely useful, because they are high dimensional (and so are universal approximators) and can be trained efficiently. Furthermore, you can develop an intuitive approach to many fluid flows. I can provide a much better estimate of the drag profile for a given wing geometry than an untrained person. No such analog is possible with deep nets, which as significantly more opaque in terms of dynamics and non-linear response. The only way his comments make any sense is if you assume he isn't talking about physical models, like Navier stokes, and instead consider turbulence models, such as RANS or LES. These are parameterized models, and are used for turbulence modeling. They also have little physical intuition. However, this is not the same as saying we do not have high confidence in physical-based systems such as Navier stokes. Source: I have a PhD in CFD and several ML publications.

ericjang 7y ago
Yann was not making a statement about deep nets' ability (or lack thereof) to fly a plane in a way that matches expert design. He's making the point that the ML field's obsession with causal inference (and causal discovery) is overrated, precisely because our gold standards of interpretable, safety-critical systems (airplane flight) are based on Navier Stokes/CFD. Planes were made to fly based on empirical validation of these models, long before we gained a more detailed understanding of how causality (the equations and models themselves are time reversible, implying that they contain imperfect knowledge of causality)
- ssivark 7y ago

opless 7y ago

I challenged LeCun about explain-ability of neural nets a few years ago, he seemed to dismissed the need for it. As I recall his explanation was that the weights & configuration of the neural net is enough to give you an equation which is explanation enough. I'd link the post but it seems to have disappeared. LeCun is also heavily biased against Bayes.

deleted 7y ago
[deleted]
lars 7y ago
He's not wrong though. We know precisely how any given neural network works, it's just not understandable to us. The point is though, the functions we're trying to learn, like image classifiers, have no responsibility to us to be understandable. In fact, the brightest minds of several generations have worked hard on trying to write down rules to do image classification, and they never came up with anything that works. There is a huge space of functions that are beyond what a human can understand, where we can't write down rules to express the function. This is precisely where we need to use machine learning to find the functions. Lack of explainability is not a bug, the entire point of a neural nets is to find functions that are beyond what we can understand.
- opless 7y ago
  Lack of explainability is not a bug, it's a wholly missing feature. There are a whole set of mathematical functions that are beyond the range of a typical human mind to understand. But there are proofs that exist that explain how they are correct. I don't think it's a big ask for a new model to be able to justify its own determinations. Of course, that makes things slightly more difficult for all those in research and those selling snake oil and everyone in between. But that's what the difference between science and alchemy is isn't it?
  - lars 7y ago
    If I show you a picture and ask you if its a dog or a cat, you can answer the question, but you can't tell me why. You can make up a story to justify why it's a dog or a cat, but if someone tries to implement the procedure you give, it won't work. If it did, we could do computer vision by implementing your justification. You are a black box AI. I can nevertheless trust you to classify dogs vs cats.

xaryk 7y ago

Not all explanations are causal. The explanation literature in the philosophy of science goes pretty far back, but here are some of the highlights: The Deductive Nomological Model (Hempel and Oppenheim, 1948) tries to explain a phenomenon using a deductive argument where the premises include particular facts and a general lawlike statement (like a law of nature) and the conclusion is the thing to be explained.[1] The Statistical Relevance Model (Wesley Salmon) attempts to fix some shortcomings in the DN model that allowed explanations using particular facts and general laws that were not at all relevant to the phenomenon being explained. The idea is that you can explain why X hasn't become pregnant by saying that X has taken birth control, and people who take birth control do not become pregnant, and that would fit the DN model, but this explanation is not statistically relevant if X is male.[2] Unificationist accounts (Philip Kitcher) seek to unify scientific explanations under a common umbrella as was done with, e.g. electromagnetism. If it is possible to have a unified theory of something, each element becomes more explainable based on its position within that unified theory [3] pragmatic and psychological accounts tend to fit more closely with the kinds of rationalizations that we've seen as some explanations of AI. They can be fictional, but they don't have to be [4] IMO we don't currently have an adequate account of explanation within the philosophy of science that works for deep neural networks. This is what my dissertation research focuses on. [1] https://en.wikipedia.org/wiki/Deductive-nomological_model https://en.wikipedia.org/wiki/Deductive-nomological_model [2] https://plato.stanford.edu/entries/scientific-explanation/#SRMod https://plato.stanford.edu/entries/scientific-explanation/#S... [3] https://plato.stanford.edu/entries/scientific-explanation/#UniAccExp https://plato.stanford.edu/entries/scientific-explanation/#U... [4] https://plato.stanford.edu/entries/scientific-explanation/#PraTheExp https://plato.stanford.edu/entries/scientific-explanation/#P...

gambler 7y ago

>Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. I nominate this as the worst analogy of the year. Airplanes are rigorously tested under the same conditions they will operate in. AI by definition is tested under conditions that are different from the environment it will operate in, because that's the whole point of AI - we want algorithms that adapt themselves to novel information.

wnoise 7y ago

I hate the comparison to how airfoils work. It's a single question that can be answered at a variety of levels, but the basics of how they work is simple and cannot be gainsaid: that they deflect air down. All the complications come in the exact details of how they deflect air down. How much is lower than ambient pressure above the wing redirecting slipstreams vs higher than ambient pressure below the wing doesn't fundamentally change the answer, though those details certainly matter (especially when designing a wing). In contrast, even for a single "AI", how it responds differently to different input is unlikely to be even remotely explainable by the same high level principles, and it's not clear that the details don't matter.

jsmith45 7y ago
Hmm... I'm certainly rusty on physics principles, but as far as I remember the (flawed) classic bernoulli effect explanation of airfoil (assume equal travel time ... pressure differential causes lift) does not really directly talk about moving air down at all, and would hypothetically still work if all downward air movement from the top of the wing were magically canceled out, (and airfoil had an attitude of 0 degrees, so no downward air movement from the bottom). Now granted even with that magic, there would be downward air movement relative to the airfoil, due to the airfoil rising, but that would be an effect of the lift, not a cause. Now I could well be overlooking something here, and having invoked magic to violate conservation of momentum, this thought experiment is not rigorous, and is discounting the fact that the equal time hypothesis is unfounded, and even wrong. (But in reality the air above moves faster, so we would see a larger effect than predicted, even before accounting for redirected airflow.) Of course there is no magic canceling out any downward momentum of the air from the top of the wing, and planes normally have a nose up attitude, so the bottom of the wing also directly reflects air downward, so downward directed airflow is definately also a cause of lift.

kiterunner2346 7y ago

leCun hit big when neural networks got a boost from Moore's Law and he was one of the few remaining working on them. Now he's the toast of the town. Is LeCon a genius, a drudge or does he just lack inventiveness? My guess is he's the man with a broken watch: right once b/c he was standing still. In any case le Con is unequivocally and unabashedly a one-trick (deep neural nets) pony: https://www.technologyreview.com/s/608911/is-ai-riding-a-one-trick-pony/ https://www.technologyreview.com/s/608911/is-ai-riding-a-one... Does le Con really believe that NN are the be-all and end-all of AI? How many here seriously believe that NNs will be the last, or even most important, hurdle towards AI?

omarhaneef 7y ago

Interesting exchange in the comments: Gaurav Mukherjee: Well argued! Yes, it is true that we don’t have causal relationships for a number of phenomena. But the absence of evidence is not the evidence of absence. So corner cases of failure may exist in all these phenomena which can cost lives. Does this mean that we halt the fast paced progress of AI research or any other scientific pursuit? No! But leveraging empirical evidence that can’t be fully explained in situations where lives are at stake, should require a very high bar of regulation. Responsible scientists and engineers agree that causality is important to understand and they do all they can to understand how systems work. However there are likely many among us who do not employ similar standards to the application of ill understood techniques. When it comes to regulation, we must pay heed to the worst in us. Yann LeCun: Actually, there is no clear definition of causality in classical and quantum physics, simply because the elementary equations of motions are time reversible (if you also reverse charge and parity). For every phenomenon that can occur, the same phenomenon can occur backwards in time (with corresponding anti-particles). Take a Feynman diagram where an electron and a positron annihilate to produce a gamma ray photon. It can be interpreted as a gamma ray photon spontaneously creating an electron-positron pair. It's the same diagram where time goes right to left instead of left to right. How can one possibly say that A causes B, when B could very have caused A in a time-reversed view point? Even worse, most physical phenomena have loopy causal graphs. Motion "causes" friction. But friction limits the velocity of motion. Most differential equations have coupled terms with loopy interactions in which quantity x affects quantity y and vice versa. You rarely have y(t+dt)=f(x(t)) in physics. More often than not, you have coupled equations y(t+dt)=f(x(t)) and x(t+dt)=g(y(t)). In these (frequent) cases, x cause y, but y also causes x. There is something like that going on in fluid dynamics, which is why it's difficult to come up with "simple" causal explanations. Only when collective phenomena are considered does the "arrow of time" appear to have a definite direction (with the 2nd law of thermodynamics). Edit: para spacing

omarhaneef 7y ago
Placing this in a comment in case someone else is interested in some other aspect, but I am confused by this time-reversal point. Yes, the equation can go in either direction but time itself -- as we understand it -- goes in one direction, basically divided by our memory. So why not just stipulate that the equation is one directional, and there is another (inverse, but otherwise identical) equation for the reverse process?

6gvONxR4sf7o 7y ago

Okay, I feel like a crazy person arguing against a luminary of the field, but this is so misleading as to be disingenuous and make me trust the guy less. He's throwing causality and explainability into the same bucket and arguing against the need for the latter. >We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. >Yes, that can be useful. Or at least reassuring. >But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation. It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships. >Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing. >But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence. The physical models we have are causal ones. The intuitive abstractions like bernoulli's principle may not work, but analysis based on navier stokes sure does. You plug your shape (cause) into the equations and see what forces (effect) occur. That's causation. >You know what other life-critical phenomena we don't have good causal explanations for? > The mechanism of action of many drugs (if not most of them). Using an industry that's nearly synonymous with a randomized controlled trial as a refutation for the need of a causal relationship is crazy talk. The mechanism may be missing, but the causal explanation is that via a series of RCTs it's established that the drug causes the effects. I get that half of this is trying to go against a percieved need for intuitive explanations, but it weirdly lumps causation in there.

nabla9 7y ago
> The physical models we have are causal ones. The question is do you need a detailed causal explanation you can understand. In the case of lift, you can simulate aircraft model and calculate the change of vertical momentum in particles going above the wing and below the wing. Usually 80% of the lift comes from above the wing. If you are modelling a brick, it's the other way around. But do aircraft designers need to know that or are they just satisfied with less?
- 6gvONxR4sf7o 7y ago
  No I don't think that is the question. The questions (plural) are whether you need detailed explanations or intuitively understandable explanations or explanations at all, as well as whether you have to build a detailed causal model or an intuitive causal model or establish causation at all. Aircraft designers are happy to test something in a wind tunnel and establish that the change in shape caused a change in performance. But they knew to try that particular new shape that because of all the physical (causal) models at various levels of intuitive understandability.

tomrod 7y ago

For those interested in a great read on why causality really does matter, give "The Book of Why" a read. I'm not convinced Judea Pearl's modeling approach is the most rigorous, but it does a clear and convincing job putting causality (esp. w/ data fusion!) at the heart of modern systems (including ML). For those on a more mathematical bend, check out "Causality" by the same author or "Causal Inference for Statistics, Social, and Biomedical Sciences" by Imbens and Rubin

huyegn 7y ago

I agree with Yann here ... I think the demand for explainability is like a person asking for a "faster horse" when what they really need is a car. When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates . Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it. For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know! I've been doing a review of techniques that are becoming more popular in this area: https://blog.everyhue.me/posts/why-uncertainty-matters/ https://blog.everyhue.me/posts/why-uncertainty-matters/

wnoise 7y ago
I think that unlike Yann, you're making a good point.
opless 7y ago
I disagree with LeCun but you have a good point. Telling you how much uncertainty there is in a result is a useful metric. But it is only part of what is required (by my definition of explainability). My counterpoint is when a model trained on dog breeds misidentifies a breed with a high certainty factor, the model should be able to justify (explain) why it made that judgement.
- huyegn 7y ago
  That's an interesting idea and I get the motivation. I often wonder if we're already able explain predictions. There are things like gradcam to determine which inputs caused which activations. It's just that the explanations aren't very understandable or satisfying from a human perspective.
  - naresh_xai 7y ago
    You should definitely look at advances in the field then. There a lot more promising work beyond grad cam. And there are a few techniques for human oriented explanations - namely TCAV (human oriented explanations) and PatternNet/PatternAttribution as well as contrastive LRP (in terms of much more informative heatmapping techniques which are dependent on intermediate layers as well.)

halayli 7y ago

I wonder where/if incompleteness theorem fits in this picture. https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...

joe_the_user 7y ago

From article: "How does lithium treat bipolar disorder?" The thing is, that this isn't a question that's insignificant. There has been quite a bit of debate whether various drugs reverse the basic process happening with a given psychiatric disorder or whether they provide a different change which allows a person function. The phrase "chemical imbalance" has been based on the supposition that various drugs that change brain chemical distributions "cure" various conditions - but the question these drug are directly reversing a condition or whether they are adding something more seems important, even if we assume the drugs are broadly useful for helping people function in society.

screye 7y ago

Yann is undoubtedly a pillar of the deep learning era of AI. But, I strongly disagree with what he is saying. Firstly, the Navier-stokes equations existed before mechanized flight was discovered. It was not explicitly invented to reason about flight, in that way that some of the deep learning "theory" is being projected. It also worked perfectly (in sub-sonic speeds) for almost every situation involving flight. ML has time and again proven to not be internally consistent, with contradictions on nearly every corner. Batch norm was "theoretically" thought to be an essential part of neural networks, until suddenly it wasn't. Drop out was considered essential to avoid overfitting, until it wasn't. We still do not know if an information bottle neck is good or bad. ADAM's whole formula was incorrect and no one realized that for 6 years. > The mechanism of action of many drugs (if not most of them). > An example? How does lithium treat bipolar disorder? This example does more to disapprove his point that approve of it. Medical and nutritional sciences are among the least understood fields out there, with some of their "fundamentals" flipping on their head over the last 50 years. The only reason these "sciences" continue being used is because medicine is essential. A reasonably effective solution with side effects is still better than dead people. It is a begrudging compromise, not a example to be emulated. AI is not essential. AI imposes on your life without consent. AI will soon be ingrained into every single aspect of your life. Yann seems to be conflating explainability with causality. Explanaibility can also mean fully observed correlation. Explainbility can mean predictable and reproducible behavior of ML models given a hypothesis. Explanability can mean the ability to ascertain if the change in model performance was because of the hypothesis or a way to exploit an unintended aspect of the data/model architecture. Explainability fundamentally allows ML researchers to make strides in the field in a meaningful way, and not blindly throw different computational structures at thousands of GPUs and let luck of the draw decide what works and when. Looking back at ideas such as Transformers and ResNets, there was literally no way for the authors to guess that this new computational structure would revolutionize the field. It could easily have been an idea someone else tried or rotted in someone's TO-TRY list. Explainbility and some theoretical logic around NN development would allow for a systematic way to go about research in the ML community as whole. That's unlikely to happen, but I would rather see people strive for it than not.

naresh_xai 7y ago

When a person’s claim to fame and research is dependent on ignoring explainability and causality in research, he will ignore it to the best of his means. To him, all the precursors to clinical trials (selection of a molecule from restricted set of molecules which satisfy certain causal criteria) then followed by extensive multi million dollar experiments to safeguard the last 1-2% of uncertainty is equivalent to a barely preprocessed neural network. I mean let’s ignore the basic principles of pharmacology and medicine and just run every possible molecule through humans since that is a valid approach according to him.

haecceity 7y ago

In re airplanes, deflecting air down is a perfectly intuitive explanation. Change in momentum of air means change in momentum of wing in opposite direction by the third law.

leereeves 7y ago

Yann LeCun and others debated this for an hour at NeurIPS 2017. https://www.youtube.com/watch?v=93Xv8vJ2acI https://www.youtube.com/watch?v=93Xv8vJ2acI There's a lot of good points in there from both sides, but what really stuck with me is that given the choice between an explainable model and a black box model that works better (more accurate predictions), most people choose the black box model.

opless 7y ago
I think your insight is a perfect description of humans authority bias. People love to be told what to do. Even if in the face of overwhelming real world evidence people blindly follow their GPS, and to ignore it can cause a lot of stress.
naresh_xai 7y ago
Which is strongly misleading tradeoff. For a ton of tasks, deep learning methods are no better than white box regression or tree ensemble methods. And there is no reason to expect that a deep learning model has to be unexplainable. He’s putting up a mystical interpretability vs accuracy tradeoff which does not exist. There is a ton of research out there making deep neural networks (slightly to significantly interpretable)

anthony_doan 7y ago

Yeah AI world have a loose definition of explanability and interpretablity. I also see this very dogmatic mindset that Deep Learning will do prediction and interpretability. What is stopping you from building two models? a regression statistic model to do interpretability/explanability and another deep learning for prediction? Like each coefficient in a regression have a t-test for significant in correlation with response. You don't have something like that in deep learning. Also I've seen many MLer use logistic regression as a classifier and ignoring the probability aspect like the Titanic dataset highlight the different mindset between statistician and ML. ML often will see this as a classify problem dead or not dead. Statistician will phrase to "What's the probability of this person dying with these covariates?" You know why this matter? It really matter in health/medical/social science. Often time inference is what they want and they need to know what affect your health not just shoving in tons of data and covariates/features. Not only that you many not even have enough data for these data hungry ML models. Another example is biostatistician figure out threshold between the benefit of taking an invasive procedure versus not taking it. We figure it out but giving a percentage and the doctor and experts will tell you where the threshold is, 20%, 40%? It's certainly not 50% that many MLer do. > We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications. Yes, that can be useful. Or at least reassuring. To me this just an excuse to not learn statistic. He should really look into Propensity modeling under Rubin-Neymer causality model. This is what statistic is going into after regression for observational data. With all the criticism I have for ML. I think it's just the mind set. I think the ML algorithms have a place and they're very good in certain domain such as NLP and computer vision. But to act as if they're the end all be all when statistic models have been there and use extensive in biostatistic and econometric fields is just hubris and ignorance. While ML is making excuses for causality. Econometric and statistician are working to build causality model. IIRC econometric is doing structure equation while statistician are going for Rubin-Neyman model. There is debate on which model is better but that's ongoing we'll wait and see from all the research papers.

datastoat 7y ago

If Facebook lawyers understood the implications of LeCun's argument, they wouldn't be happy! There are two types of explanations here: (1) why did the data come to be as it is, (2) why did my ML make the prediction it did. Science looks for the answer to (1), and causal models are a great way to think about it. Science and engineering, when they go hand in hand, build a machine by saying "Here is data, let me do science to understand nature's underlying laws, and my machine shall be based on those laws". The machine is inherently explainable because it's based on scientific laws. In the ML world, we can bypass the "learn scientific laws" part, and jump straight to "build a machine based on data". So the best answer to (2) has got to be "my ML made the prediction it did because of its training data". As Pearl said, ML is just curve fitting, so the only way to "explain" a ML prediction is to say "here are the points that the curve was fitted to". Prediction is just reading a value off the curve. Think the machine is biased? Look for bias in the training dataset! Think the machine is inaccurate? Look for sparsity or conflict in the training dataset! So the consequence of LeCun's distinction is that, when the GDPR calls for explainability of ML decision making, it is really calling for sharing of the training data. Facebook, watch out!

Rumudiez 7y ago

The word the author was looking for when they wrote "explainability" is actually "explicable," as in something that can be explained, and is converse to the more commonly used word "inexplicable."

thanatropism 7y ago
Good point. Compare with efficiency vs efficacy.

monadic2 7y ago

How do you validate the model without understanding where you might find problematic results? That doesn’t require a narrative, maybe, but it does require not using black-box-only testing.