4 ms·

This is important because there exists a trade-off in statistical learning models: in general, the more flexible your models are, the less understandable they b

by wyc 10y ago

This is important because there exists a trade-off in statistical learning models: in general, the more flexible your models are, the less understandable they become[0]. Modern machine learning techniques are typically very flexible. To gain intuition and reasoning of a model is to have understanding and trust--transparency. When you strike a nail with a hammer, it's pretty predictable what might happen: the nail could get hit, the hammer could miss, or very rarely, the hammer's head may fly off of the handle. When you replace the hammer with a black box that works correctly 99.999% of the time, but for 0.001%, something completely unpredictable happens, then there's a problem with volatility because that unpredictable event may have unacceptable consequences. I think explainable AI could help with intuitive and more fine-grained risk analysis, and that's certainly a good thing in high-stakes applications such as defense. [0] ISLR, Page 25: http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.pdf http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.p...

taneq 10y ago
This is interesting because usually "human-level performance" is the benchmark by which AI is judged. If it can do the thing as well as a human can do the thing, then that's good. Human intelligence isn't explainable, though. We can make up explanations but they're not based on our actual neural function (heck, we don't even know how that works in general, let alone how we generate specific outcomes). https://en.wikipedia.org/wiki/Introspection_illusion https://en.wikipedia.org/wiki/Introspection_illusion
- Cybiote 10y ago
  Illusory Introspection and confabulation are fascinating but generally have more to do with what was going on in your head when you took that action? than can you explain why you think that?. XAI is not interested in psychological state or an explanation in terms of ion concentration, voltage changes and synaptic activations. Or in the case of AI, the meaning of the contents in vectors after maps on matrix vector products. A large part of human success is that we are able to transmit knowledge to each other and accumulate experience over generations. A human starting from scratch is not very much more than its ape cousins. Human beings can explain efficiently because we must learn concepts decomposably and can take deductive shortcuts. In its purest form we have mathematical proofs as explanations. Explanations work with core concepts and generates on that basis. E.g. We want to know A and we know B, B is applicable in condition C. We will show condition C applies. With B, A can be shown in terms of... When you give big data to machines they work with raw correlations and output predictions. But humans have to be more creative because our scratch space is so small. Newton took Brahe's big data and compressed it to a theory of universal gravitation. Darwin made a relatively small number of observations, all things considered, and came up with On the Origin of Species.
  - Houshalter 10y ago
    Only a tiny fraction of human knowledge is transmitted through knowledge. Humans can't describe the wiring of their visual cortex, or the intuitions they feel. Expert chess players find it impossible to explain why they make the moves they do. Mathematicians can make proofs and explanations, but they still find it very difficult to transmit their underlying mathematical intuition. Even the purest fields of math can only really be learned through experience and practice, not just following a few proofs mechanically.
    - seanmcdirmid 10y ago
      Language allows us to communicate abstractions, which are then of course subject to interpretation. This is much better than other animals, which lack language facilities as far as we can tell. The entirety of human knowledge cannot be passed, but enough information is communicated that we can rebuild much of it in every generation.
    - Cybiote 10y ago
      That doesn't contradict anything I said, see my first paragraph. The core point is that you don't need to have full metacognition in order to build a culture that learns over time by explaining things and transmitting concepts. You don't need to explain every dendritic cluster in order to extract core, composable and generative abstractions useful enough to shorten time to understanding by orders of magnitude (discovery of zero vs how quickly it can be taught).
    - dharma1 10y ago
      That is very true. Transmittable knowledge doesn't capture the wirings that was required to produce the knowledge. I was watching the film on the Indian mathematician Ramanujan who had an immense intuition of mathematics, and really wondering how mathematical intuition, or intuition in general develops, and what intuition even means. I am still not sure but after seeing Schmidhuber's talk last week, I think it is being able to find regularities/patterns, resulting in a compressed representation that allows to understand and compress some aspect of the world, and building layers (abstractions) of those compressed representations. This allows you to navigate through very large search spaces much much faster. It's a fascinating subject, and at the core of what we are. I recommend the film. https://en.wikipedia.org/wiki/The_Man_Who_Knew_Infinity_(film) https://en.wikipedia.org/wiki/The_Man_Who_Knew_Infinity_(fil...
- deleted 10y ago
  [deleted]
- imh 10y ago
  Maybe human intelligence isn't explainable, but knowledge usually is. I think there's a ton of truth in that quote, "If you can't explain it simply, you don't understand it well enough."
- snowwrestler 10y ago
  It might simply be a matter of comfort. We don't know how human cognition works, but we have a ton of practical experience with it, so we feel like we have a pretty good handle on why people make the decisions that they do, and how to train and shape those decisions. As a specific example, a human doesn't typically trigger their own death/destruction without a significant reason that can be expressed and potentially detected. We know a lot more about how machine learning works--since we developed it ourselves recently--but because it is so new, we feel the need for deeper understanding to mitigate risks. To go back to the suicide example, a computer-driven system will destroy itself immediately for all sorts of reasons, including incredibly trivial errors.

by wyc 10y ago

taneq 10y ago

This is interesting because usually "human-level performance" is the benchmark by which AI is judged. If it can do the thing as well as a human can do the thing, then that's good. Human intelligence isn't explainable, though. We can make up explanations but they're not based on our actual neural function (heck, we don't even know how that works in general, let alone how we generate specific outcomes). https://en.wikipedia.org/wiki/Introspection_illusion https://en.wikipedia.org/wiki/Introspection_illusion

Cybiote 10y ago
Illusory Introspection and confabulation are fascinating but generally have more to do with what was going on in your head when you took that action? than can you explain why you think that?. XAI is not interested in psychological state or an explanation in terms of ion concentration, voltage changes and synaptic activations. Or in the case of AI, the meaning of the contents in vectors after maps on matrix vector products. A large part of human success is that we are able to transmit knowledge to each other and accumulate experience over generations. A human starting from scratch is not very much more than its ape cousins. Human beings can explain efficiently because we must learn concepts decomposably and can take deductive shortcuts. In its purest form we have mathematical proofs as explanations. Explanations work with core concepts and generates on that basis. E.g. We want to know A and we know B, B is applicable in condition C. We will show condition C applies. With B, A can be shown in terms of... When you give big data to machines they work with raw correlations and output predictions. But humans have to be more creative because our scratch space is so small. Newton took Brahe's big data and compressed it to a theory of universal gravitation. Darwin made a relatively small number of observations, all things considered, and came up with On the Origin of Species.
- Houshalter 10y ago
  Only a tiny fraction of human knowledge is transmitted through knowledge. Humans can't describe the wiring of their visual cortex, or the intuitions they feel. Expert chess players find it impossible to explain why they make the moves they do. Mathematicians can make proofs and explanations, but they still find it very difficult to transmit their underlying mathematical intuition. Even the purest fields of math can only really be learned through experience and practice, not just following a few proofs mechanically.
  - seanmcdirmid 10y ago
    Language allows us to communicate abstractions, which are then of course subject to interpretation. This is much better than other animals, which lack language facilities as far as we can tell. The entirety of human knowledge cannot be passed, but enough information is communicated that we can rebuild much of it in every generation.
  - Cybiote 10y ago
    That doesn't contradict anything I said, see my first paragraph. The core point is that you don't need to have full metacognition in order to build a culture that learns over time by explaining things and transmitting concepts. You don't need to explain every dendritic cluster in order to extract core, composable and generative abstractions useful enough to shorten time to understanding by orders of magnitude (discovery of zero vs how quickly it can be taught).
  - dharma1 10y ago
    That is very true. Transmittable knowledge doesn't capture the wirings that was required to produce the knowledge. I was watching the film on the Indian mathematician Ramanujan who had an immense intuition of mathematics, and really wondering how mathematical intuition, or intuition in general develops, and what intuition even means. I am still not sure but after seeing Schmidhuber's talk last week, I think it is being able to find regularities/patterns, resulting in a compressed representation that allows to understand and compress some aspect of the world, and building layers (abstractions) of those compressed representations. This allows you to navigate through very large search spaces much much faster. It's a fascinating subject, and at the core of what we are. I recommend the film. https://en.wikipedia.org/wiki/The_Man_Who_Knew_Infinity_(film) https://en.wikipedia.org/wiki/The_Man_Who_Knew_Infinity_(fil...
deleted 10y ago
[deleted]
imh 10y ago
Maybe human intelligence isn't explainable, but knowledge usually is. I think there's a ton of truth in that quote, "If you can't explain it simply, you don't understand it well enough."
snowwrestler 10y ago
It might simply be a matter of comfort. We don't know how human cognition works, but we have a ton of practical experience with it, so we feel like we have a pretty good handle on why people make the decisions that they do, and how to train and shape those decisions. As a specific example, a human doesn't typically trigger their own death/destruction without a significant reason that can be expressed and potentially detected. We know a lot more about how machine learning works--since we developed it ourselves recently--but because it is so new, we feel the need for deeper understanding to mitigate risks. To go back to the suicide example, a computer-driven system will destroy itself immediately for all sorts of reasons, including incredibly trivial errors.