4 ms·

This sentence from the article stood out to me: > The explanations have to be wrong, because if their explanations were always right, you could just replace th

by kypro 4y ago

This sentence from the article stood out to me: > The explanations have to be wrong, because if their explanations were always right, you could just replace the black box with the explanations. I've never thought of it like that, but I think that really gets to the core of the issue... To be able to reason about the behaviour of a neural network with 100% confidence you would need an explanation that incorporated every weight, otherwise you have to accept some degree of uncertainty. But the idea you could ever explain something with billions of parameters in a way that a human could comprehend seems ridiculous. Imperfect generalisation therefore would inevitably be required – "this bit of the network seems to perform x function". But in doing this you have to accept that this generalisation comes with uncertainty because such a high-level approximation is unlikely to capture all of the nuances. And if it did capture all nuances then the black box isn't needed.

flir 4y ago
The map is not the territory. Or, at more length and in more depth, The Analytical Language of John Wilkins by Borges.
- janalsncm 4y ago
  Thank you for sharing, the wiki has a pretty funny note: > Lewis Carroll, in Sylvie and Bruno Concluded (1893), made the point humorously with his description of a fictional map that had "the scale of a mile to the mile". A character notes some practical difficulties with such a map and states that "we now use the country itself, as its own map, and I assure you it does nearly as well."
  - flir 4y ago
    I'm reminded of: "I have the world's largest collection of seashells. I keep it on all the beaches of the world... perhaps you've seen it." - Steven Wright
foobarbecue 4y ago
All models are wrong, but some models are useful.
danielmarkbruce 4y ago
Obviously debatable, but this seems wrong. It could easily be that we can't figure out how to do something but when presented with a solution understand it. Many people understand public private key cryptography but could never in a million years come up with it. Same for much of physics, a lot of math etc. As for the sheer complexity of millions/billions of parameters - people can explain and understand molecular dynamics simulations perfectly well. They can't use a small number of words to explain a specific configuration, but it's explainable.
- janalsncm 4y ago
  I think there’s a major difference though. You can understand public key crypto at varying levels of complexity because it is a topic created from first principles. Any sub-topic within the umbrella of public key crypto can be explored further. In this sense the knowledge is like a large program. Neural networks aren’t like that. The whole thing works together simultaneously to derive a result. So I’m skeptical that there is a high level explanation other than “it’s doing it’s best to approximate the data it was trained on” which is not that useful.
  - low_tech_love 4y ago
    There must be some kind of pattern in the high-dimensional space of NN decision making that can be used to explain its decision, at least broadly (given some well-defined query). I’m not saying it’s easy, and I’m not saying the end user will have access to it, but surely someone in the pipeline needs to be able to acquire that. Or else there will never be a serious certification process before general and widespread use.
    - janalsncm 4y ago
      I guess this raises the question of levels of explanation. What kind of explanation would you like to be able to get? It reminds me of something the physicist Tim Maudlin described on a recent podcast (I will give a poor summary): Suppose you have the spinning color wheel on your MacBook. You give it to a computer scientist who says that the program is in an infinite loop, and the wheel will never stop spinning. Then you give the computer to a superhuman physicist who studies all of the components including the state of the transistors and internal components and concludes that the wheel will spin for 9 years before the screen gives out. [1] The point is that both are correct, but we’d really prefer the first explanation, since it gives insight into what the program is doing. The problem is that neural networks aren’t like programs. They don’t have reducible internal components unless you reduce down to the weights of the network itself. [2] [1] https://m.youtube.com/watch?v=k0CQ_D7dwA0 https://m.youtube.com/watch?v=k0CQ_D7dwA0 [2] https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35
zmgsabst 4y ago
Sure — abstractions elide details. But “this portion of the network appears to recognize faces” is still a useful abstraction. In the same way it is for neurology. Even if both NNs and brains sometimes see faces that aren’t there because the network isn’t “recognizing faces” but a system which signals on a complex set of heuristics that mostly detects faces. How does the answer change if I only want 99% confidence and accept the existence of illusions? — can we describe the bulk of the behavior in a summary?
- janalsncm 4y ago
  It would be very satisfying if we could have an explanation like that. But that’s far from guaranteed, unless that was part of the explicit design of an MMOE for example. Very likely if you ask “what part of the network recognizes faces” the best answer is “the whole thing”.
  - int_19h 3y ago
    There's some evidence for emergent "modules" inside NNs. https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-modularity-the-separability-of-capabilities-in-large https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-m...
low_tech_love 4y ago
We certainly cannot comprehend the entire thing at the same time. ”XAI” is a bit of a buzzword right now and every HCI researcher struggling with finding their grip have jumped on the hype boat, meaning lots of them will just say things because it sounds nice to people with even less knowledge of NNs as them. However, to think we cannot comprehend them at all is too much. We can certainly ”query” a NN to understand the weights and activations that led to some decision in some specific scenario. How to consume and use that information is another matter, of course, but a debugging process is certainly feasible (as it is with any large scale computer system). Plus, the architectures of neural networks are not as complicated as most people think. Sure it gets weird after training, but the basic architecture of components are not nearly as complicated as the hardware of a GPU for example, or even that of a complex OS kernel.
psyklic 4y ago
The neural net gives a very noisy approximation of some ideal structure. So you're looking for what that ideal structure is, which very possibly does not require a neural network to represent.

by kypro 4y ago

flir 4y ago

The map is not the territory. Or, at more length and in more depth, The Analytical Language of John Wilkins by Borges.

janalsncm 4y ago
Thank you for sharing, the wiki has a pretty funny note: > Lewis Carroll, in Sylvie and Bruno Concluded (1893), made the point humorously with his description of a fictional map that had "the scale of a mile to the mile". A character notes some practical difficulties with such a map and states that "we now use the country itself, as its own map, and I assure you it does nearly as well."
- flir 4y ago
  I'm reminded of: "I have the world's largest collection of seashells. I keep it on all the beaches of the world... perhaps you've seen it." - Steven Wright

foobarbecue 4y ago

All models are wrong, but some models are useful.

danielmarkbruce 4y ago

Obviously debatable, but this seems wrong. It could easily be that we can't figure out how to do something but when presented with a solution understand it. Many people understand public private key cryptography but could never in a million years come up with it. Same for much of physics, a lot of math etc. As for the sheer complexity of millions/billions of parameters - people can explain and understand molecular dynamics simulations perfectly well. They can't use a small number of words to explain a specific configuration, but it's explainable.

janalsncm 4y ago
I think there’s a major difference though. You can understand public key crypto at varying levels of complexity because it is a topic created from first principles. Any sub-topic within the umbrella of public key crypto can be explored further. In this sense the knowledge is like a large program. Neural networks aren’t like that. The whole thing works together simultaneously to derive a result. So I’m skeptical that there is a high level explanation other than “it’s doing it’s best to approximate the data it was trained on” which is not that useful.
- low_tech_love 4y ago
  There must be some kind of pattern in the high-dimensional space of NN decision making that can be used to explain its decision, at least broadly (given some well-defined query). I’m not saying it’s easy, and I’m not saying the end user will have access to it, but surely someone in the pipeline needs to be able to acquire that. Or else there will never be a serious certification process before general and widespread use.
  - janalsncm 4y ago
    I guess this raises the question of levels of explanation. What kind of explanation would you like to be able to get? It reminds me of something the physicist Tim Maudlin described on a recent podcast (I will give a poor summary): Suppose you have the spinning color wheel on your MacBook. You give it to a computer scientist who says that the program is in an infinite loop, and the wheel will never stop spinning. Then you give the computer to a superhuman physicist who studies all of the components including the state of the transistors and internal components and concludes that the wheel will spin for 9 years before the screen gives out. [1] The point is that both are correct, but we’d really prefer the first explanation, since it gives insight into what the program is doing. The problem is that neural networks aren’t like programs. They don’t have reducible internal components unless you reduce down to the weights of the network itself. [2] [1] https://m.youtube.com/watch?v=k0CQ_D7dwA0 https://m.youtube.com/watch?v=k0CQ_D7dwA0 [2] https://karpathy.medium.com/software-2-0-a64152b37c35 https://karpathy.medium.com/software-2-0-a64152b37c35

zmgsabst 4y ago

Sure — abstractions elide details. But “this portion of the network appears to recognize faces” is still a useful abstraction. In the same way it is for neurology. Even if both NNs and brains sometimes see faces that aren’t there because the network isn’t “recognizing faces” but a system which signals on a complex set of heuristics that mostly detects faces. How does the answer change if I only want 99% confidence and accept the existence of illusions? — can we describe the bulk of the behavior in a summary?

janalsncm 4y ago
It would be very satisfying if we could have an explanation like that. But that’s far from guaranteed, unless that was part of the explicit design of an MMOE for example. Very likely if you ask “what part of the network recognizes faces” the best answer is “the whole thing”.
- int_19h 3y ago
  There's some evidence for emergent "modules" inside NNs. https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-modularity-the-separability-of-capabilities-in-large https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-m...

low_tech_love 4y ago

We certainly cannot comprehend the entire thing at the same time. ”XAI” is a bit of a buzzword right now and every HCI researcher struggling with finding their grip have jumped on the hype boat, meaning lots of them will just say things because it sounds nice to people with even less knowledge of NNs as them. However, to think we cannot comprehend them at all is too much. We can certainly ”query” a NN to understand the weights and activations that led to some decision in some specific scenario. How to consume and use that information is another matter, of course, but a debugging process is certainly feasible (as it is with any large scale computer system). Plus, the architectures of neural networks are not as complicated as most people think. Sure it gets weird after training, but the basic architecture of components are not nearly as complicated as the hardware of a GPU for example, or even that of a complex OS kernel.

psyklic 4y ago

The neural net gives a very noisy approximation of some ideal structure. So you're looking for what that ideal structure is, which very possibly does not require a neural network to represent.