5 ms·

I’ve always wondered what an sufficient explanation of a neural network would entail. At a very low level, there’s no secret: all of the weights are there, all

I’ve always wondered what an sufficient explanation of a neural network would entail. At a very low level, there’s no secret: all of the weights are there, all of the relationships between weights are known. The problem is that it doesn’t tell you anything about the emergent properties of the network, in the same way that quantum physics doesn’t give much insight into biology. It may be possible that there is no English sentence you can utter about the network which is both explanatory and fully accurate. What is the network doing? It’s trying to approximate the function you’ve given it. That’s it. You can try other things like ablation to find the effects of lobotomizing the network in certain ways, but this also can’t fully explain 2nd and higher order relationships.

sdwr 4y ago
Same thing as the brain pretty much, except we have root control over the NN in a way we don't with the brain. To my knowledge, some angles have been explored. Finding which inputs maximally stimulate indiv neurons, and tracing that up the chain. There was a fantastic article that did this on ImageNet - the takeaway was that neurons adjacent to the input encode sharp, basic features (like edges), and "higher-level" ones encode more nuanced stuff (textures, fur, wheels). Then ablations, as you mentioned. And finally, rerunning the same training data on differently sized + shaped architectures. AI systems are missing at least one level of complexity. All neurons in a NN layer fire simultaneously, unlike brain neurons, which trigger each other async.
- janalsncm 4y ago
  The brain is heterogenous in ways that a neural network needn’t be. Certain brain regions definitely perform certain functions, certain regions are hardwired to other regions. You can chop out a chunk of a person’s brain and they’ll be completely unable to do certain things. Afaik the properties you’re referring to on AlexNet (the network trained to perform on Image Net) have to do with the nature of repeated convolution operations which, while interesting, is not as deeply insightful as I believe the article is aiming for.
obviouslynotme 4y ago
>It may be possible that there is no English sentence you can utter about anything which is both explanatory and fully accurate. FTFY. Philosophy aside, humans understand complex things by building simpler abstractions for them. I think the key to NN understandability centers on building simpler NNs focused on specific tasks that build into bigger ones.
kypro 4y ago
This sentence from the article stood out to me: > The explanations have to be wrong, because if their explanations were always right, you could just replace the black box with the explanations. I've never thought of it like that, but I think that really gets to the core of the issue... To be able to reason about the behaviour of a neural network with 100% confidence you would need an explanation that incorporated every weight, otherwise you have to accept some degree of uncertainty. But the idea you could ever explain something with billions of parameters in a way that a human could comprehend seems ridiculous. Imperfect generalisation therefore would inevitably be required – "this bit of the network seems to perform x function". But in doing this you have to accept that this generalisation comes with uncertainty because such a high-level approximation is unlikely to capture all of the nuances. And if it did capture all nuances then the black box isn't needed.
- flir 4y ago
  The map is not the territory. Or, at more length and in more depth, The Analytical Language of John Wilkins by Borges.
  - janalsncm 4y ago
    Thank you for sharing, the wiki has a pretty funny note: > Lewis Carroll, in Sylvie and Bruno Concluded (1893), made the point humorously with his description of a fictional map that had "the scale of a mile to the mile". A character notes some practical difficulties with such a map and states that "we now use the country itself, as its own map, and I assure you it does nearly as well."
    - flir 4y ago
      I'm reminded of: "I have the world's largest collection of seashells. I keep it on all the beaches of the world... perhaps you've seen it." - Steven Wright
- foobarbecue 4y ago
  All models are wrong, but some models are useful.
hyoogle 4y ago
> It may be possible that there is no English sentence you can utter about the network which is both explanatory and fully accurate. reminds me of: "The Tao that can be told is not the eternal Tao."
mxkopy 4y ago
I've thought about pattern matching NN weights against known graph structures. I'd imagine, for example, that decision trees emerge at some point in GPT. The real problem IMO is our ability to represent natural concepts as such graphs. Is love, for example, a DFA? Can we search for its isomorphism?
keithwinstein 4y ago
One thing that would be fun to know is, e.g., "If the LLM answers question X correctly, then what's a minimal-sized set of things we could remove from the training set and cause it to get that question wrong?" I think with current methods this would be pretty expensive to find out, but, in principle I'm guessing it would be pretty illuminating.
- hansvm 4y ago
  Causal modeling lets you determine where particular facts are stored without too much computational cost and also how you can edit those facts in the model. That might allow you to decouple the problem into first finding the desired set of gradients (converging at the target modification, zero for the weights you don't care about), and just do a linear solve (since probabilistic weightings on how likely an input is in the training set will linearly impact each of the gradients in basically every neural architecture of note, including most LLM modifications) to find an approximately minimal (IIRC, a true minimal subset is NP-hard or something) subset of the inputs which when combined would give you the inverse of the target gradient. Remove that much of the weighting for each of the inputs.
bane 4y ago
> the same way that quantum physics doesn’t give much insight into biology This is very insightful. I've long maintained that "understanding" requires some predictive ability in addition to an ability to explain what's happened in the past. In the case of neural networks, we should be able to "simply" generate the network that has the desired behavior. But we can't. We're still shackled to compiling some collection of training data that may or may not produce the desired behavior, tested by some other collection of testing data that may or may not test the desired phenomenon.
andrewflnr 4y ago
Perhaps a sequence of sentences, almost a series in the mathematical sense, where you iterate your explanation-finder, generating more sentences in the series, until it's sufficiently detailed for the question you care about today. Also, remember that the explanation doesn't really have to fully contain the semantics of the model. It doesn't have to fully encode the weights, just activate sufficiently sympathetic structures in a human brain.

sdwr 4y ago

Same thing as the brain pretty much, except we have root control over the NN in a way we don't with the brain. To my knowledge, some angles have been explored. Finding which inputs maximally stimulate indiv neurons, and tracing that up the chain. There was a fantastic article that did this on ImageNet - the takeaway was that neurons adjacent to the input encode sharp, basic features (like edges), and "higher-level" ones encode more nuanced stuff (textures, fur, wheels). Then ablations, as you mentioned. And finally, rerunning the same training data on differently sized + shaped architectures. AI systems are missing at least one level of complexity. All neurons in a NN layer fire simultaneously, unlike brain neurons, which trigger each other async.

janalsncm 4y ago
The brain is heterogenous in ways that a neural network needn’t be. Certain brain regions definitely perform certain functions, certain regions are hardwired to other regions. You can chop out a chunk of a person’s brain and they’ll be completely unable to do certain things. Afaik the properties you’re referring to on AlexNet (the network trained to perform on Image Net) have to do with the nature of repeated convolution operations which, while interesting, is not as deeply insightful as I believe the article is aiming for.

obviouslynotme 4y ago

>It may be possible that there is no English sentence you can utter about anything which is both explanatory and fully accurate. FTFY. Philosophy aside, humans understand complex things by building simpler abstractions for them. I think the key to NN understandability centers on building simpler NNs focused on specific tasks that build into bigger ones.

kypro 4y ago

This sentence from the article stood out to me: > The explanations have to be wrong, because if their explanations were always right, you could just replace the black box with the explanations. I've never thought of it like that, but I think that really gets to the core of the issue... To be able to reason about the behaviour of a neural network with 100% confidence you would need an explanation that incorporated every weight, otherwise you have to accept some degree of uncertainty. But the idea you could ever explain something with billions of parameters in a way that a human could comprehend seems ridiculous. Imperfect generalisation therefore would inevitably be required – "this bit of the network seems to perform x function". But in doing this you have to accept that this generalisation comes with uncertainty because such a high-level approximation is unlikely to capture all of the nuances. And if it did capture all nuances then the black box isn't needed.

flir 4y ago
The map is not the territory. Or, at more length and in more depth, The Analytical Language of John Wilkins by Borges.
- janalsncm 4y ago
  Thank you for sharing, the wiki has a pretty funny note: > Lewis Carroll, in Sylvie and Bruno Concluded (1893), made the point humorously with his description of a fictional map that had "the scale of a mile to the mile". A character notes some practical difficulties with such a map and states that "we now use the country itself, as its own map, and I assure you it does nearly as well."
  - flir 4y ago
    I'm reminded of: "I have the world's largest collection of seashells. I keep it on all the beaches of the world... perhaps you've seen it." - Steven Wright
foobarbecue 4y ago
All models are wrong, but some models are useful.

hyoogle 4y ago

> It may be possible that there is no English sentence you can utter about the network which is both explanatory and fully accurate. reminds me of: "The Tao that can be told is not the eternal Tao."

mxkopy 4y ago

I've thought about pattern matching NN weights against known graph structures. I'd imagine, for example, that decision trees emerge at some point in GPT. The real problem IMO is our ability to represent natural concepts as such graphs. Is love, for example, a DFA? Can we search for its isomorphism?

keithwinstein 4y ago

One thing that would be fun to know is, e.g., "If the LLM answers question X correctly, then what's a minimal-sized set of things we could remove from the training set and cause it to get that question wrong?" I think with current methods this would be pretty expensive to find out, but, in principle I'm guessing it would be pretty illuminating.

hansvm 4y ago
Causal modeling lets you determine where particular facts are stored without too much computational cost and also how you can edit those facts in the model. That might allow you to decouple the problem into first finding the desired set of gradients (converging at the target modification, zero for the weights you don't care about), and just do a linear solve (since probabilistic weightings on how likely an input is in the training set will linearly impact each of the gradients in basically every neural architecture of note, including most LLM modifications) to find an approximately minimal (IIRC, a true minimal subset is NP-hard or something) subset of the inputs which when combined would give you the inverse of the target gradient. Remove that much of the weighting for each of the inputs.

bane 4y ago

> the same way that quantum physics doesn’t give much insight into biology This is very insightful. I've long maintained that "understanding" requires some predictive ability in addition to an ability to explain what's happened in the past. In the case of neural networks, we should be able to "simply" generate the network that has the desired behavior. But we can't. We're still shackled to compiling some collection of training data that may or may not produce the desired behavior, tested by some other collection of testing data that may or may not test the desired phenomenon.

andrewflnr 4y ago

Perhaps a sequence of sentences, almost a series in the mathematical sense, where you iterate your explanation-finder, generating more sentences in the series, until it's sufficiently detailed for the question you care about today. Also, remember that the explanation doesn't really have to fully contain the semantics of the model. It doesn't have to fully encode the weights, just activate sufficiently sympathetic structures in a human brain.