5 ms·

They cannot. Just because one internal operation is understandable, doesn't imply that the whole network is understandable. Take even something much simpler:

They cannot. Just because one internal operation is understandable, doesn't imply that the whole network is understandable. Take even something much simpler: decision trees. Textbooks give these as an example of understandable systems. A tree where you make one decision based on one feature at a time then at the leaves you output something. Like a bunch of if statements. And in the 90s when computers were slow and trees were small this was true. Today massive decision trees and approaches like random forests can create trees with millions of nodes. Nothing is interpretable about them. We have a basic math gap when it comes to understanding complex systems. Yet another network type solves nothing.

ImHereToVote 2y ago
A formula or equation that enables you to reason about complex systems might simply not exists. It could very well be that to reason about complexity forces you to actually do the complexity.
empath75 2y ago
Even extremely complicated decision trees are interpretable to some extent because you can just walk through the tree and answer questions like: "If this had not been true, would the result have been different?". It may not be possible to hold the entire tree in your head at once, but it's certainly possible to investigate the tree as needed to understand the path that was taken through it.
- svboese 2y ago
  But couldn‘t the same be said about standard MLPs or NNs in general?
  - empath75 2y ago
    _Sometimes_, and people do find features in neural networks by tweaking stuff and seeing how the neurons activate, but in general, no. Any given weight or layer or perceptron or whatever can be reused for multiple purposes and it's extremely difficult to say "this is responsible for that", and if you do find parts of the network responsible for a particular task, you don't know if it's _also_ responsible for something else. Whereas with a decision tree it's pretty simple to trace causality and tweak things without changing unrelated parts of the tree. Changing weights in a neural network leads to unpredictable results.
    - tomhallett 2y ago
      If a KAN has multiple layers, would tweaking the equations of a KAN be more similar to tweaking the weights in a MLP/NN, or more similar to tweaking a decision tree? EDIT: I gave the above thread (light_hue_1 > empath75 > svboese > empath75) to chatgpt and had it write a question to learn more, and it gave me "How do KAN networks compare to decision trees or neural networks when it comes to tracing causality and making interpretability more accessible, especially in large, complex models?". Either shows me and ai are on the right track, or i'm as dumb as a statistical token guessing machine.... https://imgur.com/3dSNZrG https://imgur.com/3dSNZrG
  - Scene_Cast2 2y ago
    LIME (local linear approximation basically) is one popular technique to do so. Still has flaws (such as not being close to a decision boundary).
    - pkage 2y ago
      LIME and other post-hoc explanatory techniques (deepshap, etc.) only give an explanation for a singular inference, but aren't helpful for the model as a whole. In other words, you can make a reasonable guess as to why a specific prediction was made but you have no idea how the model will behave in the general case, even on similar inputs.
      - Narhem 2y ago
        The purpose of post-prediction explanations would be to increase confidence of a practitioner to use said inference. It’s a disconnect between finding a real life “AI” and trying to find something which works and you can have a form of trust with.
        solidninja 2y ago
        Is there a study of "smooth"/"stable" "AI" algorithms - i.e. if you feed them input that is "close" then then the output is also "close"? (smooth as in smoothly differentiable/stable as in stable sorted)
  - ljosifov 2y ago
    You are right and IDK why you are downvoted. Few units of perceptrons, few nodes in a decision tree, few of anything - they are "interpretable". Billions of the sames - are not interpretable any more. This b/c our understanding of "interpretable" is "an array of symbols that can fit a page or a white board". But there is no reason to think that all the rules of our world would be such that they can be expressed that way. Some maybe, others maybe not. Interpretable is another platitudinous term that seems appealing at 1st sight, only to be found to not be that great after all. We humans are not interpretable, we can't explain how we come up with the actions we take, yet we don't say "now don't move, do nothing, until you are interpretable". So - much ado about little.
  - __mharrison__ 2y ago
    (IMO) to a lesser extent.
t_mann 2y ago
I think of it as "Could Newton have used this to find the expressions for the forces he was analyzing (eg gravitational force = g m_1 m_2 / d^2)?". I once asked a physics prof whether that was conceivable in principle, and he said yes. It seems to me like KANs should be able to find expressions like these given experimental data. If that was true, then I don't see how that wouldn't deserve being called interpretability.
- fjkdlsjflkds 2y ago
  > It seems to me like KANs should be able to find expressions like these given experimental data. Perhaps, but this is not something unique to KANs: any symbolic regression method can (at least in theory) find such simple expressions. Here is an example of such type of work (using non-KAN neural networks): https://www.science.org/doi/10.1126/sciadv.aay2631 https://www.science.org/doi/10.1126/sciadv.aay2631 Rephrasing: just because you can reach simple expressions with symbolic regression methods based on neural networks (or KANs) does not necessarily imply that neural networks (or KANs) are inherently interpretable (particularly once you start stacking multiple layers).
- nathan_compton 2y ago
  Just giving the force law hardly counts as interpret-ability. You probably know that the 1/r^2 in the force law comes from the dimensionality of space. That is the interpretation.
  - t_mann 2y ago
    It seems like you're asking for quite a lot here. Are there any examples of common interpretable ML methods that would give you anything like that answer? The most common methods that are called interpretable would give you hints like "Mass and distance matter for gravity, color doesn't" or "Gravity gets stronger with mass and weaker with distance". Both are clearly less informative than the formula. The only way I could think of to get anywhere near such an answer would be to use symbolic regression first and then ask an LLM to interpret the result. And that would probably take quite some more original research to get it anywhere near working, and even then probably primarily for problems where the answer is already known. I agree that this kind of answer would be useful, but we also have to be honest that that's not what currently meant by interpretability. And that's what should matter for evaluating the claim - it's not misleading if it delivers what one can reasonably expect. Whether we should update our interpretability definitions is a different (interesting) discussion.
baq 2y ago
yeah. you can run SHAP[0] on your xgboosted trees, results are kinda interesting, but it doesn't actually explain anything IME. [0] https://shap.readthedocs.io/en/latest/index.html https://shap.readthedocs.io/en/latest/index.html
- cubefox 2y ago
  No wonder. "Shapley values" have the problem that they assume all necessary conditions are equally important. Say a successful surgery needs both a surgeon and a nurse, otherwise the patient dies. Shapley values will then assume that both have contributed equally to the successful surgery. Which isn't true, because surgeons are much less available (less replaceable) than nurses. If the nurse gets ill, a different nurse could probably do the task, while if the surgeon gets ill, the surgery may well have to be postponed. So the surgeons are more important for (contribute more to) a successful surgery.
  - adammarples 2y ago
    Clearly both are equally important, 100% necessary. This doesn't account for rarity, nor does it account for wages, agreeability, smell or any of the other things it isn't trying to measure. You'll need a different metric for that and if you want to take both into account you should.
    - cubefox 2y ago
      Shapley values try to measure importance of contributions, and for this, bare necessity isn't a sufficient indicator. I think it comes down to probability. The task of the surgeon is, from a prior perspective, less likely to be fulfilled because it is harder to get hold of a surgeon. Similarly: What what was the main cause of the match getting lit? The match being struck? Or the atmosphere containing oxygen? Both are necessary in the sense that if either hadn't occurred the match wouldn't be lit. But it seems clear that the main cause was the match being struck, because matches being struck is relatively rare, and hence unlikely, while the atmosphere contains oxygen pretty much always. So I think the contributions calculated for Shapley values should be weighted by the inverse of their prior probabilities. Though it is possible that such probabilities are not typically available in the machine learning context in which SHAP operates.
__mharrison__ 2y ago
Many folks call tree ensembles "black boxes". I would call them grey boxes or dark grey boxes. You could interpret them if you want to. But who wants to go through 500 trees in practice?
- seventh12 2y ago
  You could also interpret networks if you want to. But who wants to go through millions of neurons?
deepsquirrelnet 2y ago
I generally agree, and I think interpretability is a fools errand for any sufficiently complex nonlinear model. That said, I’d be surprised if there weren’t eventually successful breakthroughs from the fields of nonlinear dynamics / pattern formation.

ImHereToVote 2y ago

A formula or equation that enables you to reason about complex systems might simply not exists. It could very well be that to reason about complexity forces you to actually do the complexity.

empath75 2y ago

Even extremely complicated decision trees are interpretable to some extent because you can just walk through the tree and answer questions like: "If this had not been true, would the result have been different?". It may not be possible to hold the entire tree in your head at once, but it's certainly possible to investigate the tree as needed to understand the path that was taken through it.

svboese 2y ago
But couldn‘t the same be said about standard MLPs or NNs in general?
- empath75 2y ago
  _Sometimes_, and people do find features in neural networks by tweaking stuff and seeing how the neurons activate, but in general, no. Any given weight or layer or perceptron or whatever can be reused for multiple purposes and it's extremely difficult to say "this is responsible for that", and if you do find parts of the network responsible for a particular task, you don't know if it's _also_ responsible for something else. Whereas with a decision tree it's pretty simple to trace causality and tweak things without changing unrelated parts of the tree. Changing weights in a neural network leads to unpredictable results.
  - tomhallett 2y ago
    If a KAN has multiple layers, would tweaking the equations of a KAN be more similar to tweaking the weights in a MLP/NN, or more similar to tweaking a decision tree? EDIT: I gave the above thread (light_hue_1 > empath75 > svboese > empath75) to chatgpt and had it write a question to learn more, and it gave me "How do KAN networks compare to decision trees or neural networks when it comes to tracing causality and making interpretability more accessible, especially in large, complex models?". Either shows me and ai are on the right track, or i'm as dumb as a statistical token guessing machine.... https://imgur.com/3dSNZrG https://imgur.com/3dSNZrG
- Scene_Cast2 2y ago
  LIME (local linear approximation basically) is one popular technique to do so. Still has flaws (such as not being close to a decision boundary).
  - pkage 2y ago
    LIME and other post-hoc explanatory techniques (deepshap, etc.) only give an explanation for a singular inference, but aren't helpful for the model as a whole. In other words, you can make a reasonable guess as to why a specific prediction was made but you have no idea how the model will behave in the general case, even on similar inputs.
    - Narhem 2y ago
      The purpose of post-prediction explanations would be to increase confidence of a practitioner to use said inference. It’s a disconnect between finding a real life “AI” and trying to find something which works and you can have a form of trust with.
      - solidninja 2y ago
        Is there a study of "smooth"/"stable" "AI" algorithms - i.e. if you feed them input that is "close" then then the output is also "close"? (smooth as in smoothly differentiable/stable as in stable sorted)
- ljosifov 2y ago
  You are right and IDK why you are downvoted. Few units of perceptrons, few nodes in a decision tree, few of anything - they are "interpretable". Billions of the sames - are not interpretable any more. This b/c our understanding of "interpretable" is "an array of symbols that can fit a page or a white board". But there is no reason to think that all the rules of our world would be such that they can be expressed that way. Some maybe, others maybe not. Interpretable is another platitudinous term that seems appealing at 1st sight, only to be found to not be that great after all. We humans are not interpretable, we can't explain how we come up with the actions we take, yet we don't say "now don't move, do nothing, until you are interpretable". So - much ado about little.
- __mharrison__ 2y ago
  (IMO) to a lesser extent.

t_mann 2y ago

I think of it as "Could Newton have used this to find the expressions for the forces he was analyzing (eg gravitational force = g m_1 m_2 / d^2)?". I once asked a physics prof whether that was conceivable in principle, and he said yes. It seems to me like KANs should be able to find expressions like these given experimental data. If that was true, then I don't see how that wouldn't deserve being called interpretability.

fjkdlsjflkds 2y ago
> It seems to me like KANs should be able to find expressions like these given experimental data. Perhaps, but this is not something unique to KANs: any symbolic regression method can (at least in theory) find such simple expressions. Here is an example of such type of work (using non-KAN neural networks): https://www.science.org/doi/10.1126/sciadv.aay2631 https://www.science.org/doi/10.1126/sciadv.aay2631 Rephrasing: just because you can reach simple expressions with symbolic regression methods based on neural networks (or KANs) does not necessarily imply that neural networks (or KANs) are inherently interpretable (particularly once you start stacking multiple layers).
nathan_compton 2y ago
Just giving the force law hardly counts as interpret-ability. You probably know that the 1/r^2 in the force law comes from the dimensionality of space. That is the interpretation.
- t_mann 2y ago
  It seems like you're asking for quite a lot here. Are there any examples of common interpretable ML methods that would give you anything like that answer? The most common methods that are called interpretable would give you hints like "Mass and distance matter for gravity, color doesn't" or "Gravity gets stronger with mass and weaker with distance". Both are clearly less informative than the formula. The only way I could think of to get anywhere near such an answer would be to use symbolic regression first and then ask an LLM to interpret the result. And that would probably take quite some more original research to get it anywhere near working, and even then probably primarily for problems where the answer is already known. I agree that this kind of answer would be useful, but we also have to be honest that that's not what currently meant by interpretability. And that's what should matter for evaluating the claim - it's not misleading if it delivers what one can reasonably expect. Whether we should update our interpretability definitions is a different (interesting) discussion.

baq 2y ago

yeah. you can run SHAP[0] on your xgboosted trees, results are kinda interesting, but it doesn't actually explain anything IME. [0] https://shap.readthedocs.io/en/latest/index.html https://shap.readthedocs.io/en/latest/index.html

cubefox 2y ago
No wonder. "Shapley values" have the problem that they assume all necessary conditions are equally important. Say a successful surgery needs both a surgeon and a nurse, otherwise the patient dies. Shapley values will then assume that both have contributed equally to the successful surgery. Which isn't true, because surgeons are much less available (less replaceable) than nurses. If the nurse gets ill, a different nurse could probably do the task, while if the surgeon gets ill, the surgery may well have to be postponed. So the surgeons are more important for (contribute more to) a successful surgery.
- adammarples 2y ago
  Clearly both are equally important, 100% necessary. This doesn't account for rarity, nor does it account for wages, agreeability, smell or any of the other things it isn't trying to measure. You'll need a different metric for that and if you want to take both into account you should.
  - cubefox 2y ago
    Shapley values try to measure importance of contributions, and for this, bare necessity isn't a sufficient indicator. I think it comes down to probability. The task of the surgeon is, from a prior perspective, less likely to be fulfilled because it is harder to get hold of a surgeon. Similarly: What what was the main cause of the match getting lit? The match being struck? Or the atmosphere containing oxygen? Both are necessary in the sense that if either hadn't occurred the match wouldn't be lit. But it seems clear that the main cause was the match being struck, because matches being struck is relatively rare, and hence unlikely, while the atmosphere contains oxygen pretty much always. So I think the contributions calculated for Shapley values should be weighted by the inverse of their prior probabilities. Though it is possible that such probabilities are not typically available in the machine learning context in which SHAP operates.

__mharrison__ 2y ago

Many folks call tree ensembles "black boxes". I would call them grey boxes or dark grey boxes. You could interpret them if you want to. But who wants to go through 500 trees in practice?

seventh12 2y ago
You could also interpret networks if you want to. But who wants to go through millions of neurons?

deepsquirrelnet 2y ago

I generally agree, and I think interpretability is a fools errand for any sufficiently complex nonlinear model. That said, I’d be surprised if there weren’t eventually successful breakthroughs from the fields of nonlinear dynamics / pattern formation.