8 ms·

> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anyth

> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has. I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers. In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities". Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog. Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.

xg15 4y ago
How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug? Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question. Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour? I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
- FartyMcFarter 4y ago
  > E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned. Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").
Zak 4y ago
Two things are new: 1. Machine learning is increasingly being involved in important decisions. 2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain. If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.
djmips 4y ago
Side question, what's better at chess today - neutral network approach or deep blue approach?
squaredot 4y ago
I think these kinds of articles are geared toward people who do not know (even superficially) about the models. Perhaps the sentiment they want to elicit is "If the person who created these models cannot explain how they work, then he is not so different from me."
mdp2021 4y ago
> anything new It is nothing new. It is called (at least) "the problem of transparency". The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info). It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals. And of course, "understanding" the engine is required to advance it. It is a key problem.
- randcraw 4y ago
  In 1963 Marvin Minsky called it “The Credit Assignment Problem” — which, among a multitude of variables, were most important in solving an AI task?
  - mberh7 4y ago
- alexpetralia 4y ago
  I suppose this is why it is called "machine learning" and not "human learning"!
  - mdp2021 4y ago
    We deal with intelligence to either solve problems, or to benefit from intelligence to advance our own. If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal". Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.
  - jhbadger 4y ago
tromp 4y ago
> Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.
- bombcar 4y ago
  Exactly, the Deep Blue's move explanations are boring - they all boil down to "based on the inputs and rules programmed, this line of moves has the best overall outcome to a depth of X" where X is however deep it goes. You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.
  - Spivak 4y ago
    I don’t think that’s weird at all, I think humans can intuitively grasp “I simulated 100,000 games starting from the current board state and going at most 50 moves ahead and in games where you did X you reached a strong position most often.” Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.
    - nanidin 4y ago
      >in games where you did X you reached a strong position most often. FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.
deleted 4y ago
[deleted]
lostlogin 4y ago
> Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain. My thought on reading your comment was Douglas Adams and Deep Though’s 42.
HPsquared 4y ago
The explanations used to explain or justify human behaviour are just as speculative though, really. We can try and hypothesize, but that's it - there is no engineering design specification for the brain, nor even a debugger.
- visarga 4y ago
  We practically invent our explanations post facto, first decide, then verbalise.
UncleMeat 4y ago
Older ML techniques are explainable in nice ways. With something like logistic regression you get a very nice "this feature contributed this much to the final classification" breakdown. It just turns out that logistic regression is considerably less effective than RNNs, which have all these synthetic features within the inner layers that aren't easily meaningful to humans.
- tand22 4y ago
  I implemented an interpretable LSTM from this paper https://arxiv.org/pdf/1905.12034.pdf https://arxiv.org/pdf/1905.12034.pdf. It works by segmenting the context for each input variable, and then applying an attention layer to interpret each variables effect. I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.

xg15 4y ago

How would you debug such a system? E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug? Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question. Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour? I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.

FartyMcFarter 4y ago
> E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations. If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned. Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").

Zak 4y ago

Two things are new: 1. Machine learning is increasingly being involved in important decisions. 2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain. If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.

djmips 4y ago

Side question, what's better at chess today - neutral network approach or deep blue approach?

squaredot 4y ago

I think these kinds of articles are geared toward people who do not know (even superficially) about the models. Perhaps the sentiment they want to elicit is "If the person who created these models cannot explain how they work, then he is not so different from me."

mdp2021 4y ago

> anything new It is nothing new. It is called (at least) "the problem of transparency". The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info). It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals. And of course, "understanding" the engine is required to advance it. It is a key problem.

randcraw 4y ago
In 1963 Marvin Minsky called it “The Credit Assignment Problem” — which, among a multitude of variables, were most important in solving an AI task?
- mberh7 4y ago
alexpetralia 4y ago
I suppose this is why it is called "machine learning" and not "human learning"!
- mdp2021 4y ago
  We deal with intelligence to either solve problems, or to benefit from intelligence to advance our own. If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal". Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.
- jhbadger 4y ago

tromp 4y ago

> Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.

bombcar 4y ago
Exactly, the Deep Blue's move explanations are boring - they all boil down to "based on the inputs and rules programmed, this line of moves has the best overall outcome to a depth of X" where X is however deep it goes. You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.
- Spivak 4y ago
  I don’t think that’s weird at all, I think humans can intuitively grasp “I simulated 100,000 games starting from the current board state and going at most 50 moves ahead and in games where you did X you reached a strong position most often.” Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.
  - nanidin 4y ago
    >in games where you did X you reached a strong position most often. FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.

deleted 4y ago

[deleted]

lostlogin 4y ago

> Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain. My thought on reading your comment was Douglas Adams and Deep Though’s 42.

HPsquared 4y ago

The explanations used to explain or justify human behaviour are just as speculative though, really. We can try and hypothesize, but that's it - there is no engineering design specification for the brain, nor even a debugger.

visarga 4y ago
We practically invent our explanations post facto, first decide, then verbalise.

UncleMeat 4y ago

Older ML techniques are explainable in nice ways. With something like logistic regression you get a very nice "this feature contributed this much to the final classification" breakdown. It just turns out that logistic regression is considerably less effective than RNNs, which have all these synthetic features within the inner layers that aren't easily meaningful to humans.

tand22 4y ago
I implemented an interpretable LSTM from this paper https://arxiv.org/pdf/1905.12034.pdf https://arxiv.org/pdf/1905.12034.pdf. It works by segmenting the context for each input variable, and then applying an attention layer to interpret each variables effect. I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.