4 ms·

Making all actions and effects explicitly explainable in terms of; why this happens, to me, it something in area of crystal ball, not science. Extreme example

by neatze 5y ago

Making all actions and effects explicitly explainable in terms of; why this happens, to me, it something in area of crystal ball, not science. Extreme example would be; gravity is not explainable, and yet, I don't need to have analytical solutions to learn how to play tennis, baseball or soccer.

karpierz 5y ago
Explainability isn't needed when your training set encompasses your test set, in the conditions that are meaningful. You have a heuristic which says "stuff tends to fall down to Earth", which you learned by tossing things in the air. The relevant condition you trained it on was "am on the surface of the Earth", which turns out to be the only thing you really need. However, if gravity was very sensitive to the number of cars on the road, or the number of street signs, I think you'd run into a lot of trouble without an underlying model.
- andy99 5y ago
  Your example is good, but I would argue it points to a good confidence (uncertainty) estimate as the most important element of trusting a model, rather than a human-interpretable explanation. Like you say, we want to know that the test data is distributed the same as the training data, and that our model repeatably gets predictions right in the presence of irrelevant perturbations (those two statements are at least partially equivalent). If we have this information, and are happy with the train/validation performance, then the actual process that our model uses to make a prediction really is not that important.
  - neatze 5y ago
    I might be wrong here, but you also need uncertainty estimate for training/test data set in relation to empirical domain, at least in cases with prohibitively large problem state space.
  - karpierz 5y ago
    How do you know whether the test data is distributed similarly to the training data without an underlying model of how the system works? Put differently, how do you know what matters for your tests? If you have a machine-learned model, you've likely trained it on sparse data. How do you know that your sparse data covers the set of signals required to make accurate predictions, and how do you know that your model actually uses those signals? Edit: I would say that to have a notion of confidence requires that you have an underlying model of how the system works (IE, explainability).
    - neatze 5y ago
      You can list all system variables without having any explanation what so ever how this variables related with each other, and then build model without explicit explanation with sufficient performance.

by neatze 5y ago

karpierz 5y ago

Explainability isn't needed when your training set encompasses your test set, in the conditions that are meaningful. You have a heuristic which says "stuff tends to fall down to Earth", which you learned by tossing things in the air. The relevant condition you trained it on was "am on the surface of the Earth", which turns out to be the only thing you really need. However, if gravity was very sensitive to the number of cars on the road, or the number of street signs, I think you'd run into a lot of trouble without an underlying model.

andy99 5y ago
Your example is good, but I would argue it points to a good confidence (uncertainty) estimate as the most important element of trusting a model, rather than a human-interpretable explanation. Like you say, we want to know that the test data is distributed the same as the training data, and that our model repeatably gets predictions right in the presence of irrelevant perturbations (those two statements are at least partially equivalent). If we have this information, and are happy with the train/validation performance, then the actual process that our model uses to make a prediction really is not that important.
- neatze 5y ago
  I might be wrong here, but you also need uncertainty estimate for training/test data set in relation to empirical domain, at least in cases with prohibitively large problem state space.
- karpierz 5y ago
  How do you know whether the test data is distributed similarly to the training data without an underlying model of how the system works? Put differently, how do you know what matters for your tests? If you have a machine-learned model, you've likely trained it on sparse data. How do you know that your sparse data covers the set of signals required to make accurate predictions, and how do you know that your model actually uses those signals? Edit: I would say that to have a notion of confidence requires that you have an underlying model of how the system works (IE, explainability).
  - neatze 5y ago
    You can list all system variables without having any explanation what so ever how this variables related with each other, and then build model without explicit explanation with sufficient performance.