3 ms·
Your example is good, but I would argue it points to a good confidence (uncertainty) estimate as the most important element of trusting a model, rather than a h
by andy99 6y ago
Your example is good, but I would argue it points to a good confidence (uncertainty) estimate as the most important element of trusting a model, rather than a human-interpretable explanation.
Like you say, we want to know that the test data is distributed the same as the training data, and that our model repeatably gets predictions right in the presence of irrelevant perturbations (those two statements are at least partially equivalent). If we have this information, and are happy with the train/validation performance, then the actual process that our model uses to make a prediction really is not that important.
- neatze 6y agoI might be wrong here, but you also need uncertainty estimate for training/test data set in relation to empirical domain, at least in cases with prohibitively large problem state space.
- karpierz 6y agoHow do you know whether the test data is distributed similarly to the training data without an underlying model of how the system works? Put differently, how do you know what matters for your tests? If you have a machine-learned model, you've likely trained it on sparse data. How do you know that your sparse data covers the set of signals required to make accurate predictions, and how do you know that your model actually uses those signals? Edit: I would say that to have a notion of confidence requires that you have an underlying model of how the system works (IE, explainability).
- neatze 6y agoYou can list all system variables without having any explanation what so ever how this variables related with each other, and then build model without explicit explanation with sufficient performance.