4 ms·

Do people really use the phrase “over confident” in this way? It is very misleading. What is happening is called “over fitting”. Think of data as dots. A mode

Do people really use the phrase “over confident” in this way? It is very misleading. What is happening is called “over fitting”. Think of data as dots. A model that generalizes well will create as simple of a function as possible that fits the training data points pretty well. But keep training and parameters will often get very large, creating huge up and down swings in the function curve, far outside the actual data values, in order to pass through the training data points exactly. So it’s technically a better fit to the training data, but it is now a crazy function, often producing extreme outputs on new data. Practically a worst case lack of generalization. Thus, “over fitting”. And “over fitting” isn’t the same as “memorization”. Large models can memorize small datasets without over fitting. They have so many parameters, it takes few changes to fit the training data. At which time, learning stops at an otherwise random function, and generalization is never achieved. That case is called “underdetermined”. There are models that produce both outputs and confidences (essentially predict their own error standard deviation per output, based on the input). So “over confident” can mean a model that predicted high confidence (low error deviation) inaccurately.

jph00 3y ago
No I don't use the term overfitting for a model where the accuracy is getting better. I think it's misleading.
- mjburgess 3y ago
  Accuracy is very rarely a useful metric. It's more an engineering metric than something a user would ever care about. What users want is to have their own credences properly calibrated by engaging with some system. From a physics textbook, they want a systematic presentation of ideas which allows them to build intuitions etc. It's important to formulate the actual goal of the system, rather than just the engineer's goal (consider eg., "width of pipes" vs., "clean running water"). In the case of statistical AI systems, the goal is often best formulated in terms of the confidences of the system not its output. Since its output accuracy is kinda nonlinear and discontinuous in those confidences. So from a statical AI Q&A system we dont want The Answer, we want the system to have expert-like confidences over possible answers. Of course, as soon as you start formulating these metrics, all the SoA 99%+ accuracy hype evaporates. Since most of these systems have terrible confidence distributions. Consider, eg., ChatGPT whose answers are often plausibly accurate (they count as an answer) but just repeat some silicon valley hype in a way an expert wouldnt. ChatGPT rarely has the careful scepticism of an expert, rarely presents ideas in an even handed way, rarely mentions the opposite. It makes generating reference materials on areas with expert disagreement quite dangerous. ChatGPT presents the non-expert credence distribution. (And indeed, always does, since it just models (Q,A) frequencies which are not truth-apt)
  - DougBTX 3y ago
    This is mixing two meanings of confidence which could lead to confusion. The OP is using confidence to describe how high the per-token probability scores are, while you are talking about the confidence expressed in the tone of voice of the language generated by the model. Really those are orthogonal issues. (Eg, a model could predict with high probability that a output should be “I don’t know”)
    - mjburgess 3y ago
      It seems like i'm mixing them, but i'm not. I'm saying as a matter of fact ChatGPT should have different confidences in propositions. My issue isnt the tone of voice, my issue is the content of what it's saying is wrong wrt what we care about, ie., expert credences (/confidences) in the claims it's generating. It can "express confidently" scepticism; it does not. That's the issue. In my lang above i was mostly using credence to talk about the strength of the mental state of belief; and confidence to talk about the model of that used in statistical AI.
- Nevermark 3y ago
  Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a well established effect that occurs for all curve fitting, function approximation, parameter optimizing, and model training algorithms. You can Google it with no other context: “over fitting”. [0] “Confidence” isn’t its name and its meaning has nothing to do with the effect. Nothing wrong with making up terminology for new effects, but this one is an oldie. [0] https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF...
  - Nevermark 3y ago
    Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)
hashhar 3y ago
This was an awesome explainer - thanks a lot. It helps clear up a lot of jargon I keep hearing in very precise ways.
yashap 3y ago
I do think it’s a form of overfitting - loss on the training set improved while loss on the validation set got worse. However, it’s not the common form of overfitting, where accuracy on the validation set gets worse. In this case, accuracy on the validation data set continued to improve. But when it was wrong, it gave a higher confidence in its wrong answer than before. e.g. before it may have incorrectly thought the answer was X, with 60% confidence, now it still thinks the answer is X, but with higher confidence, say 70%. I do think it’s a form of overfitting, but a weird one. Overconfidence seems like a good, more specific term to me.
ActivePattern 3y ago
If we are considering the function to be the neural network with an argmax applied to the output probabilities, it's not overfitting at all. Its classification accuracy over unseen data (validation set) continues to improve. The issue here is one of calibration: https://en.m.wikipedia.org/wiki/Calibration_(statistics) https://en.m.wikipedia.org/wiki/Calibration_(statistics). That is, the output probabilities of the neural network do not reflect the true (observed) probabilities. If it is systematically underestimating the probabilities, it is termed "underconfident", and if overestimating the probabilities, "overconfident". Note that in these cases, it may still be improving as a classifier on unseen data, while still showing higher validation loss as calibration degrades.

jph00 3y ago

No I don't use the term overfitting for a model where the accuracy is getting better. I think it's misleading.

mjburgess 3y ago
Accuracy is very rarely a useful metric. It's more an engineering metric than something a user would ever care about. What users want is to have their own credences properly calibrated by engaging with some system. From a physics textbook, they want a systematic presentation of ideas which allows them to build intuitions etc. It's important to formulate the actual goal of the system, rather than just the engineer's goal (consider eg., "width of pipes" vs., "clean running water"). In the case of statistical AI systems, the goal is often best formulated in terms of the confidences of the system not its output. Since its output accuracy is kinda nonlinear and discontinuous in those confidences. So from a statical AI Q&A system we dont want The Answer, we want the system to have expert-like confidences over possible answers. Of course, as soon as you start formulating these metrics, all the SoA 99%+ accuracy hype evaporates. Since most of these systems have terrible confidence distributions. Consider, eg., ChatGPT whose answers are often plausibly accurate (they count as an answer) but just repeat some silicon valley hype in a way an expert wouldnt. ChatGPT rarely has the careful scepticism of an expert, rarely presents ideas in an even handed way, rarely mentions the opposite. It makes generating reference materials on areas with expert disagreement quite dangerous. ChatGPT presents the non-expert credence distribution. (And indeed, always does, since it just models (Q,A) frequencies which are not truth-apt)
- DougBTX 3y ago
  This is mixing two meanings of confidence which could lead to confusion. The OP is using confidence to describe how high the per-token probability scores are, while you are talking about the confidence expressed in the tone of voice of the language generated by the model. Really those are orthogonal issues. (Eg, a model could predict with high probability that a output should be “I don’t know”)
  - mjburgess 3y ago
    It seems like i'm mixing them, but i'm not. I'm saying as a matter of fact ChatGPT should have different confidences in propositions. My issue isnt the tone of voice, my issue is the content of what it's saying is wrong wrt what we care about, ie., expert credences (/confidences) in the claims it's generating. It can "express confidently" scepticism; it does not. That's the issue. In my lang above i was mostly using credence to talk about the strength of the mental state of belief; and confidence to talk about the model of that used in statistical AI.
Nevermark 3y ago
Over fitting literally means what it says, fitting the training data too well to maintain a well formed function. This is many decades old terminology for a well established effect that occurs for all curve fitting, function approximation, parameter optimizing, and model training algorithms. You can Google it with no other context: “over fitting”. [0] “Confidence” isn’t its name and its meaning has nothing to do with the effect. Nothing wrong with making up terminology for new effects, but this one is an oldie. [0] https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF-8&hl=en-us&client=safari https://www.google.com/search?q=over+fitting&ie=UTF-8&oe=UTF...
- Nevermark 3y ago
  Addendum: Most data has a fraction which is noise, or is dependent on information not completely captured by training data. In both cases, training to a perfect fit of training data makes no sense. Learning the particular noise in training data guarantees worse results on real data, where the noise will, by definition, be different. The same is true for exactly reproducing data that depended on some unaccounted for information. Once applied, the unaccounted information in the training data will have no predictive quality. So perfectly fitting data is usually a terrible idea, even when it can be done. Training data captures a problem to be solved, as best it can. But it isn’t the same as the actual problem. —— Best practice: 1. Use training data to optimize a model 2. Use separate validation data to approximate current generalization quality, and stop at (or revert to) the point where validation performance was best. 3. Use separate test data, not used for model design in any way, for a completely independent approximation of generalization performance. Wide disparities between validation & test performance suggest problems. Ideally they should track each other fairly closely. If not, more data is probably needed to characterize the problem more reliably. NEVER just retrain, or tweak training parameters, until training & the validation stop produce good test performance. That means the test data was actually used in the design and is no longer an independent measure of performance! —— Assuming you are having trouble getting similar validation & test performances: A good way to ensure test performance is real, is to retrain the model in exactly the same way, several times, with different random divisions of training, validation & test data. If test results are good regardless of data divisions, then they are reliable. Then you can eliminate the test performance dependency, by randomly selecting any of the models. Don’t choose the one with best test results! That takes discipline! In that case, the mean of all test performances is your best estimate of generalization performance, regardless of model. (Throwing out models with the worst and best test performances is ok, it avoids outlier training failures or successes equally. Note I said “avoids”, not “eliminates”, since the worst & best test performances are estimates of generalization, not a measure of actual generalization.)

hashhar 3y ago

This was an awesome explainer - thanks a lot. It helps clear up a lot of jargon I keep hearing in very precise ways.

yashap 3y ago

I do think it’s a form of overfitting - loss on the training set improved while loss on the validation set got worse. However, it’s not the common form of overfitting, where accuracy on the validation set gets worse. In this case, accuracy on the validation data set continued to improve. But when it was wrong, it gave a higher confidence in its wrong answer than before. e.g. before it may have incorrectly thought the answer was X, with 60% confidence, now it still thinks the answer is X, but with higher confidence, say 70%. I do think it’s a form of overfitting, but a weird one. Overconfidence seems like a good, more specific term to me.

ActivePattern 3y ago

If we are considering the function to be the neural network with an argmax applied to the output probabilities, it's not overfitting at all. Its classification accuracy over unseen data (validation set) continues to improve. The issue here is one of calibration: https://en.m.wikipedia.org/wiki/Calibration_(statistics) https://en.m.wikipedia.org/wiki/Calibration_(statistics). That is, the output probabilities of the neural network do not reflect the true (observed) probabilities. If it is systematically underestimating the probabilities, it is termed "underconfident", and if overestimating the probabilities, "overconfident". Note that in these cases, it may still be improving as a classifier on unseen data, while still showing higher validation loss as calibration degrades.