4 ms·

>The second link main claim seems to be contradicted by their Figure 12 left panel which shows ~0 correlation between model-predicted and actual truth. The cla

>The second link main claim seems to be contradicted by their Figure 12 left panel which shows ~0 correlation between model-predicted and actual truth. The claim in the abstract is: """We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks.""" The plot is much denser in the origin and top right. How is that 0 correlation ? Depending on the number of their held-out test set, that could be pretty strong correlation even. And how does that contradict the claims they've made, especially on calibration (Fig 13 down) ?

foobarqux 2y ago
Figure 13 right panel also shows there isn't a y=x relationship on out-of-sample tests. First we agree by observation that outside of the top-right and bottom-left corners there isn't any meaningful relationship in the data, regardless of what the numerical value of the correlation is. Second, in those corners it is not clear to me what the relationship is but it looks flattish (i.e. if the ground truth is ~0 then the model-guess-for-truth could be anywhere from 0 to 0.5). This is also consistent with the general behavior displayed in figure 13. If you have some other interpretation of the data you should lay it out. The authors certainly did not do that. edit: By the way there are people working on a re-sampling algorithm based on the entropy and variance of the output logits called entropix: if the output probabilities for the next token are spread evenly for example (and not have overwhelming probability for a single token) they prompt for additional clarification. They don't really claim anything like the model "knows" whether it's wrong but they say it improves performance.
- famouswaffles 2y ago
  >Figure 13 right panel also shows there isn't a y=x relationship on out-of-sample tests. A y=x relationship is not necessary for meaningful correlation and the abstract is quite clear on out of sample performance either way. >Second, in those corners it is not clear to me what the relationship is but it looks flattish (i.e. if the ground truth is ~0 then the model-guess-for-truth could be anywhere from 0 to 0.5). The upper bound for guess-for-truth is not as important as the frequency. Yes it could guess 0.5 for 0 but how often compared to reasonable numbers? A test set on TriviaQA could well be thousands of questions. >edit: By the way there are people working on a re-sampling algorithm based on the entropy and variance of the output logits called entropix I know about entropix. It hinges strongly on the model's representations. If it works, then choosing to call it "knowing" or not is just semantics.
  - foobarqux 2y ago
    > A y=x relationship is not necessary for meaningful correlation I’m not concerned with correlation (which may or may not indicate an actual relationship) per se, I’m concerned with whether there is a meaningful relationship between predicted and actual. The 12 plot clearly shows that predicted isn’t tracking actual even in the corners. I think one of the lines (predicting 0% but actual is like 40%, going from memory on my phone) of Figure 13 right even more clearly shows there isn’t a meaningful relationship. In any case the authors haven’t made any argument about how those plots support their arguments and I don’t think you can either. > the abstract is quite clear on out of sample performance either way. Yes I’m saying the abstract is not supported by the results. You might as well say the title is very clear. > The upper bound for guess-for-truth is not as important as the frequency. Yes it could guess 0.5 for 0 but how often compared to reasonable numbers? A test set on TriviaQA could well be thousands of questions. Now we’ve gone from “the paper shows” to speculating about what the paper might have shown (and even that is probably not possible based on the Figure 13 line I described above) > choosing to call it "knowing" or not is just semantics. Yes it’s semantics but that implies it’s meaningless to use the term instead of actual underlying properties.
    - famouswaffles 2y ago
      I think the lines of the largest model show a meaningful relationship. The abstract also quite literally states that models struggle with out of distribution tests so again, what is the contradiction here ? Would it have been hard to simply say you found the results unconvincing? There is nothing contradictory in the paper.
      - foobarqux 2y ago
        For the red Lambada line in Fig 13 when the model predicts ~0 the ground truth is 0.7. No one can look at that line and say there is a meaningful relationship. The Py Func Synthesis line also doesn't look good above 0.3-0.4. > The abstract also quite literally states that models struggle with out of distribution tests so again, what is the contradiction here ? Out of distribution is the only test that matters. If it doesn't work out of distribution it doesn't work. Surely you know that. > Would it have been hard to simply say you found the results unconvincing? Anyone can look at the graphs, especially Figure 13, and see this isn't a matter of opinion. > There is nothing contradictory in the paper. The results contradict the claim the titular claim that "Language Models (Mostly) Know What They Know".

foobarqux 2y ago

Figure 13 right panel also shows there isn't a y=x relationship on out-of-sample tests. First we agree by observation that outside of the top-right and bottom-left corners there isn't any meaningful relationship in the data, regardless of what the numerical value of the correlation is. Second, in those corners it is not clear to me what the relationship is but it looks flattish (i.e. if the ground truth is ~0 then the model-guess-for-truth could be anywhere from 0 to 0.5). This is also consistent with the general behavior displayed in figure 13. If you have some other interpretation of the data you should lay it out. The authors certainly did not do that. edit: By the way there are people working on a re-sampling algorithm based on the entropy and variance of the output logits called entropix: if the output probabilities for the next token are spread evenly for example (and not have overwhelming probability for a single token) they prompt for additional clarification. They don't really claim anything like the model "knows" whether it's wrong but they say it improves performance.

famouswaffles 2y ago
>Figure 13 right panel also shows there isn't a y=x relationship on out-of-sample tests. A y=x relationship is not necessary for meaningful correlation and the abstract is quite clear on out of sample performance either way. >Second, in those corners it is not clear to me what the relationship is but it looks flattish (i.e. if the ground truth is ~0 then the model-guess-for-truth could be anywhere from 0 to 0.5). The upper bound for guess-for-truth is not as important as the frequency. Yes it could guess 0.5 for 0 but how often compared to reasonable numbers? A test set on TriviaQA could well be thousands of questions. >edit: By the way there are people working on a re-sampling algorithm based on the entropy and variance of the output logits called entropix I know about entropix. It hinges strongly on the model's representations. If it works, then choosing to call it "knowing" or not is just semantics.
- foobarqux 2y ago
  > A y=x relationship is not necessary for meaningful correlation I’m not concerned with correlation (which may or may not indicate an actual relationship) per se, I’m concerned with whether there is a meaningful relationship between predicted and actual. The 12 plot clearly shows that predicted isn’t tracking actual even in the corners. I think one of the lines (predicting 0% but actual is like 40%, going from memory on my phone) of Figure 13 right even more clearly shows there isn’t a meaningful relationship. In any case the authors haven’t made any argument about how those plots support their arguments and I don’t think you can either. > the abstract is quite clear on out of sample performance either way. Yes I’m saying the abstract is not supported by the results. You might as well say the title is very clear. > The upper bound for guess-for-truth is not as important as the frequency. Yes it could guess 0.5 for 0 but how often compared to reasonable numbers? A test set on TriviaQA could well be thousands of questions. Now we’ve gone from “the paper shows” to speculating about what the paper might have shown (and even that is probably not possible based on the Figure 13 line I described above) > choosing to call it "knowing" or not is just semantics. Yes it’s semantics but that implies it’s meaningless to use the term instead of actual underlying properties.
  - famouswaffles 2y ago
    I think the lines of the largest model show a meaningful relationship. The abstract also quite literally states that models struggle with out of distribution tests so again, what is the contradiction here ? Would it have been hard to simply say you found the results unconvincing? There is nothing contradictory in the paper.
    - foobarqux 2y ago
      For the red Lambada line in Fig 13 when the model predicts ~0 the ground truth is 0.7. No one can look at that line and say there is a meaningful relationship. The Py Func Synthesis line also doesn't look good above 0.3-0.4. > The abstract also quite literally states that models struggle with out of distribution tests so again, what is the contradiction here ? Out of distribution is the only test that matters. If it doesn't work out of distribution it doesn't work. Surely you know that. > Would it have been hard to simply say you found the results unconvincing? Anyone can look at the graphs, especially Figure 13, and see this isn't a matter of opinion. > There is nothing contradictory in the paper. The results contradict the claim the titular claim that "Language Models (Mostly) Know What They Know".