5 ms·

This when that entropy is high i feel like models should have an escape hatch to trigger that the answers overall certainty was low, and hell add it up and scor

This when that entropy is high i feel like models should have an escape hatch to trigger that the answers overall certainty was low, and hell add it up and score it so at the end the user can see if during the generation the certainty of the answer was shit, and should be thrown out ore replaced with a "i'm not sure"

trq_ 2y ago
Yeah that's been my thinking as well. There are definitely times when entropy can be high but not actually be uncertain (again synonyms are the best), but it seems promising. I want to build a visualizer using the OpenAI endpoints.
nopinsight 2y ago
The new Claude Sonnet 3.5 does something like that in my experience.
- trq_ 2y ago
  Yeah wouldn't be surprised if the big labs are doing more than just arg max in the sampling.
radarsat1 2y ago
The problem is that deep net classifiers in general are not well statistically calibrated by default. So while the entropy is often high when they are "not sure", models can very often also be "confidently wrong". So using entropy of the logits as an indicator of confidence can easily be very misleading. I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.
- trq_ 2y ago
  I want to build intuition on this by building a logit visualizer for OpenAI outputs. But from what I've seen so far, you can often trace down a hallucination. Here's an example of someone doing that for 9.9 > 9.11: https://x.com/mengk20/status/1849213929924513905 https://x.com/mengk20/status/1849213929924513905
  - z3t4 2y ago
    I'm thinking versioning. 9.9, 9.10, 9.11 etc because in my native language we use the comma, for decimal separation 9,11 9,22 9,90
- modeless 2y ago
  My understanding is that base models are reasonably well calibrated but the RLHF and other tuning that turns them into chat assistants screws up the calibration.
  - scottmf 2y ago
    There’s much that is lost but imo gpt-4-base would be borderline unusable for most of us compared to its descendants — perhaps even more so than GPT-3 davinci, at least relative to its time. 4 can be an absolute demonic hallucinating machine.
- mumblemumble 2y ago
  I'm not an expert, either, but I've poked at this a little. From what I've seen, token logprobs are correlated enough with correctness of the answer to serve as a useful signal at scale, but it's a weak enough correlation that it probably isn't great for evaluating any single output. My best guess is that somewhere close to the root of the problem is that language models still don't really distinguish syntagmatic and paradigmatic relationships. The examples in this article are a little bit forced in that respect because the alternatives it shows in the illustrations are all paradigmatic alternatives but roughly equivalent from a syntax perspective. This might relate to why, within a given GPT model generation, the earlier versions with more parameters tend to be more prone to hallucination than the newer, smaller, more distilled ones. At least for the old non-context-aware language models (the last time I really spent any serious time digging deep into language models), it was definitely the case that models with more parameters would tend to latch onto syntagmatic information so firmly that it could kind of "overwhelm" the fidelity of representation of semantics. Kind of like a special case of overfitting just for language models.
  - singularity2001 2y ago
    maybe this signal needs to be learned in the final step of reinforcement learning where people decide whether "I don't know" is the right answer
tkellogg 2y ago
Entropix gives you a framework for doing that sort of thing. The architecture is essentially to detect the current state, and then adjust sampler settings or swap in an entirely new sampler strategy. You absolutely could experiment with pushing it into a denial, and I highly encourage you to try it out. The smollm-entropix repo[1] implements the whole thing in a Jupyter notebook, so it's easier to try out ideas. [1]: https://github.com/SinatrasC/entropix-smollm https://github.com/SinatrasC/entropix-smollm
- edwdt 2y ago
  you can also try https://github.com/EdwardDali/EntropixLab https://github.com/EdwardDali/EntropixLab
danielmarkbruce 2y ago
We are almost certainly going to see lots of additional tokens added to vocabularies (like the thinking token, but also could be a "<LOGIC FAIL>" token), lots of sophisticated decoding strategies etc. Just need to generate the data.
vark90 2y ago
Yep, usually it's called abstention or rejection. When people in this field compare various methods of quantifying model uncertainty, they often perform what is called rejection verification. Basically, you continuously reject data points where uncertainty is high, and see how average quality of the remaining outputs increases. A good uncertainty estimate is highly correlated with output quality, and thus low-uncertainty outputs should have higher average quality. We use exactly this approach in our recent benchmark of uncertainty estimation approaches for LLMS [1] and have an open-source library under development [2] which allows for such benchmarking. It also can produce uncertainty scores for a given model output, so ppl in industry can integrate it into their applications as well. [1] https://arxiv.org/abs/2406.15627 https://arxiv.org/abs/2406.15627 [2] https://github.com/IINemo/lm-polygraph https://github.com/IINemo/lm-polygraph
throwawaymaths 2y ago
That's not really trivially compatible with the transformer scheme used to pick tokens and generate results. Transformers are generative AI, not classifiers. They throw out a lot of statistics in the service of forward progress and completing the generative task. This project is a rudimentary attempt to regenerate those stats

trq_ 2y ago

Yeah that's been my thinking as well. There are definitely times when entropy can be high but not actually be uncertain (again synonyms are the best), but it seems promising. I want to build a visualizer using the OpenAI endpoints.

nopinsight 2y ago

The new Claude Sonnet 3.5 does something like that in my experience.

trq_ 2y ago
Yeah wouldn't be surprised if the big labs are doing more than just arg max in the sampling.

radarsat1 2y ago

The problem is that deep net classifiers in general are not well statistically calibrated by default. So while the entropy is often high when they are "not sure", models can very often also be "confidently wrong". So using entropy of the logits as an indicator of confidence can easily be very misleading. I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.

trq_ 2y ago
I want to build intuition on this by building a logit visualizer for OpenAI outputs. But from what I've seen so far, you can often trace down a hallucination. Here's an example of someone doing that for 9.9 > 9.11: https://x.com/mengk20/status/1849213929924513905 https://x.com/mengk20/status/1849213929924513905
- z3t4 2y ago
  I'm thinking versioning. 9.9, 9.10, 9.11 etc because in my native language we use the comma, for decimal separation 9,11 9,22 9,90
modeless 2y ago
My understanding is that base models are reasonably well calibrated but the RLHF and other tuning that turns them into chat assistants screws up the calibration.
- scottmf 2y ago
  There’s much that is lost but imo gpt-4-base would be borderline unusable for most of us compared to its descendants — perhaps even more so than GPT-3 davinci, at least relative to its time. 4 can be an absolute demonic hallucinating machine.
mumblemumble 2y ago
I'm not an expert, either, but I've poked at this a little. From what I've seen, token logprobs are correlated enough with correctness of the answer to serve as a useful signal at scale, but it's a weak enough correlation that it probably isn't great for evaluating any single output. My best guess is that somewhere close to the root of the problem is that language models still don't really distinguish syntagmatic and paradigmatic relationships. The examples in this article are a little bit forced in that respect because the alternatives it shows in the illustrations are all paradigmatic alternatives but roughly equivalent from a syntax perspective. This might relate to why, within a given GPT model generation, the earlier versions with more parameters tend to be more prone to hallucination than the newer, smaller, more distilled ones. At least for the old non-context-aware language models (the last time I really spent any serious time digging deep into language models), it was definitely the case that models with more parameters would tend to latch onto syntagmatic information so firmly that it could kind of "overwhelm" the fidelity of representation of semantics. Kind of like a special case of overfitting just for language models.
- singularity2001 2y ago
  maybe this signal needs to be learned in the final step of reinforcement learning where people decide whether "I don't know" is the right answer

tkellogg 2y ago

Entropix gives you a framework for doing that sort of thing. The architecture is essentially to detect the current state, and then adjust sampler settings or swap in an entirely new sampler strategy. You absolutely could experiment with pushing it into a denial, and I highly encourage you to try it out. The smollm-entropix repo[1] implements the whole thing in a Jupyter notebook, so it's easier to try out ideas. [1]: https://github.com/SinatrasC/entropix-smollm https://github.com/SinatrasC/entropix-smollm

edwdt 2y ago
you can also try https://github.com/EdwardDali/EntropixLab https://github.com/EdwardDali/EntropixLab

danielmarkbruce 2y ago

We are almost certainly going to see lots of additional tokens added to vocabularies (like the thinking token, but also could be a "<LOGIC FAIL>" token), lots of sophisticated decoding strategies etc. Just need to generate the data.

vark90 2y ago

Yep, usually it's called abstention or rejection. When people in this field compare various methods of quantifying model uncertainty, they often perform what is called rejection verification. Basically, you continuously reject data points where uncertainty is high, and see how average quality of the remaining outputs increases. A good uncertainty estimate is highly correlated with output quality, and thus low-uncertainty outputs should have higher average quality. We use exactly this approach in our recent benchmark of uncertainty estimation approaches for LLMS [1] and have an open-source library under development [2] which allows for such benchmarking. It also can produce uncertainty scores for a given model output, so ppl in industry can integrate it into their applications as well. [1] https://arxiv.org/abs/2406.15627 https://arxiv.org/abs/2406.15627 [2] https://github.com/IINemo/lm-polygraph https://github.com/IINemo/lm-polygraph

throwawaymaths 2y ago

That's not really trivially compatible with the transformer scheme used to pick tokens and generate results. Transformers are generative AI, not classifiers. They throw out a lot of statistics in the service of forward progress and completing the generative task. This project is a rudimentary attempt to regenerate those stats