6 ms·

This is naive, but can you ask the model to provide a confidence rating for sections of the document?

• UnlockedSecrets 2y ago
  You can ask, and it will be made up not grounded in reality
  • j_bum 2y ago
    Sure, but I’m curious if it would serve to provide some self-regulation. E.g., all of this “thinking” trend that’s happening. It would be interesting if the model does a first pass, scored its individual outputs, then reviews its scores and censors/flags scores that are low. I know it’s all “made up”, but generally I have a lot of success asking the model to give 0-1 ratings on confidence for its answers, especially for new niche questions that are likely out of the training set.
    • rafram 2y ago
      It doesn’t. Asking for confidence doesn’t prompt it to make multiple passes, and there’s no real concept of “passes” when you’re talking about non-reasoning models. The model takes in text and image tokens and spits out the text tokens that logically follow them. You can try asking it to think step by step, or you can use a reasoning model that essentially bakes that behavior into the training data, but I haven’t found that to be very useful for OCR tasks. If the encoded version of your image doesn’t resolve to text in the model’s latent space, it never will, no matter how much the model “reasons” (spits out intermediate text tokens) before giving a final answer.
• thatjoeoverthr 2y ago
  More broadly, it’s not trained to have any self awareness and this is a factor in other “hallucinations”. If you ask, for example, to describe the “marathon crater”, it doesn’t recognize that there’s no such thing in its corpus, but will instead start by writing an answer (“sure! The marathon crater is..”) and freestyle from there. Same if you ask it why it did something, or details about itself, etc. You should access one directly (not through an app like chatGPT) and build a careful suite of tests to learn more. Really fascinating.
  • _delirium 2y ago
    Yes, there’s research showing that models’ self-assessment of probabilities (when you ask them via prompting) don’t even match the same models’ actual probabilities, in cases where you can measure the probabilities directly (e.g. by looking at the logits): https://arxiv.org/abs/2305.13264 https://arxiv.org/abs/2305.13264
    • anon291 2y ago
      Logits are not probabilities... at least not in the way you understand probability. Probabilities mathematically are anything that broadly behaves like a probability, whereas colloquially probabilities represent the likelihood or the preponderance of a particular phenomenon. Logits are not either of those.
      • _delirium 2y ago
        The probability of token generation is a function of the logits. Do you have an actual point related to the linked paper?
        • anon291 2y ago
          That is one way of sampling tokens. It is not the only way. Logits do not map neatly to belief, although it is convenient to behave as if they do
• ttyprintk 2y ago
  It’s not naive; tesseract does this.
  • rafram 2y ago
    Tesseract doesn’t use an LLM. LLMs don’t know how confident they are; Tesseract’s model does.
    • touisteur 2y ago
      With most Machine Learning algorithms I used to get shapley values or other 'explainable AI' metrics (for a large cost compared to simple inference, yes), it's very unsettling and frustrating to work without them now on LLMs.
    • hansvm 2y ago
      Kind of. Tesseract's confidence is just a raw model probability output. You could easily use the entropy associated with each token coming out of an LLM to do the same thing.
      • rafram 2y ago
        True, but LLM token probability doesn't map nearly as cleanly to "how readable was the text".
        • hansvm 2y ago
          Why not though? Both kinds of models jumble around the data and spit out a probability distribution. Why is the tesseract distribution inherently more explainable (aside from the UI/UX problem of the uncertainty being per-token instead of per-character)?