6 ms·

That's an interesting point about calibration. I hadn't considered that - the Jeopardy framing for ML: don't answer unless you're relatively sure - is actually

That's an interesting point about calibration. I hadn't considered that - the Jeopardy framing for ML: don't answer unless you're relatively sure - is actually the right way to do it, and has since been forgotten in most applications.

Scoundreller 4y ago
I have a similar dream for speech-to-text systems. I wish it could highlight the guesses it wasn’t totally confident in so those could be easily manually reviewed. It shouldn’t be just a binary “didn’t recognize that” and “this is what you said”.
- KRAKRISMOTT 4y ago
  Last I checked that's called a softmax function. Most neural models have this built in.
  - opportune 4y ago
    Been a while since I did DL, but I thought softmax was used in training so that “pick the most likely among X” would converge while training, not because it actually computed the probability distribution among the output classes in isolation. When running inference the softmax function yields the class the model thinks is most likely, it’s not a truly valuable measure of how likely the answer is correct, right? I could be wrong, but let’s take an example like a model trained to take pictures and return either “dog, cat, bird”. If I put in a picture of a fish, softmax will give me a probability distribution across dog, cat, and bird and find eg bird is the most likely - but that does not mean the model is telling me it is confident the image is a bird, just that it’s more confident that it’s a bird vs a dog or cat. I think for DL you’d need to design the model so that “unknown” is an acceptable output and has a lower (but non-zero) error/loss compared to an incorrect proper classification. Maybe people already do that or something similar
    - KRAKRISMOTT 4y ago
      I am curious how that could be implemented. Given a cat/dog classifier, theoretically assuming everything is iid, an "unknown" would be equally far from either category in latent space. So in some ways, you can gate the softmax output if it's too close to 50-50. But this is more of a hack then a real solution to representing the idea of "unknown".
      - opportune 4y ago
        By introducing the concept of “unknown” you are also introducing to the model the concept of a false positive and (if there are inputs either in training or in applying the model that are truly unknown) false negative alongside “wrong”. I think you’d want to make this assessment before applying softmax as with softmax you are only getting the relative probabilities of various classifications, when “unknown” should be done based on absolute probability - I don’t care if Bird is 100x more likely than Dog or Cat if it’s still only 1% chance. I am not a DL genius so if I had to do this, I would probably just make “unknown” a true output and introduce some extra data / augmented data to make it a true input. I am guessing to do this the right way you would need a way to incentivize some convergence towards unknown, maybe involving some loss function looking at the false positive and false negative rate across a batch.
      - nothing0001 4y ago
        Just make a loss function that gives +1 for correct answer, -1 for incorrect answers and 0 for unknown. Since this idea took me like 10 seconds, I suppose something like this must/should have been used before. Unrelated, what happen when we train a model using a discontinuous function? Could the trained model be used to detect some pattern in the data?, for example if the input vector to such a model is a direct sum of two independent variables could such model be used to detect that the problem can be decomposed in two independent problems. Sorry of being off-topic and thinking aloud. Edited: The following result is from (1): In summary, while softmax classifier probabilities are not directly useful as confidence estimates, simple statistics derived from softmax distributions provide a surprisingly effective way to determine whether an example is misclassified or from a different distribution from the training data, as demonstrated by our experimental results (1) https://arxiv.org/abs/1610.02136 https://arxiv.org/abs/1610.02136
- zaroth 4y ago
  I’m still waiting for a system that can reasonably apply a 2nd pass to the text it produces from my input (whether it be tapping the screen or talking) that can identify pure nonsense and make a better guess at what I was trying to input. Just assume I’m not inputting hot garbage. (“I outing hut gabbage” for example is clearly not applying this technique!)
- rprenger 4y ago
  We put this in our ASR based keyboard app back in 2016. Basically kept the ASR decoders beam search candidates around and could use their differences and relative scores to suggest replacements. I would've assumed newer keyboard apps would have a similar option, but I haven't checked: https://techcrunch.com/2016/10/03/baidus-new-talktype-keyboard-app-emphasizes-voice-input-over-typing/ https://techcrunch.com/2016/10/03/baidus-new-talktype-keyboa...
galaxyLogic 4y ago
Right, ChatGPT does it ever say "I don't know"?
- ivalm 4y ago
  Yeah, many factual queries it returns something along the lines of I don't know, but usually more verbose
  - mlyle 4y ago
    The funny thing is, it often answers depending on what it "thinks" it should know based on conversational context, not what it actually knows.
    - ivalm 4y ago
      With sampling, these models can be relatively well-calibrated [0] but I agree that for a single output, it's a crapshoot. [0] https://arxiv.org/abs/2207.08143 https://arxiv.org/abs/2207.08143
    - galaxyLogic 4y ago
      So how does Chatbot know anything about itself? Is it because its creators wrote something about it and then let it read that?
  - Dylan16807 4y ago
    Are you talking about when it objects to answering because it's a large language model blah blah blah? I feel like that doesn't count because most of the time that triggers it's something it does know or can do just fine.
- Stratoscope 4y ago
  Yes, but it uses a lot more words. "What is a typical Hacker News home page?" > Hacker News is a social news website focusing on computer science and entrepreneurship, run by the startup incubator Y Combinator. A typical Hacker News home page consists of a list of articles and links submitted by users, which are then voted up or down by other users. The articles with the most votes will appear at the top of the front page, while older articles and those with fewer votes will be pushed towards the bottom. The site also features discussion forums, job postings, and various other features related to the tech and startup communities. "What is today's Hacker News home page?" > I'm sorry, as an AI language model I do not have access to live data and the current Hacker News homepage can only be viewed by visiting the website directly. However, a typical Hacker News homepage includes articles, links, and discussions submitted by users and ranked based on voting.
roxgib 4y ago
When studying data science we had a major assignment that involved stock predictions. I came out on top of the class because I told my model to only bet when it was >90% sure of the outcome, discarding the result otherwise. The rest of the class didn't even think of that, and lost despite many of them have much better models than me when all the results were considered. They were too focused on the technicalities of the problem too see the big picture, n issue I see a lot in data science students who often struggle to grapple with the problem they're supposed to be solving.
- largepeepee 4y ago
  Depending on results that give "90% sure" outcomes only work under specific constraints like if capital is low or there is near unlimited opportunities. Sounds like the rest of the class didn't read the fine print if that was the model that won.
- PaulHoule 4y ago
  That's exactly what I'm pointing out. If you look at Kaggle answers or arXiv papers you might not even think calibration was a thing, but it's a key tool when going from academic ML to industrial ML. I got into calibration through text retrieval where the TREC methodology rates relevance functions by rank in such a way that you don't get points for being calibrated or well-calibrated. There are numerous things mainstream text retrieval systems don't do, most notably it is hard to build an alerting feature without calibration. Practically you have to set some relevance threshold to avoid getting too much irrelevant stuff. I calibrated a rather good search engine by fitting a curve to the score and found that the best it would ever give is p=0.7 to be relevant and that it gave that very rarely. So even with a calibrated score you wouldn't be able to set a very high threshold or expect to get many documents. Watson gets at this problem by having a large number of question answering models each which has a high p to be right when it is right but each of which also has very low recall, something you can do when using p as a universal score.

Scoundreller 4y ago

I have a similar dream for speech-to-text systems. I wish it could highlight the guesses it wasn’t totally confident in so those could be easily manually reviewed. It shouldn’t be just a binary “didn’t recognize that” and “this is what you said”.

KRAKRISMOTT 4y ago
Last I checked that's called a softmax function. Most neural models have this built in.
- opportune 4y ago
  Been a while since I did DL, but I thought softmax was used in training so that “pick the most likely among X” would converge while training, not because it actually computed the probability distribution among the output classes in isolation. When running inference the softmax function yields the class the model thinks is most likely, it’s not a truly valuable measure of how likely the answer is correct, right? I could be wrong, but let’s take an example like a model trained to take pictures and return either “dog, cat, bird”. If I put in a picture of a fish, softmax will give me a probability distribution across dog, cat, and bird and find eg bird is the most likely - but that does not mean the model is telling me it is confident the image is a bird, just that it’s more confident that it’s a bird vs a dog or cat. I think for DL you’d need to design the model so that “unknown” is an acceptable output and has a lower (but non-zero) error/loss compared to an incorrect proper classification. Maybe people already do that or something similar
  - KRAKRISMOTT 4y ago
    I am curious how that could be implemented. Given a cat/dog classifier, theoretically assuming everything is iid, an "unknown" would be equally far from either category in latent space. So in some ways, you can gate the softmax output if it's too close to 50-50. But this is more of a hack then a real solution to representing the idea of "unknown".
    - opportune 4y ago
      By introducing the concept of “unknown” you are also introducing to the model the concept of a false positive and (if there are inputs either in training or in applying the model that are truly unknown) false negative alongside “wrong”. I think you’d want to make this assessment before applying softmax as with softmax you are only getting the relative probabilities of various classifications, when “unknown” should be done based on absolute probability - I don’t care if Bird is 100x more likely than Dog or Cat if it’s still only 1% chance. I am not a DL genius so if I had to do this, I would probably just make “unknown” a true output and introduce some extra data / augmented data to make it a true input. I am guessing to do this the right way you would need a way to incentivize some convergence towards unknown, maybe involving some loss function looking at the false positive and false negative rate across a batch.
    - nothing0001 4y ago
      Just make a loss function that gives +1 for correct answer, -1 for incorrect answers and 0 for unknown. Since this idea took me like 10 seconds, I suppose something like this must/should have been used before. Unrelated, what happen when we train a model using a discontinuous function? Could the trained model be used to detect some pattern in the data?, for example if the input vector to such a model is a direct sum of two independent variables could such model be used to detect that the problem can be decomposed in two independent problems. Sorry of being off-topic and thinking aloud. Edited: The following result is from (1): In summary, while softmax classifier probabilities are not directly useful as confidence estimates, simple statistics derived from softmax distributions provide a surprisingly effective way to determine whether an example is misclassified or from a different distribution from the training data, as demonstrated by our experimental results (1) https://arxiv.org/abs/1610.02136 https://arxiv.org/abs/1610.02136
zaroth 4y ago
I’m still waiting for a system that can reasonably apply a 2nd pass to the text it produces from my input (whether it be tapping the screen or talking) that can identify pure nonsense and make a better guess at what I was trying to input. Just assume I’m not inputting hot garbage. (“I outing hut gabbage” for example is clearly not applying this technique!)
rprenger 4y ago
We put this in our ASR based keyboard app back in 2016. Basically kept the ASR decoders beam search candidates around and could use their differences and relative scores to suggest replacements. I would've assumed newer keyboard apps would have a similar option, but I haven't checked: https://techcrunch.com/2016/10/03/baidus-new-talktype-keyboard-app-emphasizes-voice-input-over-typing/ https://techcrunch.com/2016/10/03/baidus-new-talktype-keyboa...

galaxyLogic 4y ago

Right, ChatGPT does it ever say "I don't know"?

ivalm 4y ago
Yeah, many factual queries it returns something along the lines of I don't know, but usually more verbose
- mlyle 4y ago
  The funny thing is, it often answers depending on what it "thinks" it should know based on conversational context, not what it actually knows.
  - ivalm 4y ago
    With sampling, these models can be relatively well-calibrated [0] but I agree that for a single output, it's a crapshoot. [0] https://arxiv.org/abs/2207.08143 https://arxiv.org/abs/2207.08143
  - galaxyLogic 4y ago
    So how does Chatbot know anything about itself? Is it because its creators wrote something about it and then let it read that?
- Dylan16807 4y ago
  Are you talking about when it objects to answering because it's a large language model blah blah blah? I feel like that doesn't count because most of the time that triggers it's something it does know or can do just fine.
Stratoscope 4y ago
Yes, but it uses a lot more words. "What is a typical Hacker News home page?" > Hacker News is a social news website focusing on computer science and entrepreneurship, run by the startup incubator Y Combinator. A typical Hacker News home page consists of a list of articles and links submitted by users, which are then voted up or down by other users. The articles with the most votes will appear at the top of the front page, while older articles and those with fewer votes will be pushed towards the bottom. The site also features discussion forums, job postings, and various other features related to the tech and startup communities. "What is today's Hacker News home page?" > I'm sorry, as an AI language model I do not have access to live data and the current Hacker News homepage can only be viewed by visiting the website directly. However, a typical Hacker News homepage includes articles, links, and discussions submitted by users and ranked based on voting.

roxgib 4y ago

When studying data science we had a major assignment that involved stock predictions. I came out on top of the class because I told my model to only bet when it was >90% sure of the outcome, discarding the result otherwise. The rest of the class didn't even think of that, and lost despite many of them have much better models than me when all the results were considered. They were too focused on the technicalities of the problem too see the big picture, n issue I see a lot in data science students who often struggle to grapple with the problem they're supposed to be solving.

largepeepee 4y ago
Depending on results that give "90% sure" outcomes only work under specific constraints like if capital is low or there is near unlimited opportunities. Sounds like the rest of the class didn't read the fine print if that was the model that won.
PaulHoule 4y ago
That's exactly what I'm pointing out. If you look at Kaggle answers or arXiv papers you might not even think calibration was a thing, but it's a key tool when going from academic ML to industrial ML. I got into calibration through text retrieval where the TREC methodology rates relevance functions by rank in such a way that you don't get points for being calibrated or well-calibrated. There are numerous things mainstream text retrieval systems don't do, most notably it is hard to build an alerting feature without calibration. Practically you have to set some relevance threshold to avoid getting too much irrelevant stuff. I calibrated a rather good search engine by fitting a curve to the score and found that the best it would ever give is p=0.7 to be relevant and that it gave that very rarely. So even with a calibrated score you wouldn't be able to set a very high threshold or expect to get many documents. Watson gets at this problem by having a large number of question answering models each which has a high p to be right when it is right but each of which also has very low recall, something you can do when using p as a universal score.