4 ms·

That's worse than 500 error. Not only is the provided answer is wrong, but the confidence remains high.

• deleted 4y ago
  [deleted]
• kridsdale1 4y ago
  The Internet is overflowing with high confidence low accuracy information produced by biological humans. What could set this apart from them would be an indication to the user of its confidence value, which Watson did for his debut on Jeopardy.
  • palata 4y ago
    Except that I don't think it can have any notion of confidence. It would need to understand the world, for that. This tool just makes it orders of magnitude easier to produce noise. You think the web is overflowing with low accuracy information? This may send us in a whole new level! I actually wonder if this has the potential to break search engines as we know them, because they can't judge the quality of the content.
    • yunwal 4y ago
      These llms are Bayesian models. I’m not quite sure what the calculation would be but it should be possible to get a confidence score from some combination of each probability.
      • whatshisface 4y ago
        The problem is that it would be a confidence score that the words would follow each other in that order in a real text, which would be both way lower than the chances of them being correct (remember, the models are trained to reproduce their training samples), and also more correlated with what people were writing online than with the truth.
      • palata 4y ago
        They have "some confidence metrics" indeed. But those are completely different from what we humans mean in "are you 100% sure that the Earth is flat?"
        • yunwal 4y ago
          Sure but they’re correlated. I think a team like openAI could definitely come up with some kind of useful “confidence score” based on Bayesian probability scores plus some other metrics, even if they’ll never have 100% confidence