4 ms·

Imagine you have function that is called "is_true" but it only gets it right 60% of the time. We're doing this within CS/ML, so lets call that "correctness" or

by diggan 1y ago

Imagine you have function that is called "is_true" but it only gets it right 60% of the time. We're doing this within CS/ML, so lets call that "correctness" or something fancier. In order for that function to be valuable, would we need to hit a 100% in correctness? I mean probably most of the time, yeah. But sometimes, maybe even rarely, we're fine with it being less than 100%, but still as high as possible. So in this point of view, it's not a bug or error that it currently sits at 60%, but if we manage to find a way to hit 70%, it would be better. But in order to figure this out, we need to call this "correct for most part, but could be better" concept something. So we look at what we already know and are familiar with, and try to draw parallels, maybe even borrow some names/words.

bee_rider 1y ago
This doesn’t seem too different from my third thing, error (in the “accumulation of numerical error” sense).
timewizard 1y ago
> but if we manage to find a way to hit 70%, it would be better. Yet still absolutely worthless. > "correct for most part, but could be better" concept something. When humans do that we just call it "an error." > so lets call that "correctness" or something The appropriate term is "confidence." These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating. We can quibble over terms but more appropriately this is just "garbage." It's a giant waste of energy and resources that produces flawed results. All of that money and effort could be better used elsewhere.
- vrighter 1y ago
  and even those confidence ratings are useless, imo. If trained with wrong data, it will report high confidence for the wrong answer. And curating a dataset is a black art in the first place
- furyofantares 1y ago
  > These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating. Why do you believe they could give you a confidence rating? They can't, at least not a meaningful one.
- diggan 1y ago
  > Yet still absolutely worthless. Depends on the context, doesn't it? Nothing is usually 100% worthless or 100% "worthy", there are grey areas in life where we're fine with "kind of right, most of the time". Are you saying these scenarios absolutely never exists in your world? I guess I'd be grateful if my life was so easy always.
deleted 1y ago
[deleted]

by diggan 1y ago

bee_rider 1y ago

This doesn’t seem too different from my third thing, error (in the “accumulation of numerical error” sense).

timewizard 1y ago

> but if we manage to find a way to hit 70%, it would be better. Yet still absolutely worthless. > "correct for most part, but could be better" concept something. When humans do that we just call it "an error." > so lets call that "correctness" or something The appropriate term is "confidence." These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating. We can quibble over terms but more appropriately this is just "garbage." It's a giant waste of energy and resources that produces flawed results. All of that money and effort could be better used elsewhere.

vrighter 1y ago
and even those confidence ratings are useless, imo. If trained with wrong data, it will report high confidence for the wrong answer. And curating a dataset is a black art in the first place
furyofantares 1y ago
> These LLM tools all could give you a confidence rating with each and every "fact" it attempts to relay to you. Of course they don't actually do that because no one would use a tool that confidently gives you answers based on a 70% self confidence rating. Why do you believe they could give you a confidence rating? They can't, at least not a meaningful one.
diggan 1y ago
> Yet still absolutely worthless. Depends on the context, doesn't it? Nothing is usually 100% worthless or 100% "worthy", there are grey areas in life where we're fine with "kind of right, most of the time". Are you saying these scenarios absolutely never exists in your world? I guess I'd be grateful if my life was so easy always.

deleted 1y ago

[deleted]