4 ms·
In such discourse I never see discussion on this: There is no doubt that LLMs have gotten more accurate as newer models were released. At what point should we
by simianwords 1y ago
In such discourse I never see discussion on this:
There is no doubt that LLMs have gotten more accurate as newer models were released. At what point should we say "look this is accurate enough to be useful"?
We should acknowledge that nothing is ever 100% accurate. You won't go to a doctor expecting 100% accuracy. You know that the doctor's accuracy is high enough for the effort of making an appointment, listening to them to be worth it. Maybe they are 60% accurate?
My point is that LLM's are maybe at 20-30% accuracy where the benefit clearly exists even if we collectively acknowledge that 20-30% is not that high.
I find it amusing to think about an LLM that is 1% accurate (which could have been achieved way earlier in 2010's). What could have been possible with such an LLM with the right mindset?
- demaga 1y agoRealistic-looking lorem ipsum
- gambiting 1y ago>>You won't go to a doctor expecting 100% accuracy. The way LLMs work at the moment is equivalent to going to a doctor with a set of symptoms, and the doctor telling you "ah yes, of course, it's illness X and you need to take medicine Y to cure it" and then you check and neither X nor Y exists. That's not "accuracy" that's just straight up fraud? I wouldn't have any problem with Google's AI saying "I don't know" or "I don't have enough sources to provide a good answer, but my best approximation is this". But I literally battle misinformation produced by Google's AI search every single day, because it's telling people actual made up facts that don't exist.
- simianwords 1y agoI have a system prompt that gives me probability estimates to everything the LLM claims. Its a simple fix for your problem.
- gambiting 1y agoMy problem is people coming to communities I'm a part of with information they got "from Google" and that information being 100% wrong. Not sure how your prompt helps with that, I need Google to fix their system first.
- _shantaram 1y agoHow does that work if the LLM is the one generating the probabilities too?
- simianwords 1y agoaccording to you, if a human makes a prediction with some probability estimate it is useless because the estimate itself is inaccurate (hence probability "estimate"). in reality nothing needs to be 100% accurate for it to be useful including the estimate of probability itself.
- gambiting 1y agoIt's weird to make an assumption about OPs position and argue with that instead of what they actually wrote. Also, why make it so personal? I think it was a fair question to ask - you didn't answer how it works - just got weirdly defensive about it.
- simianwords 1y agohey that was not my intention, it was to bring to light that we assign probability estimates ourselves to our own predictions despite the estimates being not 100% accurate.
- otabdeveloper4 1y agoHate to break it to you, but the "probability estimates" it spits out are also complete bullshit.
- 1y ago