4 ms·
I certainly feel different. A 5% error rate, to me, is akin to branch prediction, where I occasionally need to rollback but it's infrequent enough that it doesn
by BoxFour 3y ago
I certainly feel different. A 5% error rate, to me, is akin to branch prediction, where I occasionally need to rollback but it's infrequent enough that it doesn't pose a significant issue.
However, when it reaches 25%, I find myself meticulously verifying everything beforehand.
- eminence32 3y agoDo you feel confident in your ability to detect when the LLM is wrong? For me personally, I don't have the confidence to do this , which is why I feel like I need to verify everything, even in the 5% case.
- freedomben 3y agoFor me it comes down to "how important is it to be right"? For many of the queries, it's not that important and if I lose the 5% gamble, it's annoying but ultimately inconsequencial. If it's an important thing to be right about, then I will verify it from both.
- delecti 3y agoI don't know how else to ask this: how are you so okay with disregarding accuracy? What questions are you asking where you don't care if the answer is wrong? I guess I just fundamentally don't understand what the point is. Why not just bookmark the "random article" link on Wikipedia if it doesn't matter anyway?
- sdenton4 3y agoA significant subset of my questions are basically, 'here's the weird statistics thing I would like to do; what are the academic jargon keywords I should look into?' All of the LLMs are pretty good at this, and when it hallucinates a response it's totally a decent indicator that there isn't a lot of literature on the subject.
- famouswaffles 3y agoYou have to be okay with being wrong regardless. The little snippets after a Google search are sometimes wrong. The blogs or links can be misinformed (For example, most seem to be wrong about Staind's new album release date). The idea of perfection is silly because it doesn't exist LLM or not. You're not going to get it so it's a matter how often it's right.
- freedomben 3y agoI ask a fair number of code questions where bugs and stuff will be caught either immediately or shortly after first use. If 5% of those are wrong, totally fine with me. I also sometimes ask questions like "what is the tallest mountain in the US" or "what is the hottest desert on Earth" or similar. If I really need to know that the answer is correct, at a minimum it gives me a name to search for to verify height in feet compared to others, etc.
- BoxFour 3y agoI encounter situations where I used to rely on Stack Overflow, which carries a similar (if not higher) likelihood of being incorrect, often due to outdated information. For instance, I was recently inquiring about a specific task with CMake and consulted ChatGPT. Initially, the response was inaccurate, but it was obviously so when it didn’t compile. Upon reprompting, I received the correct answer.
- _ea1k 3y agoIf it is writing code, I still have to verify it every time. Then again, that is also true when I write code. Hopefully my rate of error is way below 25%, so 25% error might not help me that much. I'd like to think my rate of error is less than 5% too, but that might be close enough that outsourcing the work is still worth it. Maybe still not worth it if I were that hypothetical programmer that writes lots of code but never writes any bugs.