3 ms·
I haven't found a human that answers every single question correctly, either. You know whom to ask a question based off that person's domain of expertise. Well,
by taylodl 2y ago
I haven't found a human that answers every single question correctly, either. You know whom to ask a question based off that person's domain of expertise. Well, AI's domain of expertise is everything (supposedly).
What gets difficult is evaluating the response, but let's not pretend that's any easier to do when interacting with a human. Experts give wrong answers all the time. It's generally other experts who point out wrong answers provided by one of their peers.
My solution? Query multiple LLMs. I'd like to have three so I can establish a quorum on an answer, but I only have two. If they agree then I'm reasonably confident the answer is correct. If they don't agree - well, that's where some digging is required.
To your point, nobody is expecting these systems to be infallible because I think we intuitively understand that nothing knows everything. Wouldn't be surprised if someone wrote a paper on this very topic.
- threeseed 2y agoa) If you ask me about surgery I will say "I don't know". LLMs won't do that. b) Experts may give wrong answers but it will happen once. LLMs will do it over and over again.
- latentsea 2y ago>b) Experts may give wrong answers but it will happen once. LLMs will do it over and over again. Well... Sometimes "experts" will give the wrong answer repeatedly.
- ThunderSizzle 2y agoAssuming they aren't convinced they are wrong - and multiple opinions can be valid while differing. The world isnt always black and white. An AI can never be convinced it's wrong permanently, and sometimes it can't be convinced ed temporarily, depending on the model
- latentsea 2y agoAlthough new models do get trained and replace older ones, so from a users perspective it's not like they'll never change their answers on things. We've seen improvements over time, so while individual models are relatively fixed,the LLM industry itself is much more dynamic.
- ThunderSizzle 2y agoYes, but the annoying part to me is that the model doesn't enhance itself based on my history with it.
- danrob 2y agoThis is a common argument to support the usage of LLMs: "Well, humans do it too." We have many rules, regulations, strategies, patterns, and legions of managers and management philosophy for dealing with humans. With humans, they're incorrect sometimes, yes, and we actively work around their failures. We expect humans to develop over time. We expect them to join a profession and give bad answers a lot. As time goes on, we expect them to produce better answers, and if they don't we have remediations to limit the negative impact they have on our business processes. We fire them. We recommend they transfer to a different discipline. We recommend they go to college. Comparing the successes and failures of LLMs to humans is silly. We would have fired them all by now. The big difference is that computers CAN get every single question correctly. They ARE better than humans. LLMs are a huge step back from the benefits we got from computers.
- taylodl 2y ago> The big difference is that computers CAN get every single question correctly. I emphatically disagree on that point. AFAIK, nobody has been able to demonstrate, even in principle, that omniscience is possible over a domain of sufficient complexity and subtlety. My gut tells me this is related to Gödel's Incompleteness Theorem.
- nunez 2y agoAlso, humans can say "I don't know," a skill that seems impossible for LLMs
- stavros 2y agoWhat's more useful to me is "I'm not sure" or "I'm very sure", but LLMs can't provide you with a level of certainty. They're very sure about everything, including things they make up.
- QuantumGood 2y agoI find this a useful frame of reference: don't assume anyone, anything is correct. Learn to work with what is, not what could be. AI is very helpful to me, as long as I don't have unrealistic expectations.