5 ms·
> Are you telling me that today’s model gets this table 85% right and the next version will get it 85.5 or 91% correct? That doesn’t help me. If there are mista
by FeepingCreature 2y ago
> Are you telling me that today’s model gets this table 85% right and the next version will get it 85.5 or 91% correct? That doesn’t help me. If there are mistakes in the table, it doesn’t matter how many there are - I can’t trust it. If, on the other hand, you think that these models will go to being 100% right, that would change everything, but that would also be a binary change in the nature of these systems, not a percentage change, and we don’t know if that’s even possible.
Of course, humans also make mistakes. There is a percentage, usually depending on the task but always below 100%, where the work is good enough to use, because that's how human labor works.
- deleted 2y ago[deleted]
- semi-extrinsic 2y agoIf I'm paying a human, even a student working part-time or something, I expect "concrete facts extracted from the literature" to be correct at least 99.99% of the time. There is a huge gap from 85% to 99.99%.
- anthuswilliams 2y agoThis does not at all resonate with my experience with human researchers, even highly paid ones. You still have to do a lot of work to verify their claims.
- FeepingCreature 2y agoSure but there's no step change.
- nuancebydefault 2y agoYou can expect that from a human, but if you don't know their reputation, you'd be lucky with the 85 percent. How do you even know if they understood the question correctly, used trusted sources, correct statistical models etc?
- smusamashah 2y agoA human WILL NOT make up non-existent facts, URLs, libraries and all other things. Unless they deliberately want to deceive. They can make mistake in understanding something and will be able to explain those mistakes in most cases. LLM and Human mistakes ARE NOT same.
- jsjohnst 2y ago> A human WILL NOT make up non-existent facts Categorically not true and there’s so many examples of this in every day practice that I can’t help but feel you’re saying this to disprove your own statement.
- dingnuts 2y agoIt's absolutely true. Humans misremember details but I'll ask an LLM what function I use to do something and it'll literally tell me a function exists named DoExactlyWhatIWant and humans never do that unless they're liars. And I don't go to liars for information -- same reason I don't trust LLMs. Tell me, does an LLM know when it lies?
- Workaccount2 2y agoI don't see a functional difference between misremembering details and lying. Sure one is innocent and one malicious, but functionally a compiler doesn't care if the source of your error is evil or not. An LLM doesn't know when it lies, but a human also doesn't know when they are (innocently) wrong.
- smusamashah 2y agoThe crucial difference is between misremembering occasionally VS consistently. You won't give any job to that kind of person.
- FeepingCreature 2y agoNon-liar humans do in fact make mistakes.
- rldjbpin 2y agohumans often don't trust (certain) other humans too. but replacing that with a random number(/token) generator is more reliable to someone, then more power to them. there is value to be had in the output of this tool. but personally i would not trust it without going through the sources and verifying the result.