4 ms·
Correct me if I’m wrong but aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to h
by klabb3 4y ago
Correct me if I’m wrong but aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to have a clear correct answer? This is a stark difference to most real-world human activity.
Add to the fact that LLMs perform much better on questions with a lot of training data.
And also add the hallucinations or more generally: they don’t ask for help or admit they don’t know, they seem unaware of their own confidence levels.
No doubt GPT is fascinating and exciting, but boy we're oversubscribing their abilities. LLMs are even worse than crypto (from a fad perspective) because we naturally anthropomorphize their higher level abilities, which are emergent and not well understood even by experts. And we’re about to plug them straight into critical business flows? Bring the popcorn!
- whatshisface 4y ago>Add to the fact that LLMs perform much better on questions with a lot of training data. The answer key can get a 100% on the exam.
- aperrien 4y agoWould a human with a photographic memory count as possessing an answer key?
- whatshisface 4y agoYes, also a human with an answer key.
- blagie 4y ago> And we’re about to plug them straight into critical business flows? Anyone who thinks this is a problem has never managed flesh-and-blood employees, and especially not minimum wage ones. LLMs don't need to be perfect. The bar they need to meet for a lot of work is just not very high. We're also about a year or two into LLMs, and their capabilities are still increasing rapidly. We don't know where their ceiling is. They might plateau where they are, improve slowly, improve linearly, follow Moore's Law, or head off for singularity.
- l33t233372 4y agoWe are not “a year or two into LLMs”. To name one example, BERT is more than 5 years old.
- FeepingCreature 4y agoIs 110 million parameters really a "large" language model though? Especially since the models gain novel skills as they scale up.
- srcreigh 4y agoSo if a new model is trained with 10x parameters of gpt3, is gpt now no longer an llm?
- blagie 4y agoIt's a little bit weird. In semiconductors, we ran into the same problem. "large scale integration" happened at 500 transistors, and "very large scale integration" at 20,000. For a while, "ultra large scale integration" happened at a million and more qualifiers were added, until everyone decided it was ridiculous, and we went back to VLSI. From my perspective, LLMs are about where the language models start to behave in ways which feel sentient and replace mainstream human tasks, such as making first drafts of emails, code, or legal filings. That breakpoint was around GPT-3. I can't predict the future. When we have 3T parameter models, we might: - Call them LLMs, and group them with GPT-3 - Call them LLMs, but shift the goal posts to where GPT-3 is no longer one - Call them VLLM - Call them AGI However, what's clear to me is that state-of-the-art models with e.g. 3B parameters are qualitatively different from GPT-3 and friends. I don't consider those to be LLMs.
- not_a_shill 4y agoNah, the new ones can just be called Xllms. Then we can just keep adding Xs every generation.
- eyelidlessness 4y ago
- 0cVlTeIATBs 4y ago>aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to have a clear correct answer? Not sure if you mean law exams, but in my experience, engineering exams: yes. Leadership exams: quite the opposite.
- narrator 4y ago>aren’t virtually all tests and exams designed to minimize ambiguity, make them fair or easy to grade and questions are designed to have a clear correct answer? The Bar Exam is designed to be very hard. Almost every single question on it is a trick question of one sort or another.
- l33t233372 4y agoHard is different from ambiguous.
- catach 4y ago"Trick question" may include or be adjacent to "ambiguous" depending on methods.
- macrolime 4y ago>And also add the hallucinations or more generally: they don’t ask for help or admit they don’t know, they seem unaware of their own confidence levels. I'm not so sure about that. If ChatGPT says something wrong and I tell it that's not correct, it will often admit its mistake, but if it is unambiguously correct it will typically keep insisting that its right. Not always though, but it does seem to have some idea about its own confidence level.
- Jensson 4y agoThose are based on similar online conversations. If lots people insist they are right in conversations ChatGPT thinks are similar then it will insist it is right, if many give up then it will give up. ChatGPT has no other way to gauge "confidence" in the outputted text, the computed confidence you get has nothing to do with how truthful the statement is, but how well the text fits given the examples ChatGPT has seen. A person insisting that a wrong statement is right could fit better and then ChatGPT would give that statement high confidence. But still the number computed is 100% unrelated to the tone it responds in.
- qualudeheart 4y agoLLMs will soon replace lawyers and law enforcement. OpenAI DV could debate a supreme court level lawyer.