5 ms·
> This will hardly seem like a controversial opinion, but LLM are overhyped. As the [excellent] paper points out, LLMs are complex functions that can be embedd
by hodgesrm 4y ago
> This will hardly seem like a controversial opinion, but LLM are overhyped.
As the [excellent] paper points out, LLMs are complex functions that can be embedded in systems to provide plausible answers to a prompt. Here's the money sentence.
LLMs are generative mathematical models of the statistical distribution
of tokens in the vast public corpus of humangenerated text, where the tokens
in question include words, parts of words, or individual characters including
punctuation marks.
Rather than focus on the limitations of this approach to answer general queries, which are manifest, it seems more interesting to ask a different question. Under what circumstances do LLMs give answers that are reliably equivalent to or better than humans? The answer would:
1. Illuminate where we can use LLMs safely.
2. Direct work to make them better.
It's already impressive that within certain scopes ChatGPT gives very good answers, indeed better than most humans.
- dragonwriter 4y ago> Under what circumstances do LLMs give answers that are reliably equivalent to or better than humans? Which humans? Humans give a... fairly wide range of responses.
- TeMPOraL 4y agoAverage humans? Within 1 stdev from the mean?
- dragonwriter 4y ago> Average humans? Within 1 stdev from the mean? This implies that performance has unqiue natural, objective, ratio-level (or at least, a unique consistent interval-level) measure. Otherwise the mean is, itself, meaningless. “How well you answer a question” doesn't seem to fit that, its maybe at best a (somewhat subjective, still) ordinal quality, so the median (or mode) is the only useful average. But I think you'll find that without restricting things more than “humans”, both the median and mode of responses to most prompts is... quite bad.
- hodgesrm 4y agoI'm a history major and love classical history. My first question to ChatGPT was: > Why was Julius Caesar murdered? The answer was the following, which would pass a standard exam question on the topic. It exhibits [the appearance of] multi-layer reasoning and has a nice conclusion. > Julius Caesar was assassinated on the Ides of March in 44 BC by a group of Roman senators who believed he had become too powerful and posed a threat to the Roman Republic. The senators were concerned that Caesar had ambitions to become king, which would have ended the Republic and given him absolute power. The senators were also concerned about Caesar's growing popularity with the people of Rome. They believed that assassinating him was the only way to prevent him from becoming too powerful and destroying the Republic. It's interesting to note that most of the evidence for this answer including 2000 years of interpretation is available in textual form on the Internet. It's easily accessible to LLMs.
- nathan_compton 4y agoThis reminds me that coding with ChatGPT felt like pair programming with a not super smart person who could google and type really fast. Not really fun!