4 ms·
> For example, to create an LLM such as ChatGPT, you'd start with an enormous quantity of text, then do a lot of computationally-intense statistical analysis to
by rafram 1y ago
> For example, to create an LLM such as ChatGPT, you'd start with an enormous quantity of text, then do a lot of computationally-intense statistical analysis to map out which words and phrases are most likely to appear near to one another. Crunch the numbers long enough, and you end up with something similar to the next-word prediction tool in your phone's text messaging app, except that this tool can generate whole paragraphs of mostly plausible-sounding word salad.
This explanation might've been passable four years ago, but it's woefully out of date now. "Mostly plausible-sounding word salad"?
- mv4 1y agoWhat would be a better explanation today?
- diggan 1y agoI think "mostly plausible-sounding" is, albeit simplified, OK for an analogy I guess. But the "word salad" part gives the impression it doesn't even look like real human text, which it kind of does at the surface. I think it's mostly "word salad" that makes it sound far off from the truth.
- relaxing 1y agoWord salad refers to human writing with poor diction and/or syntax.
- djhn 1y agoOver the past year the chat bots have improved in many ways, but their written output has regressed to the mean: the average RLHF-driven preference. It is word salad, unless you’re a young, underpaid contractor from a country previously colonised by the British or the United States.
- diggan 1y ago> Over the past year the chat bots have improved in many ways, but their written output has regressed to the mean: the average RLHF-driven preference. How could you possibly judge such a diverse set of outputs? There are thousands of models, that can each be steered/programmed with prompts and with a lot parameter-twiddling, it's always impossible you could say "the chat bots" and give some sort of one-size-fits-all judgement of all LLMs. I think your reply shows a bit of ignorance if that's all you've seen. Oxford Dictionaries says "word salad" is "a confused or unintelligible mixture of seemingly random words and phrases", and true, I'm no native speaker, but that's not commonly the output I get from LLMs. Sometimes though, some people’s opinions on the internet feel like word salad, but I guess it's hard to distinguish from bait too.
- djhn 1y agoI meant what I said: chat bots, not models or APIs. Give it a try if you don’t believe me. Try using the leading chat interfaces logged out, from a clean browser and new IP.
- diggan 1y agoSure, I don't doubt that, but still, what do you think these chat bots are using? Or are you talking about ELIZA? If so, what you say now makes a lot of sense.