30 ms·
> "Fundamentally, LLMs are super-powerful text predictors. Given a prompt, they use machine learning to try and come up with the most likely set of characters t
by raducu 4y ago
> "Fundamentally, LLMs are super-powerful text predictors. Given a prompt, they use machine learning to try and come up with the most likely set of characters that follow the prompt."
I've heard this tossed around and in the old days it used to be true.
BUT in the process of learning to predict text, just like with CNNs that learned to predict image types, it seems GPT learned some very interesting functions that bring it very close to "cognition".
I would not be at all surprised if we found those neurons in GPT, the same as I saw on fast.ai the building blocks of neurons that predicted images back in the day.
- shawabawa3 4y ago> Given a prompt, they use machine learning to try and come up with the most likely set of characters that follow the prompt. Funnily enough this is almost exactly how human children work when they're learning to talk
- raducu 4y agoIt doesn't really matter how we learn the features that we learn; I've heard numerous times that ML models tend to learn architectures/neurons that biological neurons do; I've seen convincing evidence about image recognition back in the day that this is the case. I read that animals dream in the woumb partially to pre-train their nervous system, including vision stuff. But I'm convinced GPT-3/4 did learn a lot of features/generalization/reasoning, even if part of a "text prediction". Sure, it's predicting text... but to do so it learnt some calculus and quantum mechanics -- it's just an example, I don't know the cognitive building blocks to explain how humans/future AI do their reasoning.
- pagutierrezn 4y agoMy mothe-in-law is almost deaf. However, as you said, she is a good example of a "super-powerful text predictor". She almost always predicts what you're talking about even not hearing a single word. And most of the time she offers meaningful answers just as GPT does. But not always (same as GPT) I don't expect, however, that she will program front ends any time soon
- raducu 4y ago> I don't expect, however, that she will program front ends any time soon Liar :) If she wrote/scored as good as Chat GPT-4 does, you'd be worried she's going to steal your job :). I mean, I know I'm lucky my wife doesn't know programming, lest she'd know more about anything that I do.
- TedDoesntTalk 4y agoWhat if you train her with then right data?