4 ms·

GPT-3: 174B parameters. Phi-2: 2.7B parameters. Indeed, parameter-wise that is small, 65 times smaller. However... GPT-3: trained with 300B tokens. Phi-2: tra

GPT-3: 174B parameters. Phi-2: 2.7B parameters. Indeed, parameter-wise that is small, 65 times smaller. However... GPT-3: trained with 300B tokens. Phi-2: trained with 1400B tokens. The volume of training data, on the other hand, is around 5 times larger. For curiosity, just the other day I calculated that a human baby learns a language with around 30M "token-equivalents" of learning data. This sounds, to me, a reasonable argument for innatism, that is, human "architecture" is biologically geared for language acquisition and contains some strong "guides" or constraints that reduce the hypothesis space of "possible human languages". I wonder if language models are able to find similar architectures that would allow learning with less data.

gjm11 3y ago
The baby, unlike the LLM, isn't learning just from a stream of text. Anything the LLM knows about the world that a lot of that text describes has been inferred from the structure of the text, but a baby learning (say) English learns what a car is by having someone point at a car and say "car", etc. I think this means (1) that there's a lot more than those 30M tokens of actual input and (2) the input is better structured for teaching the baby about the world as well as the language.
- lumost 3y ago
  The LLM is also trained to predict next token given a sequence of "real" tokens. This means that any case where the LLM predicts tokens that don't look like the "real" token string is Out of Domain. Children learn in an iterative manner generating language, self-evaluating their own language, given validations from parents, and interactions with others. We've also cultivated a curriculum learning approach to make sure that students are always learning language. We really don't know how small of a corpus is required for an LLM to learn. Perhaps there is an optimal curriculum which could be generated that demonstrates progressively more complex language and knowledge tasks to keep gradients large throughout training. Or a larger LLM could generate new training examples based on the cross-entropy loss of a given batch.
- GolDDranks 3y ago
  Of course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models" through multimodal input would help scaffolding language ability with less data.
  - candiodari 3y ago
    I wonder if it isn't 100x more effective for the baby that they don't learn from language per se, they learn from direct feedback. They try to duplicate/imitate others, but generally they get immediate feedback (to the point where it's becomes annoying) about exactly what they're doing/alternatives/... I've always found that this works, but adults often don't like it. Learning from direct feedback, having someone immediately check your work, so to speak, helps enormously. Even if it's in text (I learned C++ on IRC and mailing lists, for example). The books help, but you never have any direction.
  - lostmsu 3y ago
    How did you come up with "less data"? Video and audio are much larger then text even when compressed. You can put whole Dune book original series in space of a single MP3 song, and the entire text of English Wikipedia in a single high quality movie.
    - GolDDranks 3y ago
      I'm talking about linguistic information here, because I'm interested in linguistics and language learning. Humans abstract phonemes from sound, and morphemes/lexical items from phonemes. Once we have a rough measure of how many words of input the toddlers get, we can roughly equate the amounts of _linguistic_ data that goes into acquiring linguistic capabilities. While GPT gets tokens as input, from the lexical lexical level onward, there can be an argument that we can roughly estimate the volume of the _linguistic_ data that goes in to the system regardless of earlier decoding stages. I don't claim that there are a lot of other kinds of data feeding into a human brain, and the modality is totally different. (Multimodal-interactive vs. "predict the next token") GPT, therefore, is at disadvantage in that it needs to learn stuff – other than linguistic stuff – _only_ from linguistic input, whereas a human baby can use world model gained through other modalities as a scaffolding. But for acquiring linguistic capabilities, auditory and visual data seems hardly relevant as input data, other than helping building that the scaffolding that can help "top-down" understanding.
- eurekin 3y ago
  Also, a Baby is extending the learning set with iterated synthetic dreaming material :)
- seanmcdirmid 3y ago
  Won't we get the same if we train LLMs on video that has people speaking but things seen that might be related? Training data is easy to come by with cameras (although privacy concerns have to be addressed, or might prevent this from happening at all). Surely someone has thought of this already.
ckcheng 3y ago
> a human baby learns a language with around 30M "token-equivalents" of learning data I wonder if there’s a standard data set that’s reasonably like that? A common crawl of baby books and parents speech commonly heard by babies...
- GolDDranks 3y ago
  There is a wearable system for researching infants' linguistic environment called "The Language ENvironment Analysis system (LENA)". This review collects and analyses available data[1]. I just took an average AWC (adult word count) per day, and roughly calculated how much linguistic input infants get during their first 4 years. Of course, it's just a kind of a Fermi estimate, but the scale should be about right: 10000 times less input than GPT-3. It would be an interesting challenge to create a "baby-like" dataset. I guess a system like this could help collecting it. [1]: https://pubmed.ncbi.nlm.nih.gov/28824021/ https://pubmed.ncbi.nlm.nih.gov/28824021/
- rileyphone 3y ago
  TinyStories is pretty close (but synthetic). It’s been used to train a few tiny models.
YetAnotherNick 3y ago
> Phi-2: trained with 1400B tokens. Where did you get the information. Not doubting you, just I couldn't find it.
- GolDDranks 3y ago
  It says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding."
  - YetAnotherNick 3y ago
    Phi 1 was trained on 7B tokens. This is quite a jump then.
  - nojvek 3y ago
    I wish they shared their 1.4T dataset. I love that the RedPajama dataset is openly shared on HuggingFace.
Der_Einzige 3y ago
No one ever brings up the role of unsupervised learning for early children. Children don't have labels and only limited capabilities to "self-supervise" until they are toddlers. The overwhelming majority of information that they take as input is "clustered" or learned in an unsupervised fashion. Baby learns that apple and orange are different objects before it knows anything else, like names or properties of those objects. Babies take in far more data (but not text!) in an information theoretic sense than ChatGPT does. This is why I think these ideas that LLMs take far too much data for them to get good is sort of not true
- nojvek 3y ago
  The pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between tokens/words. I agree though - what’s missing in neural architectures is advanced clustering capabilities purely from observing without any labels. Then the labeling phase is cheap and doesn’t require as much data.
HarHarVeryFunny 3y ago
It's an apples and oranges comparison due to different training sets (as well as model sizes), but it seems generally their tests are indicating Phi-2 performing (in their specialized domains) at level of a more general model 5-10x it's size, not 100x. As far as training data, it seems it's more the quality/focus of the data than the quantity. They refer to "textbook quality data", unlike the LLMs like GPT-3 that are trained on web scrapes etc.
deafpolygon 3y ago
I really wish people would stop comparing LLMs to human learning. They're really not equivalent, nowhere nearly as similar - I understand that LLMs are pretty impressive, but it's really time to stop anthropomorphizing them.
- GolDDranks 3y ago
  Why though? It's interesting. I'm not claiming it's equivalent or even remotely similar. (So I agree with your point about _anthropomorphising_) But as a former linguist, in first time history, we witness a system other than humans that exhibits language-like behaviour. Studying these system by no means tells us anything about how _humans_ learn language. But it gives us ideas and viewpoints of what's possible in the "systems-learning-languages" space.

gjm11 3y ago

The baby, unlike the LLM, isn't learning just from a stream of text. Anything the LLM knows about the world that a lot of that text describes has been inferred from the structure of the text, but a baby learning (say) English learns what a car is by having someone point at a car and say "car", etc. I think this means (1) that there's a lot more than those 30M tokens of actual input and (2) the input is better structured for teaching the baby about the world as well as the language.

lumost 3y ago
The LLM is also trained to predict next token given a sequence of "real" tokens. This means that any case where the LLM predicts tokens that don't look like the "real" token string is Out of Domain. Children learn in an iterative manner generating language, self-evaluating their own language, given validations from parents, and interactions with others. We've also cultivated a curriculum learning approach to make sure that students are always learning language. We really don't know how small of a corpus is required for an LLM to learn. Perhaps there is an optimal curriculum which could be generated that demonstrates progressively more complex language and knowledge tasks to keep gradients large throughout training. Or a larger LLM could generate new training examples based on the cross-entropy loss of a given batch.
GolDDranks 3y ago
Of course, you are totally correct about the baby not being limited to text input. It would be interesting to see in the future whether building "world models" through multimodal input would help scaffolding language ability with less data.
- candiodari 3y ago
  I wonder if it isn't 100x more effective for the baby that they don't learn from language per se, they learn from direct feedback. They try to duplicate/imitate others, but generally they get immediate feedback (to the point where it's becomes annoying) about exactly what they're doing/alternatives/... I've always found that this works, but adults often don't like it. Learning from direct feedback, having someone immediately check your work, so to speak, helps enormously. Even if it's in text (I learned C++ on IRC and mailing lists, for example). The books help, but you never have any direction.
- lostmsu 3y ago
  How did you come up with "less data"? Video and audio are much larger then text even when compressed. You can put whole Dune book original series in space of a single MP3 song, and the entire text of English Wikipedia in a single high quality movie.
  - GolDDranks 3y ago
    I'm talking about linguistic information here, because I'm interested in linguistics and language learning. Humans abstract phonemes from sound, and morphemes/lexical items from phonemes. Once we have a rough measure of how many words of input the toddlers get, we can roughly equate the amounts of _linguistic_ data that goes into acquiring linguistic capabilities. While GPT gets tokens as input, from the lexical lexical level onward, there can be an argument that we can roughly estimate the volume of the _linguistic_ data that goes in to the system regardless of earlier decoding stages. I don't claim that there are a lot of other kinds of data feeding into a human brain, and the modality is totally different. (Multimodal-interactive vs. "predict the next token") GPT, therefore, is at disadvantage in that it needs to learn stuff – other than linguistic stuff – _only_ from linguistic input, whereas a human baby can use world model gained through other modalities as a scaffolding. But for acquiring linguistic capabilities, auditory and visual data seems hardly relevant as input data, other than helping building that the scaffolding that can help "top-down" understanding.
eurekin 3y ago
Also, a Baby is extending the learning set with iterated synthetic dreaming material :)
seanmcdirmid 3y ago
Won't we get the same if we train LLMs on video that has people speaking but things seen that might be related? Training data is easy to come by with cameras (although privacy concerns have to be addressed, or might prevent this from happening at all). Surely someone has thought of this already.

ckcheng 3y ago

> a human baby learns a language with around 30M "token-equivalents" of learning data I wonder if there’s a standard data set that’s reasonably like that? A common crawl of baby books and parents speech commonly heard by babies...

GolDDranks 3y ago
There is a wearable system for researching infants' linguistic environment called "The Language ENvironment Analysis system (LENA)". This review collects and analyses available data[1]. I just took an average AWC (adult word count) per day, and roughly calculated how much linguistic input infants get during their first 4 years. Of course, it's just a kind of a Fermi estimate, but the scale should be about right: 10000 times less input than GPT-3. It would be an interesting challenge to create a "baby-like" dataset. I guess a system like this could help collecting it. [1]: https://pubmed.ncbi.nlm.nih.gov/28824021/ https://pubmed.ncbi.nlm.nih.gov/28824021/
rileyphone 3y ago
TinyStories is pretty close (but synthetic). It’s been used to train a few tiny models.

YetAnotherNick 3y ago

> Phi-2: trained with 1400B tokens. Where did you get the information. Not doubting you, just I couldn't find it.

GolDDranks 3y ago
It says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding."
- YetAnotherNick 3y ago
  Phi 1 was trained on 7B tokens. This is quite a jump then.
- nojvek 3y ago
  I wish they shared their 1.4T dataset. I love that the RedPajama dataset is openly shared on HuggingFace.

Der_Einzige 3y ago

No one ever brings up the role of unsupervised learning for early children. Children don't have labels and only limited capabilities to "self-supervise" until they are toddlers. The overwhelming majority of information that they take as input is "clustered" or learned in an unsupervised fashion. Baby learns that apple and orange are different objects before it knows anything else, like names or properties of those objects. Babies take in far more data (but not text!) in an information theoretic sense than ChatGPT does. This is why I think these ideas that LLMs take far too much data for them to get good is sort of not true

nojvek 3y ago
The pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between tokens/words. I agree though - what’s missing in neural architectures is advanced clustering capabilities purely from observing without any labels. Then the labeling phase is cheap and doesn’t require as much data.

HarHarVeryFunny 3y ago

It's an apples and oranges comparison due to different training sets (as well as model sizes), but it seems generally their tests are indicating Phi-2 performing (in their specialized domains) at level of a more general model 5-10x it's size, not 100x. As far as training data, it seems it's more the quality/focus of the data than the quantity. They refer to "textbook quality data", unlike the LLMs like GPT-3 that are trained on web scrapes etc.

deafpolygon 3y ago

I really wish people would stop comparing LLMs to human learning. They're really not equivalent, nowhere nearly as similar - I understand that LLMs are pretty impressive, but it's really time to stop anthropomorphizing them.

GolDDranks 3y ago
Why though? It's interesting. I'm not claiming it's equivalent or even remotely similar. (So I agree with your point about _anthropomorphising_) But as a former linguist, in first time history, we witness a system other than humans that exhibits language-like behaviour. Studying these system by no means tells us anything about how _humans_ learn language. But it gives us ideas and viewpoints of what's possible in the "systems-learning-languages" space.