3 ms·

> a human baby learns a language with around 30M "token-equivalents" of learning data I wonder if there’s a standard data set that’s reasonably like that? A co

• GolDDranks 3y ago
  There is a wearable system for researching infants' linguistic environment called "The Language ENvironment Analysis system (LENA)". This review collects and analyses available data[1]. I just took an average AWC (adult word count) per day, and roughly calculated how much linguistic input infants get during their first 4 years. Of course, it's just a kind of a Fermi estimate, but the scale should be about right: 10000 times less input than GPT-3. It would be an interesting challenge to create a "baby-like" dataset. I guess a system like this could help collecting it. [1]: https://pubmed.ncbi.nlm.nih.gov/28824021/ https://pubmed.ncbi.nlm.nih.gov/28824021/
• rileyphone 3y ago
  TinyStories is pretty close (but synthetic). It’s been used to train a few tiny models.