4 ms·
A child needs about 100KB of "text" a day to learn a language. If anything the data requirements of LLMs are proof positive they can't bear any relation to the
by morelisp 4y ago
A child needs about 100KB of "text" a day to learn a language. If anything the data requirements of LLMs are proof positive they can't bear any relation to the human language faculty.
- pixl97 4y agoI mean we could be far overfeeding our data models too. Of course data models don't take years in real time to train either.
- morelisp 4y ago> I mean we could be far overfeeding our data models too. As far as I know evidence we have suggests the opposite, improvements are still mostly coming from more parameters and more data. > data models don't take years in real time to train either. Given the incommensurate architectures a fairer calculation of learning rate might be Wh - a human brain needs about 500Wh a day, GPT-3 was suspected to take about 1GWh to train.