2 ms·
> That the AIs need enormous amount of data I was surprised to learn that it didn’t take an enormous amount of data to train Llama. Meta used 1.4 trillion tok
by spacebanana7 3y ago
> That the AIs need enormous amount of data
I was surprised to learn that it didn’t take an enormous amount of data to train Llama.
Meta used 1.4 trillion tokens training Llama, but that’s only about a million times the Harry Potter collection [1]. Given that the kindle store has 12 million books [2], it’s credible to get 1.4T tokens just from that. Twitter has 200 billion new tweets per year, so it’d only take 7 tokens per Tweet for that to produce a Llama sized dataset every year.
[1] https://en.wikipedia.org/wiki/LLaMA https://en.wikipedia.org/wiki/LLaMA
[2] https://blog.fostergrant.co.uk/2017/08/03/word-counts-popular-books-world https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula... (1 million words)
[3] https://www.omnicoreagency.com/twitter-statistics https://www.omnicoreagency.com/twitter-statistics