3 ms·
I tried to find out how many "tokens" (I know: depends on the tokenizer) "The Pile" has but couldn't find it. As far as I understand RedPajama has 1.2T (https:
by tosh 3y ago
I tried to find out how many "tokens" (I know: depends on the tokenizer) "The Pile" has but couldn't find it.
As far as I understand RedPajama has 1.2T (https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data) and has a table in the readme listing the main parts and how many tokens each part has.