3 ms·
Anyone know how large it is? They state the 1 trillion token dataset is 5TB. Is it safe to assume this is 5TB * 30 = 150TB? The code in the HuggingFace repo
by gardnr 3y ago
Anyone know how large it is?
They state the 1 trillion token dataset is 5TB.
Is it safe to assume this is 5TB * 30 = 150TB?
The code in the HuggingFace repo downloads data from url base: https://data.together.xyz/redpajama-data-v2/v1.0.0 https://data.together.xyz/redpajama-data-v2/v1.0.0
https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2/blob/main/RedPajama-Data-V2.py https://huggingface.co/datasets/togethercomputer/RedPajama-D...
- zhangce 3y agoIt is around 100TB (84 CommonCrawl dumps, roughly 1TB per dump)
- mauriceweber 3y agoyes, small clarification: the 1TB per dump refers to the head+middle partition of the dataset and includes the text documents and the quality signals. There is another ~700GB for the minhash signatures and 1-1.5TB for the documents in the tail split.