4 ms·
Do you happen to know what the v2 dedup size is when compressed? 32.1TB is quite a bit, but if that compresses down to say 3-6TB, it would be much more manageab
by bick_nyers 3y ago
Do you happen to know what the v2 dedup size is when compressed? 32.1TB is quite a bit, but if that compresses down to say 3-6TB, it would be much more manageable. Code has a lot of whitespace, repetition, and structure/predictability, so I imagine it would compress better than average text.
- spindump8930 3y agoThose sizes refer to the data before processing and filtering. The actual training size was about 3 TB: The Stack v2 is ten times larger than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive cleaning, filtering, and subsampling of the source code, along with the incorporation of other high-quality code-related datasets, we created a training set of approximately 3TB (900B+ tokens). Source: the paper, Section 10 (https://arxiv.org/pdf/2402.19173.pdf https://arxiv.org/pdf/2402.19173.pdf)