3 ms·
> The pre-training process consists of three stages. In the first stage (S1), the model was pretrained on over 30 trillion tokens with a context length of 4K to
by miohtama 2y ago
> The pre-training process consists of three stages. In the first stage (S1), the model was pretrained on over 30 trillion tokens with a context length of 4K tokens. This stage provided the model with basic language skills and general knowledge.
As this is in trillions, where does this amount of material come from?
- tough 2y agoSynthetic Data (after reasoning breakthroughs feels like more AI laabs are betting for synthetic data to scale.) wonder at what price
- Havoc 2y agoIf they’re using vision models to extract pdf data then they can’t be shy on throwing money at it
- bionhoward 2y agoThe raw CommonCrawl has 100 trillion tokens, admittedly some duplicated. RedPajama has 30T deduplicated. That’s most of the way there, before including PDFs and Alibaba’s other data sources (Does Common Crawl include Chinese pages? Edit: Yes)