3 ms·

L3 has open pretraining data, it's just not official for obvious legal reasons: https://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/dat

• menaerus 1y ago
  Wait, whole (english speaking) web content dataset size is ~50TB?
  • zX41ZdbW 1y ago
    Yes, if we take the filtered and deduplicated HTMLs of CommonCrawl. I've made a video on this topic recently: https://www.youtube.com/watch?v=8yH3rY1fZEA https://www.youtube.com/watch?v=8yH3rY1fZEA
    • menaerus 1y ago
      Fun presentation, thanks! 72min ingestion time for ~81TB of data is ~1TB/min or ~19GB/s. Distributed or single-node? Shards? I see 50 jobs are used for parallel ingestion, and I wonder how ~19GB/s was achieved since ingestion rates were far below that figure last time I played around with CH performance. Granted, that was some years ago.
      • zX41ZdbW 1y ago
        Distributed across 20 replicas.