Interesting. I think the raw original source data would be good to have a copy of if possible and over time tools to curate and sample it should get better. Wh

Interesting. I think the raw original source data would be good to have a copy of if possible and over time tools to curate and sample it should get better. What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of. Any thoughts on the size requirements? Focusing on text.