4 ms·

You may not grasp just how large the Common Crawl dataset is. It's been growing steadily at 200-300 TB per month for the last few years. I'm not certain how lar

You may not grasp just how large the Common Crawl dataset is. It's been growing steadily at 200-300 TB per month for the last few years. I'm not certain how large the entire corpus is at this point, but it's almost certainly in the tens to low hundreds of petabytes. (This is significantly larger than the capacity of the entire Sia network, for example.) Storing a dataset of this size and making it available online is not inexpensive. Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down.

new_realist 6y ago
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content. EDIT: the state of the crawls are summarized at https://commoncrawl.github.io/cc-crawl-statistics/ https://commoncrawl.github.io/cc-crawl-statistics/.
- duskwuff 6y ago
  As best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months): https://commoncrawl.github.io/cc-crawl-statistics/plots/crawloverlap https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...
duskwuff 6y ago
(Update: the complete Common Crawl dataset is actually a little smaller than I thought, at 6.4 PB. That's still pretty big, though.)
indolering 6y ago
> Amazon has generously donated their services to handle both of these tasks; it would be foolish to turn them down. Amazon makes plenty of money from people using AWS to processes the data.
- fjeifisjf 6y ago
  That's a win-win.