3 ms·
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior sn
by new_realist 6y ago
It's not clear, but it looks like the last crawl was 280 TiB (100 TiB compressed) and contains a snapshot of the web at that point; i.e. you don't need prior snapshots unless you're interested in historical content.
EDIT: the state of the crawls are summarized at https://commoncrawl.github.io/cc-crawl-statistics/ https://commoncrawl.github.io/cc-crawl-statistics/.
- duskwuff 6y agoAs best I can gather, the crawl is an ongoing process, not a series of independent "snapshots". There's almost no overlap in URLs between each crawl archive, although it looks as though there's some repetition on a larger scale (roughly every 2 months): https://commoncrawl.github.io/cc-crawl-statistics/plots/crawloverlap https://commoncrawl.github.io/cc-crawl-statistics/plots/craw...