3 ms·
> Common Crawl, with over one billion, nine hundred and seventy thousand web pages in their archive: 345TB. Common Crawl is 300 billion webpages and 10 petabyt
by ccgreg 6mo ago
> Common Crawl, with over one billion, nine hundred and seventy thousand web pages in their archive: 345TB.
Common Crawl is 300 billion webpages and 10 petabytes. I suppose your number is 1 of our 122 crawls.
- genewitch 6mo agooh, i didn't see that the 1.97 billion pages were crawled in a 11 day period earlier this month. either way, nearly 2,000,000,000 pages fit in ~third of a petabyte... p.s. thanks for correcting me, i was using this information for something else, and now it's correct!