Good resource, admirable intention, great that it simply exists. Good sized index. I see a lot of people subscribe to the idea of this being the feeder to alte

Good resource, admirable intention, great that it simply exists. Good sized index. I see a lot of people subscribe to the idea of this being the feeder to alternative search engines. I'd guess part of the problem with doing things this way is the 'crawl priority' of what the search engine thinks are the next best pages to crawl, it's totally out of their hands or at least, they'd still need to crawl on top of the Common Crawl data. The recent UK CMA report into monopolies in online advertising estimated Google's index to be around 500-600 billion pages in size and Bing's to be 100-200 billion pages in size [0]. Of course, what you define as a 'page' is subjective given URL canonicals and page similarity. At the very least, the common crawl gets around crawl rate limiting problems by being one massive download. Would be interesting to know if there's an appreciable % of site owners blocking it, though going on past data (there is some data in the UK CMA about this also), it's not a huge issue. [0] https://assets.publishing.service.gov.uk/media/5efc57ed3a6f4023d242ed56/Final_report_1_July_2020_.pdf https://assets.publishing.service.gov.uk/media/5efc57ed3a6f4... (page 89)