#1 can somewhat be substantiated by some technical info, without knowing the true number of pages. - Using DNS "zone files", the DNS database for TLDs (which a

#1 can somewhat be substantiated by some technical info, without knowing the true number of pages. - Using DNS "zone files", the DNS database for TLDs (which are not available for all, but most) show there's circa 200 million domains registered at any moment - A large percentage of these are parked, i.e. no unique content. - Many domains are "tasted", i.e. bought, are alive for a few days then disappear, so potentially you waste time crawling them - Lots of sites are database driven and can result in millions of pages that can be created in a day - URL rewriting means you can have an almost infinite number of pages on any one site - Soft 404s and duplicate content can be hard to spot and can waste resources in gathering/removing them There's paid for resources like Majestic/Ahrefs/Moz that crawl the web to see who's linking to who and they all contain trillions of URLs. I think the most detrimental fact is that pages often disappear or change, I don't have a recent number but I'm fairly certain there's a 10-15% chance that any link you see this year, will be gone next year. "Link rot". Hard to build a DMOZ style directory on that scale with that problem. I don't think it is unmanageable, it just needs to be seen from different perspectives and managed by different groups of people.