In 1998, the Web was incomparably smaller. Google could put their whole infra into a dozen boxes, and then grow along with the Web, gradually. Modern competitor

In 1998, the Web was incomparably smaller. Google could put their whole infra into a dozen boxes, and then grow along with the Web, gradually. Modern competitors don't have this advantage. By now, crawling and indexing is a herculean task, and also quite expensive, due to the sheer size. There is Common Crawl [1]; at 400 TiB it is huge, but at 60 days refresh interval it's far from being very comprehensive or very fresh. Good for research, but likely not good for a commercial search engine. [1]: https://commoncrawl.org/ https://commoncrawl.org/