Distributed search is great to make searching less dependent on a few big players. But if many more bots start crawling your site, the load and waste of energy

Distributed search is great to make searching less dependent on a few big players. But if many more bots start crawling your site, the load and waste of energy can get very high. Sitemaps help a bit, but site needs to be crawled completely initially and changed pages need to be retrieved. It'd be more efficient if there were a standard by which websites could create and publish their own index. Web search engines could collate the pages and add page ranks. The correctness of the index can be tested by randomly sampling a few pages with a bias to the highest rated pages.