3 ms·
Instead of having many bots do inefficient crawling, web sites should publish their own index. Intermediate parties can combine indexes of the sites. Sites that
by oever 6y ago
Instead of having many bots do inefficient crawling, web sites should publish their own index. Intermediate parties can combine indexes of the sites. Sites that do not provide indexes get less visitors.
- adrianstoll 6y agoThis is the idea behind sitemaps which have existed forever.
- oever 6y agoSitemaps are lists of urls on a site. They are not a text index.
- rakoo 6y agoThat's the best way to fill your search engine with spam. There needs to be a third-party that verifies that the site-provided index is inline with the actual content. At which point said third-party can be a crawler.