2 ms·
speaking as a sysadmin: random agressive crawlers are annoying and usually get blocked/filtered out. I think the best solution would be some sort of selfcrawler
by taskforcegemini 3y ago
speaking as a sysadmin: random agressive crawlers are annoying and usually get blocked/filtered out. I think the best solution would be some sort of selfcrawler which uploads your site to a search provider. uploaders have to pay and bad uploads/seo-spam gets punished somehow (thus seo-abuse becomes expensive)
- mrkramer 3y ago>I think the best solution would be some sort of selfcrawler which uploads your site to a search provider. uploaders have to pay and bad uploads/seo-spam gets punished somehow (thus seo-abuse becomes expensive) I had the same idea; the website owners and the webmasters can crawl their own sites and work together with search engines in order to get revenue share from ads (sort of licensing deal, similar to YouTube partner revenue share program). But the problem is that the webmasters can not efficiently crawl their own sites because they do not have the money to do it every day like Google. Maybe we can have like 1 month old index for a search engine, sort of outdated and not that practical but still practical for some casual users. This idea would be better for a commercial web archiving service where website owners and webmasters crawl/archive their own website and license it to the web archiving service.