3 ms·
Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, tha
by samcal 7y ago
Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.
- vageli 7y ago> Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world. Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to every site they wanted to crawl? How many companies out there complain about being scraped by Google? How many companies benefit from search-driven traffic?
- gnud 7y agoThe alternative would have been opt-in instead of opt-out. Everything excluded by default, except what robots.txt allows you to index. Naturally, Google didn't want that.