4 ms·
Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.tx
by satyrnein 7y ago
Imagine if search engines had to "reach out to each [site owner] and figure out a business arrangement with them." The world decided that opt out via robots.txt was a better approach.
If the municipality wants to get the information out, this could be a win-win, just like search engines were. Do check for robots.text, though!
- samcal 7y agoWell, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.
- vageli 7y ago> Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world. Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to every site they wanted to crawl? How many companies out there complain about being scraped by Google? How many companies benefit from search-driven traffic?
- gnud 7y agoThe alternative would have been opt-in instead of opt-out. Everything excluded by default, except what robots.txt allows you to index. Naturally, Google didn't want that.
- klenwell 7y agoI would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.
- greglindahl 7y agoThat's not true in the actual web, however. The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting.
- floatingatoll 7y agoWe found at one job that approximate one quarter of well-known search engines blatantly use robots.txt noindex declarations as a list of URLs to index, and one openly mocked us for asking them to stop. Voluntary honor systems don’t work, because there’s no way to compel non-compliers to stop other than standard “anti-attacker arms race” approaches, such as the obstacle described at the head of this thread.
- jdc 7y agoIt sounds like scraping is a big problem for you guys. What kind of outfit is it, if you don't mind me asking?
- floatingatoll 7y agoDrop me an email and I’m happy to describe further.