5 ms·
Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like th
by yaur 12y ago
Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.
- userbinator 12y agoMaybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...
- yhager 12y agoNot completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /
- yaur 12y agoI think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.
- thefreeman 12y agoBy a single search engine which probably provides the vast majority of their traffic. While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.
- yaur 12y agoNeither of the two parties mentioned is likely to resell the data directly, both are likely to to create a derivative work which they will exploit commercially as will Google. What part of the ToS make one OK while and the other forbidden? How does this work when a smart scraper can just pull from the Google or archive.org cache?
- muyuu 12y agoNot sure it's a good idea to block all search engines but Google.
- yhager 12y agoProbably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allow some bots and not others if you like. Of course, forging your user agent, disobeying robots.txt or scraping after you were told not to, is wrong ethically.