3 ms·

I came here to say that too. Scraping has a great many legitimate uses. Search engines, scientific research, trying to use publicly available data that doesn't

by throwaway2016a 9y ago

I came here to say that too. Scraping has a great many legitimate uses. Search engines, scientific research, trying to use publicly available data that doesn't have an API. I've had to scrape government websites quite frequently because they often make public information hard to read by other means. That last one is an interesting one. I think one of the most effective way to deter a scraper might be to just provide an API! Now if you were using the scraped data to republish (copyright infringement) or use it to gain a competitive advantage (re-pricing in eCommerce comes to mind) that is a different story.

xg15 9y ago
This is actually an interesting point. If you implemented an effective scraper-detection API, you'd run risk of locking out search engines too. (Though I guess the real-life solution would be both simple and depressing: make an exception for googlebot and don't care about anyone else)
- andai 9y ago
  Yeah I've seen a lot of sites that explicitly state that, as well as in their robots.txt All robots forbidden, except googlebot
  - chrisweekly 9y ago
    ... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".
  - kuschku 9y ago
    Which is why some search engines simply ended up following only the directives given to googlebot, and ignoring the rest.
  - josteink 9y ago
    > All robots forbidden, except googlebot After which people keep claiming google search is better, not just given special treatment.
chii 9y ago
> re-pricing in eCommerce comes to mind what's re-pricing?
- throwaway2016a 9y ago
  Re-pricing is the practice of scraping your competitors and pricing your product just a little lower so that in price comparison searches you always show up at top.

xg15 9y ago

This is actually an interesting point. If you implemented an effective scraper-detection API, you'd run risk of locking out search engines too. (Though I guess the real-life solution would be both simple and depressing: make an exception for googlebot and don't care about anyone else)

andai 9y ago
Yeah I've seen a lot of sites that explicitly state that, as well as in their robots.txt All robots forbidden, except googlebot
- chrisweekly 9y ago
  ... which rule would be obeyed only by legitimate, robots.txt-honoring crawlers. This reminds me of the anti-piracy messages shown (solely) to viewers of legally-purchased media. Similar "logic", similar (counter-productive) "effectiveness".
- kuschku 9y ago
  Which is why some search engines simply ended up following only the directives given to googlebot, and ignoring the rest.
- josteink 9y ago
  > All robots forbidden, except googlebot After which people keep claiming google search is better, not just given special treatment.

chii 9y ago

> re-pricing in eCommerce comes to mind what's re-pricing?

throwaway2016a 9y ago
Re-pricing is the practice of scraping your competitors and pricing your product just a little lower so that in price comparison searches you always show up at top.