4 ms·

If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. I don't think that the inter

by jMyles 7y ago

If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them. Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.

cookie_monsta 7y ago
> If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind: - Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article) - Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
- FDSGSG 7y ago
  > scraping competitor sites for pricing info etc is exactly as sleazy as it sounds So not even a tiny bit sleazy then?
- chii 7y ago
  > While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
  - evilDagmar 7y ago
    It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
    - bydl0coder 7y ago
      Large eCommerce sites are constantly scraped by competitors, market research companies and even vendors (to ensure sellers honor MSRP). Scraping activity is usually throttled at reasonable levels.
    - FDSGSG 7y ago
      Could you tell us more about your "typical" experiences?
mrweasel 7y ago
Maybe the information being scrapped isn’t something you’re allowed to share via an API. Maybe you paid as licens fee to use it yourself. We offered an API for just two customers, with images and product information for video games. One of them preferred scrapping our page anyway, so we throttled them.
evilDagmar 7y ago
Well, you can spend a little time putting some anti-bot measures into place, or you can spend money paying a larger hosting bill for all the wasted CPU time, or you can spend some time explaining to legitimate users why the site fell over under the unwanted load. The person who started this reply thread repeated part of it twice for good reason. Bots, unchecked, can and do take up a majority of the resources on small to medium sites. I'd have been much happier with the article if the author had taken a moment to implement a bit of throttling on his end to at least go easy on the remote server.

by jMyles 7y ago

cookie_monsta 7y ago

> If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind: - Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article) - Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!

FDSGSG 7y ago
> scraping competitor sites for pricing info etc is exactly as sleazy as it sounds So not even a tiny bit sleazy then?
chii 7y ago
> While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
- evilDagmar 7y ago
  It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
  - bydl0coder 7y ago
    Large eCommerce sites are constantly scraped by competitors, market research companies and even vendors (to ensure sellers honor MSRP). Scraping activity is usually throttled at reasonable levels.
  - FDSGSG 7y ago
    Could you tell us more about your "typical" experiences?

mrweasel 7y ago

Maybe the information being scrapped isn’t something you’re allowed to share via an API. Maybe you paid as licens fee to use it yourself. We offered an API for just two customers, with images and product information for video games. One of them preferred scrapping our page anyway, so we throttled them.

evilDagmar 7y ago

Well, you can spend a little time putting some anti-bot measures into place, or you can spend money paying a larger hosting bill for all the wasted CPU time, or you can spend some time explaining to legitimate users why the site fell over under the unwanted load. The person who started this reply thread repeated part of it twice for good reason. Bots, unchecked, can and do take up a majority of the resources on small to medium sites. I'd have been much happier with the article if the author had taken a moment to implement a bit of throttling on his end to at least go easy on the remote server.