16 ms·

While this is intellectually interesting, I'm troubled by the fact that the author seems not to have given the slightest thought that he's breaking the site's T

While this is intellectually interesting, I'm troubled by the fact that the author seems not to have given the slightest thought that he's breaking the site's T&C, or of how much this abuse costs the service. I'm particularly sensitive to this because I'm constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.

vbezhenar 7y ago
Another problem is that blocking tor nodes is very easy. And with abusing tor nodes as free proxies, it makes a huge disservice to the Tor network, because more websites will ban it and ordinary users won't be able to use it.
- beardog 7y ago
  This problem is being partly addressed with Privacy Pass. The addon lets you bypass recaptcha including cloudflare recaptcha minus one initial solve per session. Of course this doesn't account for all blocks and of course some malicious bots bypass recaptcha anyway using AI/speech recognition or human labor farms, but it will make the ecosystem better for everyone. (note: it is not a captcha solving service, its a collaborative project by cloudflare, google and others) https://privacypass.github.io/ https://privacypass.github.io/
bored_hacker 7y ago
I'm actually not so sure that there terms of services prohibits scraping for personal use (or scraping at all) but maybe my understanding is incorrect. I have read through them here https://www.zillow.com/corp/Terms.htm https://www.zillow.com/corp/Terms.htm and looked through there robots.txt here https://streeteasy.com/robots.txt https://streeteasy.com/robots.txt. This is most likely because they use Distil Networks and rely on them to block web scrapers. But you are right this is something of concern and I can add a section talking about the dangers of breaking a sites terms and conditions and how scraping must be done correctly even for personal use
- CWuestefeld 7y ago
  From that page: 5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services; And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms. Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
  - mirimir 7y ago
    In many jurisdictions, "BY USING THE SERVICES, YOU AGREE ..." language is legally meaningless.
    - allannienhuis 7y ago
      The parent suggested that it may not be legally binding, so not sure what you're pointing out. What about his main point of the ethical question - people are doing something with their service that they clearly don't want them to, because it hurts/costs them. Just because someone can get away with it doesn't make it right.
      - mirimir 7y ago
        If you put data online for anyone to use, people will use it as they see fit. That's just how it is. And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively. If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
        CWuestefeld 7y ago
        I'd imagine the site has some reason for not including that functionality. Maybe they don't have developer resources to implement it, but on the other hand, maybe for some reason the computational cost puts too much strain on their servers. It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
ChrisSD 7y ago
I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places. However, I do agree that people should care more about the ethics of web scrapping.
- ohithereyou 7y ago
  >I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places. Tell that to Aaron Swartz.
iamnotacrook 7y ago
"the author seems not to have given the slightest thought that he's breaking the site's T&C" LOL! Never read 'em, never going to, couldn't care less about any of them. If it was that important they'd make people create and account and sign in and limit access that way. Don't expect people to take all the blah blah blah seriously. It's just a web page.
- bobthepanda 7y ago
  And this is how you get the entire Internet locked behind login pages. Abusing free and open leads to people locking down free and open.
  - unityByFreedom 7y ago
    I'm not sure that's true. Free and open drives more page views and is a more successful marketing strategy where it makes sense.
- sergiotapia 7y ago
  Yeah, I totally agree with you, but also make sure you're not being an asshole with websites and hammer them to the point of bringing them down. Be respectful of their hardware _and_ take the data you need. No need to be burn everything down.
  - matz1 7y ago
    I certainly wouldn't want to burn it down as long as I still need the data, other than that I couldn't care less.
- cgag 7y ago
  "lol" was my first thought as well, and I suspect many peoples, despite the down votes.
  - jrumbut 7y ago
    For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that they contact the site's owners and ask to arrange a data use agreement. And a couple have really done it and been satisfied by the result which was cleaner data and continuous updates at a reasonable cost. Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!
aiyodev 7y ago
You could easily put your pricing info on a single page. You’re not paying for bandwidth. You’re paying for obscuring information. You’re paying for going against the purpose and spirit of the internet.
detaro 7y ago
And "a fair amount of"developer time" is cheaper than serving more traffic? I'd be curious about actual numbers here, but I understand that's not necessarily something you can share.
- bdcravens 7y ago
  I suspect most of that time isn't to reduce traffic costs, but to try to thwart their competitors
lyxsus1 7y ago
Unpopular opinion here. I think that it would be better for most if scraping prevention was illegal if there's no API that allows authorised application to perform every action accessible to authorised user. Just imagine, how much better UX would be if we could get rid of those small imaginary data silos.
jMyles 7y ago
If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them. Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.
- cookie_monsta 7y ago
  > If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind: - Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article) - Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
  - FDSGSG 7y ago
    > scraping competitor sites for pricing info etc is exactly as sleazy as it sounds So not even a tiny bit sleazy then?
  - chii 7y ago
    > While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
    - evilDagmar 7y ago
      It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
katzgrau 7y ago
Can the site sit behind a CDN like Cloudflare? Seems like that would solve any issues caused by bots - and furthermore, Cloudflare can captcha challenge suspected bot traffic
latchkey 7y ago
It sounds like offering a paid API is the perfect pivot for your business.
meritt 7y ago
> breaking the site's T&C T&C's have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&C's. [1] https://www.eff.org/deeplinks/2018/04/scraping-just-automated-access-and-everyone-does-it https://www.eff.org/deeplinks/2018/04/scraping-just-automate... [2] https://www.eff.org/deeplinks/2018/04/dc-court-accessing-public-information-not-computer-crime https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub... [3] https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-down-violating-websites-terms-service-not-crime https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
- jacquesm 7y ago
  The world is a lot larger than just the United States. Besides that, even publicly available information can be copyrighted. So the scraping itself may be legal, storing that information; using it and re-publishing it very likely is not.
  - luckylion 7y ago
    > The world is a lot larger than just the United States. Where would this be significantly different? For T&C to be valid, you'll have to agree to them before using the service. Since you generally can't know, much less agree to, the T&C of a website before using that website, you have a chicken & egg problem where the law tends to come down on "these T&C aren't binding" imho. You're right about the re-publishing though. Storing is likely a different issue, even browser-caching is storing; iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy.
    - aw3c2 7y ago
      In Germany there was a recent ruling that scraping, even for competing purposes, is fine. https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scraping-BGH-legt-schriftliche-Urteilsbegruendung-vor-2236236.html https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scr...
steventhedev 7y ago
Sounds like you need to make things easier rather than harder. They have a clear economic incentive to scrape your site and will continue to do so. Rather than spending your employees time on it, open it up as an API. If someone starts hitting it too hard, send them an email to back off. If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.
- ForHackernews 7y ago
  Charge your competitors for API tokens?
- viraptor 7y ago
  This approach is not very realistic. 1. API design and auth takes engineering time and potentially brings more traffic you need to pay for. 2. You don't have an email address to send "back off" to. Even if you had, no enforcement means no change. 3. Blocking / throttling takes much less time and money than creating a new endpoint.
hueving 7y ago
It sounds like your problem is your confusion about trying to prevent devices from accessing your public data over the Internet. Have you tried adding an rss feed or another API instead?
nullwasamistake 7y ago
Your site is public, what do you expect? It's not abuse to load public websites. T&C is meaningless for public sites. Equivalent to real life sign hanging in a shop "whoever reads this is bound to the following conditions". That's not how contracts work If you block the small guys from scraping you're just helping already huge monopolies tighten their grip on the content market. Even "nice" scrapers like Google blatantly steal your content, blocking everyone else just makes their monopoly worse. They steal pricing data for "Google Shopping" regularly. If it annoys you that much just build an API and charge tiny amount to use it.
drtyolmck 7y ago
There is nothing you can do to stop scraper bots. People will always find a way to scrape your content.
segmondy 7y ago
Get over it, don't want them to scrape it, protect it and don't make it world accessible. I'm troubled by the fact that you're on the side that wants to control "public information". If your information is valuable and other's want it, provide API/RSS or ways to get it without impacting your site. If you don't want other's to get it, put it behind a walled garden.
unnouinceput 7y ago
Oh, this has a very easy counter. Make the price on your site to be multiple info coming from separate requests. TOR will most likely do those requests from different IP's and when you detect this you can either not show the price or show a bad one.

vbezhenar 7y ago

Another problem is that blocking tor nodes is very easy. And with abusing tor nodes as free proxies, it makes a huge disservice to the Tor network, because more websites will ban it and ordinary users won't be able to use it.

beardog 7y ago
This problem is being partly addressed with Privacy Pass. The addon lets you bypass recaptcha including cloudflare recaptcha minus one initial solve per session. Of course this doesn't account for all blocks and of course some malicious bots bypass recaptcha anyway using AI/speech recognition or human labor farms, but it will make the ecosystem better for everyone. (note: it is not a captcha solving service, its a collaborative project by cloudflare, google and others) https://privacypass.github.io/ https://privacypass.github.io/

bored_hacker 7y ago

I'm actually not so sure that there terms of services prohibits scraping for personal use (or scraping at all) but maybe my understanding is incorrect. I have read through them here https://www.zillow.com/corp/Terms.htm https://www.zillow.com/corp/Terms.htm and looked through there robots.txt here https://streeteasy.com/robots.txt https://streeteasy.com/robots.txt. This is most likely because they use Distil Networks and rely on them to block web scrapers. But you are right this is something of concern and I can add a section talking about the dangers of breaking a sites terms and conditions and how scraping must be done correctly even for personal use

CWuestefeld 7y ago
From that page: 5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services; And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms. Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
- mirimir 7y ago
  In many jurisdictions, "BY USING THE SERVICES, YOU AGREE ..." language is legally meaningless.
  - allannienhuis 7y ago
    The parent suggested that it may not be legally binding, so not sure what you're pointing out. What about his main point of the ethical question - people are doing something with their service that they clearly don't want them to, because it hurts/costs them. Just because someone can get away with it doesn't make it right.
    - mirimir 7y ago
      If you put data online for anyone to use, people will use it as they see fit. That's just how it is. And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively. If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
      - CWuestefeld 7y ago
        I'd imagine the site has some reason for not including that functionality. Maybe they don't have developer resources to implement it, but on the other hand, maybe for some reason the computational cost puts too much strain on their servers. It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.

ChrisSD 7y ago

I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places. However, I do agree that people should care more about the ethics of web scrapping.

ohithereyou 7y ago
>I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places. Tell that to Aaron Swartz.

iamnotacrook 7y ago

"the author seems not to have given the slightest thought that he's breaking the site's T&C" LOL! Never read 'em, never going to, couldn't care less about any of them. If it was that important they'd make people create and account and sign in and limit access that way. Don't expect people to take all the blah blah blah seriously. It's just a web page.

bobthepanda 7y ago
And this is how you get the entire Internet locked behind login pages. Abusing free and open leads to people locking down free and open.
- unityByFreedom 7y ago
  I'm not sure that's true. Free and open drives more page views and is a more successful marketing strategy where it makes sense.
sergiotapia 7y ago
Yeah, I totally agree with you, but also make sure you're not being an asshole with websites and hammer them to the point of bringing them down. Be respectful of their hardware _and_ take the data you need. No need to be burn everything down.
- matz1 7y ago
  I certainly wouldn't want to burn it down as long as I still need the data, other than that I couldn't care less.
cgag 7y ago
"lol" was my first thought as well, and I suspect many peoples, despite the down votes.
- jrumbut 7y ago
  For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that they contact the site's owners and ask to arrange a data use agreement. And a couple have really done it and been satisfied by the result which was cleaner data and continuous updates at a reasonable cost. Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!

aiyodev 7y ago

You could easily put your pricing info on a single page. You’re not paying for bandwidth. You’re paying for obscuring information. You’re paying for going against the purpose and spirit of the internet.

detaro 7y ago

And "a fair amount of"developer time" is cheaper than serving more traffic? I'd be curious about actual numbers here, but I understand that's not necessarily something you can share.

bdcravens 7y ago
I suspect most of that time isn't to reduce traffic costs, but to try to thwart their competitors

lyxsus1 7y ago

Unpopular opinion here. I think that it would be better for most if scraping prevention was illegal if there's no API that allows authorised application to perform every action accessible to authorised user. Just imagine, how much better UX would be if we could get rid of those small imaginary data silos.

jMyles 7y ago

If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them. Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.

cookie_monsta 7y ago
> If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want. Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind: - Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article) - Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
- FDSGSG 7y ago
  > scraping competitor sites for pricing info etc is exactly as sleazy as it sounds So not even a tiny bit sleazy then?
- chii 7y ago
  > While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
  - evilDagmar 7y ago
    It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.

katzgrau 7y ago

Can the site sit behind a CDN like Cloudflare? Seems like that would solve any issues caused by bots - and furthermore, Cloudflare can captcha challenge suspected bot traffic

latchkey 7y ago

It sounds like offering a paid API is the perfect pivot for your business.

meritt 7y ago

> breaking the site's T&C T&C's have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&C's. [1] https://www.eff.org/deeplinks/2018/04/scraping-just-automated-access-and-everyone-does-it https://www.eff.org/deeplinks/2018/04/scraping-just-automate... [2] https://www.eff.org/deeplinks/2018/04/dc-court-accessing-public-information-not-computer-crime https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub... [3] https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-down-violating-websites-terms-service-not-crime https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...

jacquesm 7y ago
The world is a lot larger than just the United States. Besides that, even publicly available information can be copyrighted. So the scraping itself may be legal, storing that information; using it and re-publishing it very likely is not.
- luckylion 7y ago
  > The world is a lot larger than just the United States. Where would this be significantly different? For T&C to be valid, you'll have to agree to them before using the service. Since you generally can't know, much less agree to, the T&C of a website before using that website, you have a chicken & egg problem where the law tends to come down on "these T&C aren't binding" imho. You're right about the re-publishing though. Storing is likely a different issue, even browser-caching is storing; iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy.
  - aw3c2 7y ago
    In Germany there was a recent ruling that scraping, even for competing purposes, is fine. https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scraping-BGH-legt-schriftliche-Urteilsbegruendung-vor-2236236.html https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scr...

steventhedev 7y ago

Sounds like you need to make things easier rather than harder. They have a clear economic incentive to scrape your site and will continue to do so. Rather than spending your employees time on it, open it up as an API. If someone starts hitting it too hard, send them an email to back off. If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.

ForHackernews 7y ago
Charge your competitors for API tokens?
viraptor 7y ago
This approach is not very realistic. 1. API design and auth takes engineering time and potentially brings more traffic you need to pay for. 2. You don't have an email address to send "back off" to. Even if you had, no enforcement means no change. 3. Blocking / throttling takes much less time and money than creating a new endpoint.

hueving 7y ago

It sounds like your problem is your confusion about trying to prevent devices from accessing your public data over the Internet. Have you tried adding an rss feed or another API instead?

nullwasamistake 7y ago

Your site is public, what do you expect? It's not abuse to load public websites. T&C is meaningless for public sites. Equivalent to real life sign hanging in a shop "whoever reads this is bound to the following conditions". That's not how contracts work If you block the small guys from scraping you're just helping already huge monopolies tighten their grip on the content market. Even "nice" scrapers like Google blatantly steal your content, blocking everyone else just makes their monopoly worse. They steal pricing data for "Google Shopping" regularly. If it annoys you that much just build an API and charge tiny amount to use it.

drtyolmck 7y ago

There is nothing you can do to stop scraper bots. People will always find a way to scrape your content.

segmondy 7y ago

Get over it, don't want them to scrape it, protect it and don't make it world accessible. I'm troubled by the fact that you're on the side that wants to control "public information". If your information is valuable and other's want it, provide API/RSS or ways to get it without impacting your site. If you don't want other's to get it, put it behind a walled garden.

unnouinceput 7y ago

Oh, this has a very easy counter. Make the price on your site to be multiple info coming from separate requests. TOR will most likely do those requests from different IP's and when you detect this you can either not show the price or show a bad one.