5 ms·

I'm actually not so sure that there terms of services prohibits scraping for personal use (or scraping at all) but maybe my understanding is incorrect. I have r

I'm actually not so sure that there terms of services prohibits scraping for personal use (or scraping at all) but maybe my understanding is incorrect. I have read through them here https://www.zillow.com/corp/Terms.htm https://www.zillow.com/corp/Terms.htm and looked through there robots.txt here https://streeteasy.com/robots.txt https://streeteasy.com/robots.txt. This is most likely because they use Distil Networks and rely on them to block web scrapers. But you are right this is something of concern and I can add a section talking about the dangers of breaking a sites terms and conditions and how scraping must be done correctly even for personal use

CWuestefeld 7y ago
From that page: 5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services; And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms. Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
- mirimir 7y ago
  In many jurisdictions, "BY USING THE SERVICES, YOU AGREE ..." language is legally meaningless.
  - allannienhuis 7y ago
    The parent suggested that it may not be legally binding, so not sure what you're pointing out. What about his main point of the ethical question - people are doing something with their service that they clearly don't want them to, because it hurts/costs them. Just because someone can get away with it doesn't make it right.
    - mirimir 7y ago
      If you put data online for anyone to use, people will use it as they see fit. That's just how it is. And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively. If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
      - CWuestefeld 7y ago
        I'd imagine the site has some reason for not including that functionality. Maybe they don't have developer resources to implement it, but on the other hand, maybe for some reason the computational cost puts too much strain on their servers. It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
        mirimir 7y ago
        See this: https://news.ycombinator.com/item?id=20430944 https://news.ycombinator.com/item?id=20430944 It's what I'd say, except said more clearly.
        allannienhuis 7y ago
        That describes things you want (and I agree that an api is a fairly reasonable thing to want), but they are not things you have the right to have just because you want them. It may be naive to assume people will honor the desires of a website owner, but that doesn't change the moral argument - you shouldn't try to convince yourself that you're in the right just because its difficult for the website owner to enforce their wishes for how people use their system, or because everyone does it - "that's just the way it is" doesn't make it right. I don't think we're talking about discrimination or equal access or even right to back up issues which might apply some moral right to access a site in ways not desired by a service owner - the vast majority of cases for scraping are where the scraper goes on to use that information in ways that are for the commercial benefit of the scraper and usually to the detriment of the site owner. The arguments about data access to publicly funded information has more merit but I don't see them applying in any way to private services. Am I missing some other fundamental right that applies here?
- borge 7y ago
  How valuable is that legally? If my bot is scraping that automatically, have I then agreed to that?
- krageon 7y ago
  > ethical qualms What ethical qualms are those? The fact that someone else tells you that by using something you agree with a bunch of agreements that they made up on the spot?

CWuestefeld 7y ago

From that page: 5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services; And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms. Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.

mirimir 7y ago
In many jurisdictions, "BY USING THE SERVICES, YOU AGREE ..." language is legally meaningless.
- allannienhuis 7y ago
  The parent suggested that it may not be legally binding, so not sure what you're pointing out. What about his main point of the ethical question - people are doing something with their service that they clearly don't want them to, because it hurts/costs them. Just because someone can get away with it doesn't make it right.
  - mirimir 7y ago
    If you put data online for anyone to use, people will use it as they see fit. That's just how it is. And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively. If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
    - CWuestefeld 7y ago
      I'd imagine the site has some reason for not including that functionality. Maybe they don't have developer resources to implement it, but on the other hand, maybe for some reason the computational cost puts too much strain on their servers. It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
      - mirimir 7y ago
        See this: https://news.ycombinator.com/item?id=20430944 https://news.ycombinator.com/item?id=20430944 It's what I'd say, except said more clearly.
        allannienhuis 7y ago
        That describes things you want (and I agree that an api is a fairly reasonable thing to want), but they are not things you have the right to have just because you want them. It may be naive to assume people will honor the desires of a website owner, but that doesn't change the moral argument - you shouldn't try to convince yourself that you're in the right just because its difficult for the website owner to enforce their wishes for how people use their system, or because everyone does it - "that's just the way it is" doesn't make it right. I don't think we're talking about discrimination or equal access or even right to back up issues which might apply some moral right to access a site in ways not desired by a service owner - the vast majority of cases for scraping are where the scraper goes on to use that information in ways that are for the commercial benefit of the scraper and usually to the detriment of the site owner. The arguments about data access to publicly funded information has more merit but I don't see them applying in any way to private services. Am I missing some other fundamental right that applies here?
borge 7y ago
How valuable is that legally? If my bot is scraping that automatically, have I then agreed to that?
krageon 7y ago
> ethical qualms What ethical qualms are those? The fact that someone else tells you that by using something you agree with a bunch of agreements that they made up on the spot?