6 ms·

Do you know if scraping this data violates their ToS?

by akudha 6y ago

Scoundreller 6y ago
Dunno about Atlassian and Shopify, but I can’t imagine Google launching a case against someone for scraping them. I guess robots.txt is all that really matters. Here’s Play Store’s: https://play.google.com/robots.txt https://play.google.com/robots.txt
- minnehaha 6y ago
  agreed, provided the crawler complies with robots.txt
- akudha 6y ago
  This data is quite valuable and services like this aren't competing with these app stores. If anything, they are probably inviting more competition to the app stores resulting in better apps. There is no reason for these companies to go after some small guy scraping the app store data. But who knows - Craigslist went after one man shops. These companies might too.
bahmboo 6y ago
Google's entire business is scraping websites
- catillac 6y ago
  Maybe that’s a reason to not feel bad if it violates the TOS but there are real consequences to doing so so perhaps your response isn’t that helpful.
  - intricatedetail 6y ago
    Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
    - namdnay 6y ago
      Websites can choose whether to be scraped or not, and google respects that choice
      - hire_charts 6y ago
        It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
        ben_w 6y ago
        You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
        DangitBobby 6y ago
        No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
      - k4rli 6y ago
        To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
      - intricatedetail 6y ago
        By creating an account and using their webmaster tools giving up even more of your data?
        mitchdoogle 6y ago
        No you just have to put a noindex header on requests to your website
  - reader_mode 6y ago
    >but there are real consequences to doing so What consequences ? Breaking ToS can get your account suspended if you even need an account for doing this, not sure what else breaking the TOS really implies. I don't know if Google can claim ownership of this data as it's customer generated and publically available.
  - AznHisoka 6y ago
    If anyone really wanted to stop you, they would put up an anti scraping mechanism like Imperva and not even bother with legal action.
    - luxuryballs 6y ago
      But mostly people are old or don’t care about solutions when they are too busy so it’s easier just to throw money at a lawyer and cause you trouble.
- fooey 6y ago
  Sure, but they don't scrape anyone who opts out
visox 6y ago
hm not sure how/if they will react if this blows up but it wont stop me for now.
davidgh 6y ago
In the US, there was a ruling a couple of years ago that gives some legal clarity to scraping data made publicly available (in favor of the scrapers) in the hiQ vs. LinkedIn case. You can read more about it here https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
chuckgreenman 6y ago
Even if it is, it doesn't matter. The US Court of Appeals created precedent that allows for the scraping and aggregation of publicly published information with that LinkedIn case.
- geuis 6y ago
  Got a link for this? Could open up some possibilities.
  - chuckgreenman 6y ago
    Here's the actual case: https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2019-09-09.html https://law.justia.com/cases/federal/appellate-courts/ca9/17... There is an attached PDF with the full decision, Pages 27, 28 on reference some related cases that informed the decision.
- DaiPlusPlus 6y ago
  > publicly published information Is Apple’s iOS App Store “public” though? Ostensibly it’s only open to Apple customers with an iOS device: accessing App Store pages in a web-browser just redirects you to the App Store.
  - ElFitz 6y ago
    > accessing App Store pages in a web-browser just redirects you to the App Store. Not if you are on an non-Apple device. Or access an iOS App Store page from a Mac. The pages show just fine.

Do you know if scraping this data violates their ToS?

by akudha 6y ago

Do you know if scraping this data violates their ToS?

Dunno about Atlassian and Shopify, but I can’t imagine Google launching a case against someone for scraping them. I guess robots.txt is all that really matters. Here’s Play Store’s: https://play.google.com/robots.txt https://play.google.com/robots.txt

minnehaha 6y ago
agreed, provided the crawler complies with robots.txt
akudha 6y ago
This data is quite valuable and services like this aren't competing with these app stores. If anything, they are probably inviting more competition to the app stores resulting in better apps. There is no reason for these companies to go after some small guy scraping the app store data. But who knows - Craigslist went after one man shops. These companies might too.

bahmboo 6y ago

Google's entire business is scraping websites

catillac 6y ago
Maybe that’s a reason to not feel bad if it violates the TOS but there are real consequences to doing so so perhaps your response isn’t that helpful.
- intricatedetail 6y ago
  Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
  - namdnay 6y ago
    Websites can choose whether to be scraped or not, and google respects that choice
    - hire_charts 6y ago
      It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
      - ben_w 6y ago
        You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
        DangitBobby 6y ago
        No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
    - k4rli 6y ago
      To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
    - intricatedetail 6y ago
      By creating an account and using their webmaster tools giving up even more of your data?
      - mitchdoogle 6y ago
        No you just have to put a noindex header on requests to your website
- reader_mode 6y ago
  >but there are real consequences to doing so What consequences ? Breaking ToS can get your account suspended if you even need an account for doing this, not sure what else breaking the TOS really implies. I don't know if Google can claim ownership of this data as it's customer generated and publically available.
- AznHisoka 6y ago
  If anyone really wanted to stop you, they would put up an anti scraping mechanism like Imperva and not even bother with legal action.
  - luxuryballs 6y ago
    But mostly people are old or don’t care about solutions when they are too busy so it’s easier just to throw money at a lawyer and cause you trouble.
fooey 6y ago
Sure, but they don't scrape anyone who opts out

visox 6y ago

hm not sure how/if they will react if this blows up but it wont stop me for now.

davidgh 6y ago

In the US, there was a ruling a couple of years ago that gives some legal clarity to scraping data made publicly available (in favor of the scrapers) in the hiQ vs. LinkedIn case. You can read more about it here https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

chuckgreenman 6y ago

Even if it is, it doesn't matter. The US Court of Appeals created precedent that allows for the scraping and aggregation of publicly published information with that LinkedIn case.

geuis 6y ago
Got a link for this? Could open up some possibilities.
- chuckgreenman 6y ago
  Here's the actual case: https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2019-09-09.html https://law.justia.com/cases/federal/appellate-courts/ca9/17... There is an attached PDF with the full decision, Pages 27, 28 on reference some related cases that informed the decision.
DaiPlusPlus 6y ago
> publicly published information Is Apple’s iOS App Store “public” though? Ostensibly it’s only open to Apple customers with an iOS device: accessing App Store pages in a web-browser just redirects you to the App Store.
- ElFitz 6y ago
  > accessing App Store pages in a web-browser just redirects you to the App Store. Not if you are on an non-Apple device. Or access an iOS App Store page from a Mac. The pages show just fine.