7 ms·

Google's entire business is scraping websites

catillac 5y ago
Maybe that’s a reason to not feel bad if it violates the TOS but there are real consequences to doing so so perhaps your response isn’t that helpful.
- intricatedetail 5y ago
  Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
  - namdnay 5y ago
    Websites can choose whether to be scraped or not, and google respects that choice
    - hire_charts 5y ago
      It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
      - ben_w 5y ago
        You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
        DangitBobby 5y ago
        No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
    - k4rli 5y ago
      To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
    - intricatedetail 5y ago
      By creating an account and using their webmaster tools giving up even more of your data?
      - mitchdoogle 5y ago
        No you just have to put a noindex header on requests to your website
        intricatedetail 5y ago
        So they still can use up my resources?
        Google234 5y ago
        How can they know without looking? Again, you could also use the webmasters tools to block it...
        uglygoblin 5y ago
        Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.
        Google234 5y ago
        How dare they!
        account42 5y ago
        > robots.txt can accomplish the same thing Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.
- reader_mode 5y ago
  >but there are real consequences to doing so What consequences ? Breaking ToS can get your account suspended if you even need an account for doing this, not sure what else breaking the TOS really implies. I don't know if Google can claim ownership of this data as it's customer generated and publically available.
- AznHisoka 5y ago
  If anyone really wanted to stop you, they would put up an anti scraping mechanism like Imperva and not even bother with legal action.
  - luxuryballs 5y ago
    But mostly people are old or don’t care about solutions when they are too busy so it’s easier just to throw money at a lawyer and cause you trouble.
fooey 5y ago
Sure, but they don't scrape anyone who opts out

Maybe that’s a reason to not feel bad if it violates the TOS but there are real consequences to doing so so perhaps your response isn’t that helpful.

intricatedetail 5y ago
Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
- namdnay 5y ago
  Websites can choose whether to be scraped or not, and google respects that choice
  - hire_charts 5y ago
    It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
    - ben_w 5y ago
      You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
      - DangitBobby 5y ago
        No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
  - k4rli 5y ago
    To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
  - intricatedetail 5y ago
    By creating an account and using their webmaster tools giving up even more of your data?
    - mitchdoogle 5y ago
      No you just have to put a noindex header on requests to your website
      - intricatedetail 5y ago
        So they still can use up my resources?
        Google234 5y ago
        How can they know without looking? Again, you could also use the webmasters tools to block it...
        uglygoblin 5y ago
        Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.
        Google234 5y ago
        How dare they!
        account42 5y ago
        > robots.txt can accomplish the same thing Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.
reader_mode 5y ago
>but there are real consequences to doing so What consequences ? Breaking ToS can get your account suspended if you even need an account for doing this, not sure what else breaking the TOS really implies. I don't know if Google can claim ownership of this data as it's customer generated and publically available.
AznHisoka 5y ago
If anyone really wanted to stop you, they would put up an anti scraping mechanism like Imperva and not even bother with legal action.
- luxuryballs 5y ago
  But mostly people are old or don’t care about solutions when they are too busy so it’s easier just to throw money at a lawyer and cause you trouble.

fooey 5y ago

Sure, but they don't scrape anyone who opts out