7 ms·

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

by tangue 10y ago

sjwright 10y ago
As the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest. If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic. Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.
- bigtunacan 10y ago
  As the user of large websites I don't care. I'm not going to read the TOS and I will continue to scrape what I like since it makes my life more convenient. Like OP when blocked I'll just drive my scraping through a web browser which is the same as I've done for years on various sites that never provided APIs.
  - louprado 10y ago
    "As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.
    - lerpa 10y ago
      If you don't want your data public, then don't make it public in the first place. That's a good rule of thumb.
      - vidarh 10y ago
        Putting it behind a signup page with terms that don't allow sharing is not "making it public". And while in the US that may "just" be treated as unauthorized access, in the EU, if you make the data public it's also a violation of the Data Protection Directive, putting you at risk of prosecution in every EU country from which you have included data. You may be right from a risk minimisation perspective. But for a lot of data the risk in the case of exposure is low enough that it is a totally valid risk management strategy to assume that legal protections will be a sufficient deterrent to prevent enough of the most blatant abuses.
        kuschku 10y ago
        Eh, not really. The Data Protection Directive doesn’t even apply here – if the first party (OKCupid) made it available to a third party (the scraper), then the first party can be held in violation, but not the third party.
        Silhouette 10y ago
        If you have control of personally identifiable data, it's likely that at least some of the EU data protection rules will apply to you regardless of how you got it.
        tripzilch 10y ago
        Yes but as you say, they apply regardless. More specifically, they apply to data that you have (and are storing), not the act of obtaining it. As a private individual it's not hard to comply either, for private use. If you publish it, it becomes a different story, because it's PII. And, as soon as it's in possession of a company, they need to comply with more rules about securely storing it, etc. (this isn't enforced very well, though). Private individuals can't be held to that because there's (in theory) no legal way to check it.
- deleted 10y ago
  [deleted]
_c_ 10y ago
Worse is that Google tries to stop scraping. It's like they don't want anyone to see past the first page of results. They could scrape your website and then they prevent you form scraping your own data back. The whole process is silly; it reflects the duct tape and chicken wire nature of the www. No one should have to "scrape" or "crawl". Data should be put into a open universal format (no tags) and submitted when necessary (rsynced) to a public access archive, mirrored around the world. This to bridge the gap until we reach a more content addressable system (cf. location based). Clients (text readers, media players, whatever) can download and transform the universally formatted data into markup, binary, etc. -- whatever they wish, but all the design creativity and complexity of "web pages" or "web apps" can be handled at the network edge, client-side. "Crawling" should not be necessary. No one should have to store HTML tags and other window dressing for data. Dream on.
- NamTaf 10y ago
  That's the antithesis of the world wide web because you've just centralised data storage, which makes someone 'own' the www.
  - _c_ 10y ago
    I do not understand your argument. To give an example, there is a lot of free open source software mirrored all over the internet, mostly on ftp servers, but also on http, rsync, etc. If you use Linux or BSD you probably are using some of this software. If you use the www, then you are probably accessing computers that use this software. If you drive a new Mercedes you are probably using some of this software. There are a lot of copies of this code in a lot of places. Is that centralized? Does anyone hosting a mirror ("repository") "own" the software? Is it the same person or entity hosting every mirror? Compare Google's copies of everyone else's data, also replicated in a lot of places around the world. Who "owns" this data?
Analemma_ 10y ago
Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.
- tangue 10y ago
  Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.
  - collinmanderson 10y ago
    Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)
    - lerpa 10y ago
      Exactly, market scrappers as search engines. And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively. The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case. I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.
codeddesign 10y ago
Double standard? The difference is that Google Bot is built on being unobtrusive. I can easily built a scraper that will quickly ddos a site. Linkedin for example...if they allow 10,000 people to send 100 scraping requests per second everyday then that is stolen bandwidth that Linkedin has to pay for and the scrapers get free data. The difference is that Google has standards in which site's unusually benefit from, not to mention that they allow for you to disallow their bot. It just doesn't work the same way with some random developer building a scraper.
- encoderer 10y ago
  I agree that Googlebot is well behaved. When it detects your site is slowing down, it will back itself off. Unfortunately, this is often to your detriment. In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.
  - Guest98123 10y ago
    Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.
    - encoderer 10y ago
      That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.
      - Guest98123 10y ago
        I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.
        10y ago
desireco42 10y ago
Google and others, with legitimate reasons obey robots.txt

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

by tangue 10y ago

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

sjwright 10y ago

As the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest. If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic. Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.

bigtunacan 10y ago
As the user of large websites I don't care. I'm not going to read the TOS and I will continue to scrape what I like since it makes my life more convenient. Like OP when blocked I'll just drive my scraping through a web browser which is the same as I've done for years on various sites that never provided APIs.
- louprado 10y ago
  "As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.
  - lerpa 10y ago
    If you don't want your data public, then don't make it public in the first place. That's a good rule of thumb.
    - vidarh 10y ago
      Putting it behind a signup page with terms that don't allow sharing is not "making it public". And while in the US that may "just" be treated as unauthorized access, in the EU, if you make the data public it's also a violation of the Data Protection Directive, putting you at risk of prosecution in every EU country from which you have included data. You may be right from a risk minimisation perspective. But for a lot of data the risk in the case of exposure is low enough that it is a totally valid risk management strategy to assume that legal protections will be a sufficient deterrent to prevent enough of the most blatant abuses.
      - kuschku 10y ago
        Eh, not really. The Data Protection Directive doesn’t even apply here – if the first party (OKCupid) made it available to a third party (the scraper), then the first party can be held in violation, but not the third party.
        Silhouette 10y ago
        If you have control of personally identifiable data, it's likely that at least some of the EU data protection rules will apply to you regardless of how you got it.
        tripzilch 10y ago
        Yes but as you say, they apply regardless. More specifically, they apply to data that you have (and are storing), not the act of obtaining it. As a private individual it's not hard to comply either, for private use. If you publish it, it becomes a different story, because it's PII. And, as soon as it's in possession of a company, they need to comply with more rules about securely storing it, etc. (this isn't enforced very well, though). Private individuals can't be held to that because there's (in theory) no legal way to check it.
deleted 10y ago
[deleted]

_c_ 10y ago

Worse is that Google tries to stop scraping. It's like they don't want anyone to see past the first page of results. They could scrape your website and then they prevent you form scraping your own data back. The whole process is silly; it reflects the duct tape and chicken wire nature of the www. No one should have to "scrape" or "crawl". Data should be put into a open universal format (no tags) and submitted when necessary (rsynced) to a public access archive, mirrored around the world. This to bridge the gap until we reach a more content addressable system (cf. location based). Clients (text readers, media players, whatever) can download and transform the universally formatted data into markup, binary, etc. -- whatever they wish, but all the design creativity and complexity of "web pages" or "web apps" can be handled at the network edge, client-side. "Crawling" should not be necessary. No one should have to store HTML tags and other window dressing for data. Dream on.

NamTaf 10y ago
That's the antithesis of the world wide web because you've just centralised data storage, which makes someone 'own' the www.
- _c_ 10y ago
  I do not understand your argument. To give an example, there is a lot of free open source software mirrored all over the internet, mostly on ftp servers, but also on http, rsync, etc. If you use Linux or BSD you probably are using some of this software. If you use the www, then you are probably accessing computers that use this software. If you drive a new Mercedes you are probably using some of this software. There are a lot of copies of this code in a lot of places. Is that centralized? Does anyone hosting a mirror ("repository") "own" the software? Is it the same person or entity hosting every mirror? Compare Google's copies of everyone else's data, also replicated in a lot of places around the world. Who "owns" this data?

Analemma_ 10y ago

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

tangue 10y ago
Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.
- collinmanderson 10y ago
  Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)
  - lerpa 10y ago
    Exactly, market scrappers as search engines. And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively. The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case. I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.

codeddesign 10y ago

Double standard? The difference is that Google Bot is built on being unobtrusive. I can easily built a scraper that will quickly ddos a site. Linkedin for example...if they allow 10,000 people to send 100 scraping requests per second everyday then that is stolen bandwidth that Linkedin has to pay for and the scrapers get free data. The difference is that Google has standards in which site's unusually benefit from, not to mention that they allow for you to disallow their bot. It just doesn't work the same way with some random developer building a scraper.

encoderer 10y ago
I agree that Googlebot is well behaved. When it detects your site is slowing down, it will back itself off. Unfortunately, this is often to your detriment. In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.
- Guest98123 10y ago
  Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.
  - encoderer 10y ago
    That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.
    - Guest98123 10y ago
      I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.
      - 10y ago

desireco42 10y ago

Google and others, with legitimate reasons obey robots.txt