7 ms·
I feel like this is a really common theme I've seen several times. Something like "Music Lyric site X sues Google for embedding their lyrics in the results dire
by penagwin 7y ago
I feel like this is a really common theme I've seen several times. Something like "Music Lyric site X sues Google for embedding their lyrics in the results directly" which is funny because site X got the lyrics by scraping them from other sites.
Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content".
I find it really funny that the scrapers are battling scrapers - like guys you only exist because you do THE EXACT SAME THING
- dmix 7y agoRegardless, there is legitimate value in the collection, cleaning, interlinking, and presentation of existing data. How that is interpreted by the law is one thing but merely because the data came from a variety of other public/private sources doesn't mean it derived all of its value externally.
- meowface 7y agoFor sure, but they shouldn't be hypocritical about it. If they don't consider themselves content parasites, they shouldn't consider people scraping their site to be content parasites, either. (Some sites really are just parasites, though.)
- dodobirdlord 7y agoThere's nothing hypocritical about it. Googlebot respects robots.txt configured on pages it scrapes. Google in turn expects that their own robots.txt will be respected. What's the issue? https://www.google.com/robots.txt https://www.google.com/robots.txt
- olau 7y agoCan I politely point out that the conversation is not about respecting robots.txt. If you want to talk about this in terms of robots.txt, Google is thriving on the fact that other companies don't block their content in robots.txt, but at the same time Google blocks all of its content in its robots.txt.
- dodobirdlord 7y ago> If you want to talk about this in terms of robots.txt, Google is thriving on the fact that other companies don't block their content in robots.txt, but at the same time Google blocks all of its content in its robots.txt. It seems like you're stating this as though to cast some sort of moral aspersion. I don't get it. If other companies don't want Googlebot to scrape them they just have to say so. Most companies want Googlebot to scrape their content. Google doesn't want other people's scrapers to scrape Google's content. Nobody involved in any of this has done anything unreasonable or morally objectionable.
- deleted 7y ago[deleted]
- falcor84 7y agoWell, setting up so-called Barriers to Entry[0] is econ 101. [0] https://en.wikipedia.org/wiki/Barriers_to_entry https://en.wikipedia.org/wiki/Barriers_to_entry
- cwkoss 7y agoCreating barriers to entry is an antisocial tactic that harms consumers and society at large. It is the responsibility of moral consumers to avoid spending their money with companies that use these regressive tactics.
- jibal 7y agoOne could argue that it's immoral to respond to a comment about a topic when one hasn't read the link in that comment.
- leesalminen 7y agoI think it’s important to distinguish types of barriers to entry. Some are “real” while others are “artificial”. For example, a real barrier to entry would be institutional knowledge about an industry while an artificial one would be an arbitrary TOS clause.
- TeMPOraL 7y agoAnd disallowing scraping or making it difficult while refraining from providing an API for the same data is the arbitrary kind. The default state of the web is that it's trivially scrapable - you have to go out of your way to make it harder.
- ijidak 7y ago> Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content". Yes. This is EXTREMELY frustrating. Of all companies to prevent scraping, Google is the most ironic. Especially since their goal is to organize the world's information, it shocks me that there's no way to get access to this organized information from machine to machine.
- TylerE 7y agoI d0n't really think it's inappropriate or ironic. I can easily imagine naive scrapers essentially DDOSing google.
- aledalgrande 7y agoYes, we want a Google Search API [at a decent price].
- 3xblah 7y agoPerhaps this issue will be recognised in some of the antitrust investigations. If I am not mistaken, they no longer claim "organize the world's information" as their goal.
- etaioinshrdlu 7y agoBut it still is: > https://about.google/ https://about.google/ "Our mission is to organize the world’s information and make it universally accessible and useful."
- 3xblah 7y agoAppears I am mistaken. Cheers.
- skybrian 7y agoGoogle respects robots.txt, so it's arguably not the same as scraping a website without their consent.
- aledalgrande 7y agoMost sites don't have their main data/functionality in the Disallow section though.
- dodobirdlord 7y agoSure, but they could if they wanted to and that's their own business.
- Breza 7y agoGoogle respects robots.txt files from webmasters who don't want Google to scrape their content