3 ms·

Google's _entire business_ is based not only on accessing servers without authorization, frequently in explicit violation of that site's terms of use because To

Google's _entire business_ is based not only on accessing servers without authorization, frequently in explicit violation of that site's terms of use because ToU boilerplate includes languages excluding all "crawlers, bots, or spiders", but also on flagrant violation of copyright law. They save the entire text of the web page on their servers when they crawl it (unlicensed copying), rehost it in Google Cache (unlicensed redistribution), save all the images and rehost them on Google Images (both), and so forth. All of this is absolutely illegal under current copyright law. Google was fortunately that no one sued them for these things before they got big enough to defend themselves. Many tech entrepreneurs haven't been so lucky. You may ask why a big company like Google isn't doing more to change the CFAA or copyright law. The reason is now that they're big enough, legal grey areas like those in the CFAA (particularly, "what is unauthorized access?", because it's not defined by the statute) can be fully exploited, and Google can sit secure in the knowledge that they'll never be realistically challenged on it; meanwhile, they can then threaten potential competitors for doing the same thing, since a lawsuit against a public corporation takes 10 years and $5MM-$20MM. Anyone who could mount that kind of offense against Google won't, because they benefit from the grey area too; they'll just make some backroom deal with Google and not lose their lucrative, competition-destroying ability to do things that companies with sub-$100M revenues aren't able to do.

dosaygo 11y ago
Okay, there are other aspects to Google's business beside information retrieval and search, tho the point that the access is mostly unauthorized is valid. Although the damage done by this is to some extent offset by Google's status essentially as a public utility: it universally provides a social good, "search", and the tax we pay is advertising. Your frustration for the "little guys" is mostly en pointe, since the damage done is rendered then mostly to competitors, rather than to customers, and the argument, true or no, that Google's search is "lightyears" ahead of other possible offerings, to some extent offsets the damage to consumers due to loss of competition through the possibly anti-competitive practices you highlight. So it seems the balance of good is in the favor of consumers of "search". I think this is the main force, rather than any "structural obstacles" to competition, which is the cause of the persistence of the status quo in this market. The thing about this which no one seems to see is that, since everyone is taking information dishonestly, there's a huge opportunity to actually "bring it into the light", do it honestly, and strike some kind of deals with content creators.
roywiggins 11y ago
If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right? From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.

dosaygo 11y ago

Okay, there are other aspects to Google's business beside information retrieval and search, tho the point that the access is mostly unauthorized is valid. Although the damage done by this is to some extent offset by Google's status essentially as a public utility: it universally provides a social good, "search", and the tax we pay is advertising. Your frustration for the "little guys" is mostly en pointe, since the damage done is rendered then mostly to competitors, rather than to customers, and the argument, true or no, that Google's search is "lightyears" ahead of other possible offerings, to some extent offsets the damage to consumers due to loss of competition through the possibly anti-competitive practices you highlight. So it seems the balance of good is in the favor of consumers of "search". I think this is the main force, rather than any "structural obstacles" to competition, which is the cause of the persistence of the status quo in this market. The thing about this which no one seems to see is that, since everyone is taking information dishonestly, there's a huge opportunity to actually "bring it into the light", do it honestly, and strike some kind of deals with content creators.

roywiggins 11y ago

If those sites don't want spiders, they can just specify that in robots.txt, which Google honors, right? From the point of view of the law, that might not matter, but when there's a standardized way to make clear to bots that they're not welcome and you didn't bother to implement it, you'll look pretty silly if you complain.