3 ms·

> ... copyrighted material... Isn't the majority (if not the whole) of the web copywrited? It seems like part of the reason why Google has monopoly over search

by jolmg 9y ago

> ... copyrighted material... Isn't the majority (if not the whole) of the web copywrited? It seems like part of the reason why Google has monopoly over search is because search by crawling and indexing is fundamentally illegal. It also seems like the reason they're not prosecuted is because they're so big and useful that no website owner minds that their site is crawled by Google, even though most of them set Terms and Conditions in place that forbid crawling, scraping, or otherwise automated compilation of information from their site. It's a double standard.

ghaff 9y ago
Yes, almost everything on the web is copyrighted with the exception of content from governments that do not copyright content that they create and material that is out of copyright. It is unclear that crawling and indexing are a violation of copyright. A library card catalog does not violate the copyright of the material it indexes. If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Caches are on shakier ground and archives like the IA are on shakier ground yet. Pretty much everyone turns a blind eye however because it's useful and again you can always opt out with robots.txt.
- slededit 9y ago
  The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you actually try to crawl a website you'll find yourself blackholed by a lot of websites pretty quickly.
- jolmg 9y ago
  > If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Just to be clear, it's not really Google's crawling that I meant to criticize, but rather the double standard employed by the majority of website owners that strengthens Google's monopoly. If anything, I wish more website owners wouldn't mind automated use and copying of their site from anyone and not just major players like Google.

by jolmg 9y ago

ghaff 9y ago

Yes, almost everything on the web is copyrighted with the exception of content from governments that do not copyright content that they create and material that is out of copyright. It is unclear that crawling and indexing are a violation of copyright. A library card catalog does not violate the copyright of the material it indexes. If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Caches are on shakier ground and archives like the IA are on shakier ground yet. Pretty much everyone turns a blind eye however because it's useful and again you can always opt out with robots.txt.

slededit 9y ago
The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you actually try to crawl a website you'll find yourself blackholed by a lot of websites pretty quickly.
jolmg 9y ago
> If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Just to be clear, it's not really Google's crawling that I meant to criticize, but rather the double standard employed by the majority of website owners that strengthens Google's monopoly. If anything, I wish more website owners wouldn't mind automated use and copying of their site from anyone and not just major players like Google.