4 ms·

The problem now is that it's not only the traditional media companies that support this, it's google, twitter, facebook and the like. They are big enough that

The problem now is that it's not only the traditional media companies that support this, it's google, twitter, facebook and the like. They are big enough that they can eat the cost of censorship and don't care if they become worse services, so long as no new services arise to take their place. And with laws like this in place they never will. The single biggest hurdle I found to building a distributed search engine was the various illegal materials all over the world. Be it insults to religion, hate speech, copyrighted material, child pornography, all need to be copied onto a local machine to be indexed and categorized. In most countries this would lead the police to come knocking. Any case will (probably) eventually be ruled to be legal, but the average user will never be able to deal with the case since they will probably go bankrupt long before a victory. Even the people who set out to run a system like tor, knowing full well what the risks are, as a form of mild civil disobedience eventually fold [0]. [0] https://motherboard.vice.com/en_us/article/5394ax/the-operators https://motherboard.vice.com/en_us/article/5394ax/the-operat...

walterbell 9y ago
Similar issues exist with malware and spam, where reputation lists are maintained by various parties. These lists are used by browsers and other web clients before accessing the target URL/IP. Are such lists used by search engines?
jolmg 9y ago
> ... copyrighted material... Isn't the majority (if not the whole) of the web copywrited? It seems like part of the reason why Google has monopoly over search is because search by crawling and indexing is fundamentally illegal. It also seems like the reason they're not prosecuted is because they're so big and useful that no website owner minds that their site is crawled by Google, even though most of them set Terms and Conditions in place that forbid crawling, scraping, or otherwise automated compilation of information from their site. It's a double standard.
- ghaff 9y ago
  Yes, almost everything on the web is copyrighted with the exception of content from governments that do not copyright content that they create and material that is out of copyright. It is unclear that crawling and indexing are a violation of copyright. A library card catalog does not violate the copyright of the material it indexes. If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Caches are on shakier ground and archives like the IA are on shakier ground yet. Pretty much everyone turns a blind eye however because it's useful and again you can always opt out with robots.txt.
  - slededit 9y ago
    The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you actually try to crawl a website you'll find yourself blackholed by a lot of websites pretty quickly.
  - jolmg 9y ago
    > If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Just to be clear, it's not really Google's crawling that I meant to criticize, but rather the double standard employed by the majority of website owners that strengthens Google's monopoly. If anything, I wish more website owners wouldn't mind automated use and copying of their site from anyone and not just major players like Google.
tylersmith 9y ago
Off topic, but I'd be interested in checking out your distributed search project if it's public.
- zeth___ 9y ago
  I did this 15 years ago when still in highschool. There was nothing of value code wise. The one useful thing I got out of it is the realization how much illegal content there is everywhere. I pointed a badly written spider at vairous forums and chans and let it collect data from there. The 'ranking' was a list with the number of times a post with a link was quoted. I had plans on how to propogate the numbers through the graph of links, but after I looked at what I'd gotten in 2 hours I promptly deleted everything and reformatted that drive.

walterbell 9y ago

Similar issues exist with malware and spam, where reputation lists are maintained by various parties. These lists are used by browsers and other web clients before accessing the target URL/IP. Are such lists used by search engines?

jolmg 9y ago

> ... copyrighted material... Isn't the majority (if not the whole) of the web copywrited? It seems like part of the reason why Google has monopoly over search is because search by crawling and indexing is fundamentally illegal. It also seems like the reason they're not prosecuted is because they're so big and useful that no website owner minds that their site is crawled by Google, even though most of them set Terms and Conditions in place that forbid crawling, scraping, or otherwise automated compilation of information from their site. It's a double standard.

ghaff 9y ago
Yes, almost everything on the web is copyrighted with the exception of content from governments that do not copyright content that they create and material that is out of copyright. It is unclear that crawling and indexing are a violation of copyright. A library card catalog does not violate the copyright of the material it indexes. If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Caches are on shakier ground and archives like the IA are on shakier ground yet. Pretty much everyone turns a blind eye however because it's useful and again you can always opt out with robots.txt.
- slededit 9y ago
  The real problem is that a large majority of sites do permissions by bot, whitelisting only google and possibly bing while excluding all others. Further if you actually try to crawl a website you'll find yourself blackholed by a lot of websites pretty quickly.
- jolmg 9y ago
  > If you don't want to be indexed, you can just put the appropriate entry in robots.txt and all the major search engines will comply. Just to be clear, it's not really Google's crawling that I meant to criticize, but rather the double standard employed by the majority of website owners that strengthens Google's monopoly. If anything, I wish more website owners wouldn't mind automated use and copying of their site from anyone and not just major players like Google.

tylersmith 9y ago

Off topic, but I'd be interested in checking out your distributed search project if it's public.

zeth___ 9y ago
I did this 15 years ago when still in highschool. There was nothing of value code wise. The one useful thing I got out of it is the realization how much illegal content there is everywhere. I pointed a badly written spider at vairous forums and chans and let it collect data from there. The 'ranking' was a list with the number of times a post with a link was quoted. I had plans on how to propogate the numbers through the graph of links, but after I looked at what I'd gotten in 2 hours I promptly deleted everything and reformatted that drive.