3 ms·
" A search engine doesn't discriminate in what content it indexes." That is incorrect. http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipe
by chopsueyar 14y ago
" A search engine doesn't discriminate in what content it indexes."
That is incorrect.
http://en.wikipedia.org/wiki/Robots_exclusion_standard http://en.wikipedia.org/wiki/Robots_exclusion_standard
Tubetrooper certainly discriminates.
- blauwbilgorgel 14y agoRobots.txt doesn't have anything to do with this. Google adheres to robots.txt, but doesn't discriminate in the kind of content it indexes. Search engines can certainly specialize, like in your (unnecessary) porn site example. But if they specialize in providing links to copyright-infringing material, make profit with advertisements, they can get in just as much trouble. The OP made the comparison with search engines. How come they can have copyright-infringing links and not get into trouble? It is because they index all information available to them, without discrimination. And then they still have to respond to DMCA claims. If a search engine specialized in engineering parts, it wouldn't need to host links to copyright-infringing material, so most specialized search engines are a non-issue.