3 ms·
There is a robots.txt. If this mechanism would not exist, I would argue that the publishers have a case regarding Google (they are trying). But it does exist an
by quonn 3y ago
There is a robots.txt. If this mechanism would not exist, I would argue that the publishers have a case regarding Google (they are trying). But it does exist and so they shouldn‘t.
- figassis 3y agoMissing robots.txt does not automatically grant you the right to use copyrighted content.
- cxr 3y ago<https://en.wikipedia.org/wiki/Fallacy_of_presupposition https://en.wikipedia.org/wiki/Fallacy_of_presupposition> <https://en.wikipedia.org/wiki/Loaded_question https://en.wikipedia.org/wiki/Loaded_question>
- mrkramer 3y agoSomewhere I read that a lot of websites don't even have robots.txt so I wonder does Google's crawler bots skip those websites or do they crawl them as someone said as a "fair use". Speaking generally about internet search engines; majority of the websites on the web want to get discovered and attract people to their whatever (web store, web community, web blog etc.) The interesting idea I was thinking about is that a big company which operates a search engine like a Microsoft (Bing), can pay for example popular websites like Reddit, NYT, WSJ to have exclusive crawling right and therefore block Google from crawling their websites and only allow Bing to crawl them. This would then spark search engine content war which could significantly weaken Google's monopoly because a lot of people would switch to Microsoft Bing only because Reddit results show up in their search queries. This would be akin to Netflix investing billions of dollars in the exclusive content which then brings them a lot of new subscribers and lot of new revenue. In another words - user acquisition.