3 ms·
https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
by pca006132 6y ago
https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
- syberspace 6y agoI am aware. Why do these news corporations not just write an appropriate robots.txt so Google can't take their content? Is it maybe because they want their cake AND eat it too?
- Bakary 6y agoOn one hand, it's true that they are making odd demands and are linked to Murdoch. On the other, it's disturbing that you pretty much have to play ball with Google and Facebook to survive as a media institution. Yeah, they could create their own competing service I suppose but in practice that sounds more like a non-sequitur than a workable suggestion.