4 ms·
It should be fine if they add it to their robots.txt file. It could alongside the other interesting exclusions: http://www.wired.com/robots.txt http://www.wire
by threeseed 11y ago
It should be fine if they add it to their robots.txt file.
It could alongside the other interesting exclusions:
http://www.wired.com/robots.txt http://www.wired.com/robots.txt
- codewithcheese 11y agoSome interesting disallows there such as Disallow: /topics/Brooklyn_Police_Department. Can anyone comment on why this is?
- PhasmaFelis 11y agoAnd /topics/Advil? Wha?
- chkuendig 11y agoThat seems to pop up semi-frequently, It's basically a voluntary "right to forget" provided by the publishers. http://f.cl.ly/items/1o2k3u2J343O3y3X290J/Image%202014-04-07%20at%2012.22.04%20PM.png http://f.cl.ly/items/1o2k3u2J343O3y3X290J/Image%202014-04-07... http://harpers.org/robots.txt http://harpers.org/robots.txt https://twitter.com/waxpancake/statuses/453024948475293696 https://twitter.com/waxpancake/statuses/453024948475293696
- codewithcheese 11y agoVery interesting, good to know. Thanks!
- chinathrow 11y agoAlso in there: http://www.wired.com/2007/07/fbi-spyware-how/ http://www.wired.com/2007/07/fbi-spyware-how/