4 ms·
Some interesting disallows there such as Disallow: /topics/Brooklyn_Police_Department. Can anyone comment on why this is?
by codewithcheese 11y ago
Some interesting disallows there such as Disallow: /topics/Brooklyn_Police_Department. Can anyone comment on why this is?
- PhasmaFelis 11y agoAnd /topics/Advil? Wha?
- chkuendig 11y agoThat seems to pop up semi-frequently, It's basically a voluntary "right to forget" provided by the publishers. http://f.cl.ly/items/1o2k3u2J343O3y3X290J/Image%202014-04-07%20at%2012.22.04%20PM.png http://f.cl.ly/items/1o2k3u2J343O3y3X290J/Image%202014-04-07... http://harpers.org/robots.txt http://harpers.org/robots.txt https://twitter.com/waxpancake/statuses/453024948475293696 https://twitter.com/waxpancake/statuses/453024948475293696
- codewithcheese 11y agoVery interesting, good to know. Thanks!