3 ms·

The wikipedia robots.txt

• ploxination 15y ago
  cool
• amikula 15y ago
  Can someone explain to me why this is so highly ranked? If I wanted to read Wikipedia's robots.txt, I don't need HN to suggest it to me. It's got a bunch more stuff than I might have expected. Other than that, what am I missing?
  • srl 15y ago
    Really, I think it's just interesting miscellany that acts as a /very/ partial history of many of the growing pains wikipedia's had. It's amusing, and exactly the sort of thing many in the HN crowd love.
  • derleth 15y ago
    Remember that submission about honeypots a little while ago?