panza

7 ms·

1. 91.▲
   by panza 15y ago

   "Some data is simply not worth indexing, and not worth serving up to bots." The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running
2. 92.▲
   Robots.txt is a suicide note(archiveteam.org)

   115 points by panza 15y ago|91 comments