I also wrote up an analysis of the top 1M robots.txt files: http://www.benfrederickson.com/robots-txt-analysis/ http://www.benfrederickson.com/robots-txt-analys

I also wrote up an analysis of the top 1M robots.txt files: http://www.benfrederickson.com/robots-txt-analysis/ http://www.benfrederickson.com/robots-txt-analysis/ I ended up analyzing very different things from this article though, so this article was still pretty interesting to me.