4 ms·
I have to wonder if there have been some recent changes which are causing all kinds of issues. Just a few days ago, Ryan Tomayko twittered about it completely
by mrduncan 17y ago
I have to wonder if there have been some recent changes which are causing all kinds of issues. Just a few days ago, Ryan Tomayko twittered about it completely ignoring Github's robots.txt - http://twitter.com/rtomayko/statuses/7685967826 http://twitter.com/rtomayko/statuses/7685967826
- pjhyett 17y agoOur Mysql boxes were hammered thanks to this bug. Pages that weren't supposed to be crawled because they're resource intensive were all getting hit. Very frustrating, we ended up blocking all msnbot traffic in Nginx.