3 ms·

Thanks! I don't why they don't limit Google, Bing crawlers to the homepage via robots.txt. Then crawling should not overload an already busy website. Honestly,

by rpedela 13y ago

Thanks! I don't why they don't limit Google, Bing crawlers to the homepage via robots.txt. Then crawling should not overload an already busy website. Honestly, it seems like an overreaction to me.

gtmtg 13y ago
For what it's worth, pg said that "crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory" (at https://news.ycombinator.com/item?id=3277661 https://news.ycombinator.com/item?id=3277661).