3 ms·
This has been discussed before -- see https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item?id=5955043 (for the discussion) and https:/
by gtmtg 13y ago
This has been discussed before -- see https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item?id=5955043 (for the discussion) and https://news.ycombinator.com/item?id=5955374 https://news.ycombinator.com/item?id=5955374 (for Matt Cutts' comment).
- rpedela 13y agoThanks! I don't why they don't limit Google, Bing crawlers to the homepage via robots.txt. Then crawling should not overload an already busy website. Honestly, it seems like an overreaction to me.
- gtmtg 13y agoFor what it's worth, pg said that "crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory" (at https://news.ycombinator.com/item?id=3277661 https://news.ycombinator.com/item?id=3277661).