4 ms·

I maintain web crawlers for a large internet portal and this stuff is a pain enough to deal with when people unintentionally make their websites recursive, much

• arbuge 13y ago
  You need to update those crawlers to enforce some hard limits on crawl depth, methinks, especially for low pagerank domains (if there's some easy way to estimate pagerank, i.e.).