Modern web crawlers usually have mechanisms against rabbit holes, e.g. for Apache Nutch you can configure it should only follow to a certain depth of links from

Modern web crawlers usually have mechanisms against rabbit holes, e.g. for Apache Nutch you can configure it should only follow to a certain depth of links from a root, it should only follow a redirect a few times and so on. It's always a trade-off though. If you cut off too aggressive you don't get everything, if you cut off too late you waste resources.