Its performance shines in larger scales. It’s designed for politeness to individual domains, but scales out well for very wide crawls of many domains. It’s pret

Its performance shines in larger scales. It’s designed for politeness to individual domains, but scales out well for very wide crawls of many domains. It’s pretty much endlessly configurable, but not the easiest to learn.