4 ms·

What's the largest crawl you can handle? meaning your upper limit...

• mixnode 9y ago
  We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.