4 ms·A web crawler that can handle any number of websites17 points by mixnode 9y agospoondocz 9y agoWhat's the largest crawl you can handle? meaning your upper limit...mixnode 9y agoWe've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.deleted 9y ago[deleted]