You don't need to store it indefinitely though, and there's not much point in crawling faster than you can process the data. The couple of kilobytes per docume

You don't need to store it indefinitely though, and there's not much point in crawling faster than you can process the data. The couple of kilobytes per document is the actual storage footprint. Sure you need to massage the data, but that almost entirely CPU bound. You also need a lot of RAM for keeping the hot parts of the index.