Thanks! For my logfiles it runs through them at around 3MB/sec (that's 3MB of compressed file per second). Most of it is probably I/O, my giant logs are on NFS.

Thanks! For my logfiles it runs through them at around 3MB/sec (that's 3MB of compressed file per second). Most of it is probably I/O, my giant logs are on NFS. I run the indexer in parallel on multiple log files: in general there's no way to parallelize the indexing of a single file as decompressing a gzipped file requires all the bytes up to that point. Unless you already have a gzip checkpoint around somewhere, you need to sequentially scan: this is part of what zindex does to allow quick random access. Thanks for the feedback :)