4 ms·
About how long does it take to create the index, maybe compared to a zgrep? Could it be parallelized with something like `parallel --pipe`? I haven't tried ye
by nattaylor 11y ago
About how long does it take to create the index, maybe compared to a zgrep? Could it be parallelized with something like `parallel --pipe`?
I haven't tried yet, but this seems like a great tool.
- mattgodbolt 11y agoThanks! For my logfiles it runs through them at around 3MB/sec (that's 3MB of compressed file per second). Most of it is probably I/O, my giant logs are on NFS. I run the indexer in parallel on multiple log files: in general there's no way to parallelize the indexing of a single file as decompressing a gzipped file requires all the bytes up to that point. Unless you already have a gzip checkpoint around somewhere, you need to sequentially scan: this is part of what zindex does to allow quick random access. Thanks for the feedback :)