Try just using ripgrep, sed, cut, etc. I found it is good enough for most data analytics I want to do on dumps like these. Particularly replacing grep with ripg

Try just using ripgrep, sed, cut, etc. I found it is good enough for most data analytics I want to do on dumps like these. Particularly replacing grep with ripgrep makes a big difference. If that isn't fast enough, perhaps try feeding them into clickhouse. There is a recent talk on fosdem about how to load dumps easily into it: https://fosdem.org/2023/schedule/event/fast_data_analytical_apps_with_clickhouse/ https://fosdem.org/2023/schedule/event/fast_data_analytical_...