3 ms·
sed, awk, grep and friends are just so effective at trawling through text. I dump about 150GB of Postgres logs a day (I know, it's over the top but I only keep
by aidos 2y ago
sed, awk, grep and friends are just so effective at trawling through text.
I dump about 150GB of Postgres logs a day (I know, it's over the top but I only keep a few days worth and there have been several occasions where I was saved by being able to pick through them).
At that size you even need to give up on grepping, really. I've written a tiny bash script that uses the fact that log lines start with a timestamp and `dd` for immediate extraction. This allows me to quickly binary search for the location I'm interested in.
Then I can `dd` to dump the region of the file I want. After that I have an little awk script that lets me collapse the sql lines (since they break across multiple lines) to make grepping really easy.
All in all it's a handful of old school script that makes an almost impossible task easy.
- porridgeraisin 2y agoCan you explain how you used dd here? Ive never seen it used this way, curious
- fwip 2y agodd lets you specify an offset to start reading the file at, with `skip`. This would let you perform a binary search by picking an offset in the file, reading a small chunk (say, a kilobyte), and scanning for the date/time string within it. Each read should be O(1) in terms of the size of the file, so a O(log(n)) for the binary search, whereas a grep-based approach is O(n). (The datetime in the log message is presumably sorted, or nearly so).
- aidos 2y agoSure! I've created a gist so you can see for yourself but the basic idea is as described. Read a chunk, find the first date in it and then decide if you want to read further forward or back in the file. https://gist.github.com/aidos/5a6a3fa887f41f156b282d72e1b79f43 https://gist.github.com/aidos/5a6a3fa887f41f156b282d72e1b79f... For anyone else, here's the awk for combining lines in the log files for making them greppable too: https://gist.github.com/aidos/44a9dfce3c16626e9e7834a83aed9188 https://gist.github.com/aidos/44a9dfce3c16626e9e7834a83aed91...