3 ms·

What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds: cut -d ' ' -f

by a3_nm 15y ago

What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds: cut -d ' ' -f1 log | sort | uniq -c | sort -nr | head Say you need to follow accesses to a particular file? The following quick and dirty one-liner probably works well enough: tail -f log | grep --line-buffered file.pdf How do you do that with json? Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.

tim_sw 15y ago
recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream
deno 15y ago
Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/ http://xml-coreutils.sourceforge.net/ In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same. Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.
keenerd 15y ago
> How do you do that with json? Depends on the schema used, but it would probably be something like jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head
gravitystorm 15y ago
The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more. I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).

by a3_nm 15y ago

tim_sw 15y ago

recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream

deno 15y ago

Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/ http://xml-coreutils.sourceforge.net/ In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same. Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.

keenerd 15y ago

> How do you do that with json? Depends on the schema used, but it would probably be something like jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head

gravitystorm 15y ago

The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more. I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).