4 ms·

I personally have found a lot of difficulty with JSON-based logs, because tools like `grep` and `cut` don't suffice; I usually have to write a more complex rege

by memset 14y ago

I personally have found a lot of difficulty with JSON-based logs, because tools like `grep` and `cut` don't suffice; I usually have to write a more complex regex with `sed` in order to extract useful information I want. Anyone have a good utility for managing JSON in log files?

mgurlitz 14y ago
Moving away from using regex to parse JSON, you can use Jsawk and Underscore-cli to manipulate JSON like Awk or JavaScript. https://github.com/micha/jsawk/ https://github.com/micha/jsawk/ https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli
- gbog 14y ago
  But the problem is speed. I found that if parsing logs is slow, bug hunting is far less efficient.
- robmil 14y ago
  underscore-cli is utterly brilliant for this, even better — dare I say it? — than traditional log files with cut/grep/awk. Being able to treat logs as a dataset that can be mapped/reduced/flattened/grouped/otherwise analysed is life-changingly good. About the strongest endorsement I can give is that it makes me wish more of my logs were in JSON format, rather than wishing that less were as I always had before.
jcrites 14y ago
RecordStream is a great solution for manipulating JSON log files: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream - makes it easy to perform relational-style operations, such as grouping, counting (recs-collate), and even joins (recs-join). It can also display and transform the logs in a variety of ways (recs-tocsv, recs-totable, recs-tognuplot for graphing). There are also tools to extract records from other data formats such as CSV or by querying a database. All in all, a fantastic tool for manipulating and querying complex data from the command line. I use it regularly.
dmv 14y ago
Recs (therecordstream.com) is great for this. A JSON log file would be interpreted as its native record format, so you can use all of its processing tools (like recs-grep and recs-collate and recs-sort).
michaelcampbell 14y ago
Here's a bash JSON parser. https://github.com/dominictarr/JSON.sh https://github.com/dominictarr/JSON.sh

by memset 14y ago

mgurlitz 14y ago

Moving away from using regex to parse JSON, you can use Jsawk and Underscore-cli to manipulate JSON like Awk or JavaScript. https://github.com/micha/jsawk/ https://github.com/micha/jsawk/ https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli

gbog 14y ago
But the problem is speed. I found that if parsing logs is slow, bug hunting is far less efficient.
robmil 14y ago
underscore-cli is utterly brilliant for this, even better — dare I say it? — than traditional log files with cut/grep/awk. Being able to treat logs as a dataset that can be mapped/reduced/flattened/grouped/otherwise analysed is life-changingly good. About the strongest endorsement I can give is that it makes me wish more of my logs were in JSON format, rather than wishing that less were as I always had before.

jcrites 14y ago

RecordStream is a great solution for manipulating JSON log files: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream - makes it easy to perform relational-style operations, such as grouping, counting (recs-collate), and even joins (recs-join). It can also display and transform the logs in a variety of ways (recs-tocsv, recs-totable, recs-tognuplot for graphing). There are also tools to extract records from other data formats such as CSV or by querying a database. All in all, a fantastic tool for manipulating and querying complex data from the command line. I use it regularly.

dmv 14y ago

Recs (therecordstream.com) is great for this. A JSON log file would be interpreted as its native record format, so you can use all of its processing tools (like recs-grep and recs-collate and recs-sort).

michaelcampbell 14y ago

Here's a bash JSON parser. https://github.com/dominictarr/JSON.sh https://github.com/dominictarr/JSON.sh