5 ms·
For anyone who read the README and is now thinking that agrind is "like jq but with aggregations" — it's much more than that, and (at least in the tasks I use i
by derefr 3y ago
For anyone who read the README and is now thinking that agrind is "like jq but with aggregations" — it's much more than that, and (at least in the tasks I use it for) really more complementary to pipeline transformer tools like jq.
Specifically, the "killer app" feature for agrind, is that it's a live reporting tool. You can just throw a CSV file through agrind to get a nice one-shot aggregation out. But if you stream a live data source into it — think something like "your k8s ingress controller's access logs" — then agrind will grab your TTY; draw a table to fills it; and begin live-updating that table with the report, with the rows re-ordering to stay sorted as more data is received.
Want to see e.g. the top 10 highest-duration HTTP requests by path given some particular HTTP status code? You can just start streaming logs into it, and the report will gradually come into focus as it finds matching events. Hit ^C when the data has "stabilized" (i.e. when datapoint ranks are no longer jumping around), and you'll be left with the last-refreshed state of the table in your terminal scrollback, to use in further investigations.
agrind has jq-like manipulation tools, but honestly, I mostly still use jq for that part. I have no idea how to unwrap JSON list items in agrind (or if that's even possible), but if I want to do that, I'll just:
http localhost:8001/v1/debug/inflight-requests | jq -c '.[].jsonPayload' | agrind '* | json | sum(dur) by path'
---
Bonus recommendation, as it's something I use together with jq and agrind: stern (https://github.com/rancher/stern https://github.com/rancher/stern) — for live tail(1)ing of the O(N) pods of a k8s Deployment, right from the cluster, without any external log multiplexer.
stern ingress-nginx-controller -i some_path_frag -o raw 2>/dev/null | agrind '* | json | where status == 403 | count by client_addr' 2>/dev/null
Note the two instances of `2>/dev/null` here.
The first one is to remove the "connected to pod XYZ" meta info from stern(1) output, that would muck up the agrind table.
The second one, though, is more interesting: it's there because unlike jq, agrind doesn't die when it hits an unparseable line! Instead, it just spits a parse error and continues. So you can have agrind consume a "dirty" data stream — e.g. a combined "structured JSON access log + plaintext error log" stream — and then just discard stderr, to silence the resulting errors printed from the failed attempts at parsing the error-log lines of the stream.
- cryptonector 3y agoI mean, jq can aggregate... Maybe it should have more interesting aggregations built-in though.