4 ms·
jq is incredibly powerful and I'm using it more and more. Even better, there is a whole ecosystem of tools that are similar or work in conjunction with jq: * j
by weinzierl 5y ago
jq is incredibly powerful and I'm using it more and more. Even better, there is a whole ecosystem of tools that are similar or work in conjunction with jq:
* jq (a great JSON-wrangling tool)
* jc (convert various tools’ output into JSON)
* jo (create JSON objects)
* yq (like jq, but for YAML)
* fq (like jq, but for binary)
* htmlq (like jq, but for HTML)
List shamelessly stolen from Julia Evans[1]. For live links see her page.
Just a few days ago I needed to quickly extract all JWT token expiration dates from a network capture. This is what I came up with:
fq 'grep("Authorization: Bearer.*" ) | print' server.pcap | grep -o 'ey.*$' | sort | uniq | \
jq -R '[split(".") | select(length > 0) | .[0],.[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson]' | \
jq '.[1]' | jq '.exp' | xargs -n1 -I! date '+%Y-%m-%d %H:%M:%S' -d @!
It's not a beauty but I find the fact that you can do it in one line, with proper parsing and no regex trickery, remarkable.
[1] https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/ https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-l...
- kitd 5y agoAlso highly recommended is gron [0], to make json easily searchable [0] https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron
- chriswarbo 5y agoThe yq tool also provides 'xq', which works on XML :)
- hoherd 5y agoI would definitely add dasel to that list. It's become my de facto serialized data converter, and regularly use it to convert between csv, toml, yaml, json, and xml using jq-ish syntaxes. https://github.com/tomwright/dasel https://github.com/tomwright/dasel
- toxik 5y agoIs your example not easier to write and read as a 10-something line Python script? I never understood the appeal of jq etc because of this very reason.
- deleted 5y ago[deleted]
- stormbrew 5y agotbh my biggest problem with all these tools is that I really don't want to have to learn one for each of the json-y formats I have to use every day. If jq supported toml and yaml natively I'd be much much much happier to learn its kind of obtuse syntax.
- samcal 5y agoHave you checked out https://www.nushell.sh/ https://www.nushell.sh/? It seems like exactly what you're describing. Although I know of people who are happily using it as their main shell, I only really use it when I need to read and manipulate data in files.
- chris37879 5y agoI need someone to make a `Q` wrapper that amalgamates all of them. And if that's already taken by a common utility, I vote we name it deLancie, instead.
- wwader 5y agoHi, lover of jq and author of fq here! Just wanted to mention that fq is jq so you can do things like `fq 'grep("Authorization: Bearer.*") | split("\n") | ...' file.pcap`. Also i'm working and prototyping some kind of http decoding support that will make things like select on headers and automatic/manual decoding body possible.