8 ms·
FX: An interactive alternative to jq to process JSON
- peterohler 5y agoAnother alternative is the oj app (ojg/cmd/oj) which is part of https://github.com/ohler55/ojg https://github.com/ohler55/ojg. It relies on JSONPath for extraction and manipulation of JSON.
- JimDabell 5y agoThere’s also jp, which interprets JMESPath: https://github.com/jmespath/jp https://github.com/jmespath/jp This one has the advantage of being natively understood by aws-cli, meaning you can pass a JMESPath to an AWS call and only receive the filtered / transformed result back.
- ducaale 5y agoApparently, the author wrote this tool because jid was struggling with a 7MB JSON file. See https://github.com/simeji/jid/issues/66#issuecomment-443671843 https://github.com/simeji/jid/issues/66#issuecomment-4436718...
- exhaze 5y agoJust tried out both of these for a large endpoint. - FX "expand/collapse" functionality seems way better for exploring APIs whose shape you don't know - jid is maybe marginally better for APIs where you have instant recall of the exact shape and need to rapidly query it Overall, I like FX better because it provides feedback on your query faster. I am grateful to the author(s) for creating it and I'll be using it instead of JQ whenever I need to wrangle APIs from the CLI.
- ducaale 5y agoFYI, fx has a jid-like mode that can be started by pressing the dot key. See this post[1] for more information. [1] https://medium.com/@antonmedv/discover-how-to-use-fx-effectively-668845d2a4ea https://medium.com/@antonmedv/discover-how-to-use-fx-effecti...
- exhaze 5y agoThank you! I had discovered it when I was playing around with it but forgot to mention that.
- ZeroGravitas 5y agoA generic (partial) solution to this type of thing is just to sample a number of lines from the large input, and do the investigation on that. shuf -n 1000 file This is part of coreutils. There's also jiq, which is a clone of jid (mentioned elsewhere) but with jq syntax
- dewey 5y agoThat would only work if it's line separated JSON though. If you cut off the first 1000 lines of a big JSON file it will be invalid.
- Someone 5y agoThen, use jq —-compact-output '.' | head -10 | foo That is also useful for grepping to filter on records of interest. jq also has --stream for handling large inputs.
- jbverschoor 5y agoThis simply evals the argument. Not an alternative to jq. fx “code to eval”
- kitd 5y agoAre you truly unable to see how fx adds value over plain jq? If you are, then why comment?
- ducaale 5y agoThis doesn't look to me something that can be evaluated by node fx .comments[].authors[].names You might want to take a look at this post[1] to see everything fx can do. [1] https://medium.com/@antonmedv/discover-how-to-use-fx-effectively-668845d2a4ea https://medium.com/@antonmedv/discover-how-to-use-fx-effecti...
- xconverge 5y agoI also like https://github.com/dflemstr/rq https://github.com/dflemstr/rq because it supports a few more formats (protobufs for example)
- deleted 5y ago[deleted]
- okasaki 5y agoIf you want to use jq but with Python syntax, I wrote pq: https://github.com/dvolk/pq https://github.com/dvolk/pq
- egeozcan 5y agoI was once upon a time working heavily with JSON back-ends and wrote a node.js script which, when piped json and lambdas, ran the data thorough the lambda and outputted the result. It was very productive. But then I discovered LINQPad[0] and, "The Legendary Dump". [0]: https://www.linqpad.net/ https://www.linqpad.net/
- visarga 5y agoJQ syntax feels too unusual, doesn't resemble known code, gives me the feeling of looking into cryptic Perl or regex, could never remember the simplest things. For example how would you take key k1 from a list of dicts [{k1: v1, k2: v2}, {k1: v3}]?
- prvnsmpth 5y agoYeah I too found the syntax somewhat unintuitive at times. But to answer your question, you would do 'map(.k1)'
- 1_player 5y agoI agree that jq's query language is very obtuse and probably my biggest barrier towards learning it. I have found great mileage using gron [1], which is very different from jq, but its goal is to promote exploration of a JSON file through common unix tools such as awk and grep. 1: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- mongol 5y agoI want to vouch for gron as well. Apart from being grepable, I found it is easier to orient myself where I am in a very large JSON structure. The location in the hierarchy is present on every single line, no need to scroll up or down to figure it out. Granted, many other tools can help with this as well, but gron does it well.
- skywal_l 5y agoLove this. Such a simple idea yet very helpful. It probably can't do what all jq does but it will solve most of what you usually want to do with json on the command line. Thanks for that tip!
- zimpenfish 5y ago> I have found great mileage using gron `gron` is great but doesn't seem to handle some (extreme-ish) situations that `jq` can, e.g. the json output from the fastnbt-tools. You either get a `token too long` error using `gron -s` because the input is too long (it's 90MB, that's fair) or you get only one set of outputs per key (iyswim) because they get overlapped in memory.
- ananthakumaran 5y agohttps://github.com/akavel/up https://github.com/akavel/up is another tool that can be used to make most of commands interactive
- nikolay 5y ago
- charcircuit 5y agoOn my machine jq depends on oniguruma. Both node and onigurama should be easy to install from your distro package manager.
- dewey 5y agoUsually the problem with dependencies it not that they are hard to install but that you have something additional to install on a machine where you want to use it. Sometimes you might not even have the permission to do so.
- nikolay 5y agoIt's because Homebrew (if you're on macOS) decided not to compile it statically.
- charcircuit 5y agoI'm on manjaro.
- lr1970 5y agogron [0] is another interesting JSON processor that follows UNIX philosophy: [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron "Make JSON greppable!" "gron transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute 'path' to it."
- fivea 5y agoI absolutely love gron, but I have to confess I feel dirty when I use it. It's unashamedly a brute-force tool in a world with plenty of elegant alternatives, and the main reason to use it is pure laziness to just shamelessly grep stuff around. And I love it for that.
- fiestajetsam 5y agoIt would be a good idea if this was added to one of the existing[1] comparison tools for similar tools, there's already a lot of them out there. 1: https://cburgmer.github.io/json-path-comparison/ https://cburgmer.github.io/json-path-comparison/
- greatgib 5y agoWould be cool if it was not a cli made in 'node'...
- tyingq 5y agoThat's pretty cool. Would be nice if there was an option to put the currently used filter/query into the shell history or the clipboard. So that you could experiment to find the right one, then back out and use it in a pipeline.
- gryn 5y agoCat xxx.json | fx 'whatever' | fx Is how I usually do so, if you don't want the interactive mode drop the last pipe
- tyingq 5y agoNot sure I understand. I mean using the interactive mode to figure out the right query ("[].mumble.whatever...") and then being able to save the text of the query that you figured out by pointing and clicking. Like a graphical SQL query builder allows you to do.
- FragenAntworten 5y agoYou can do this with `jid -q`: https://github.com/simeji/jid https://github.com/simeji/jid (Its interactivity is keyboard- rather than mouse-based.)
- tyingq 5y agoOh, that's great...outputs a jq-compatible query. Thanks!
- eatonphil 5y agoIf you're into the SQL side (compared to jq's custom query language) of querying arbitrary files I've got a comparison of some major tools here too. https://github.com/multiprocessio/datastation/tree/main/runner/cmd/dsq#comparisons https://github.com/multiprocessio/datastation/tree/main/runn...
- darkstarsys 5y agoLooks really nice. I wish it handled yaml as well.
- mistercow 5y agoI like to have something like this in my shell: y2j () { ruby -r json -r yaml -e 'puts JSON.dump(YAML.load(STDIN))' } Makes it easy to use json tooling for yaml, although of course it flattens out anchors etc.
- nqzero 5y agodo any of these json processing tools support wildcards in key names ? eg, '.long*' instead of '.longNameWithManyDetails'
- meepmorp 5y agoSince nobody else mentioned it, there's also jiq, which uses jq under the hood. https://github.com/fiatjaf/jiq https://github.com/fiatjaf/jiq
- kbrazil 5y agoFor those who are comfortable with python, I created Jello[0], which works like jq but uses python syntax. I also created Jellex[1], which is a TUI built on Jello to assist with building the python queries. Jello gives you the power of python but without all of the boilerplate, so it’s nicer to use in Bash scripts. [0] https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello [1] https://github.com/kellyjonbrazil/jellex https://github.com/kellyjonbrazil/jellex
- _kk_331 5y agoWould be really nice if there was a big warning somewhere in the installation section that it doesn't work on windows (outside wsl).