5 ms·

First release of jq in 5 years

114 points by drewda 3y ago

ericbarrett 3y ago
Great to see. jq is nearly as ubiquitous as bash these days. I’m excited about pick()! Could have used this so many times: > Adds new builtin pick(stream) to emit a projection of the input object or array. @pkoppstein #2656 $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | pick(.a, .b.c, .x)' { "a": 1, "b": { "c": 2 }, "x": null }
- wkdneidbwf 3y ago
  this looks so, so useful
- mpalmer 3y ago
  Delighted to see work continue on jq, which I use for altogether way too many things. I ask this as a fan of both the project and the new pick() feature: are there any other functions in jq whose interpretation of their arguments is "non-regular" in the sense the PR author means [here][1]? More concretely, the expression `.a, .b.c, .x` means something very different outside a pick call. It represents a uncollected stream of more than one value, rather than a reduction of an object into a subset of itself. The syntax inside a pick call is (I believe) impossible to implement in jq; it has to be a builtin. There's no way to get "regular jq" to extract the path expressions as strings with which to set an object's keys. Ultimately I do think like the addition, but I recognize the tradeoff of "pure" consistency for syntactical brevity. I'd be curious to hear from the maintainers A) whether I'm off-base about the pathexp syntax limitation and B) if there's some philosophy or design principle guiding why and when they might decide to add specialized interpretations of pathexps. [1]: https://github.com/jqlang/jq/pull/2656#issuecomment-1622822023 https://github.com/jqlang/jq/pull/2656#issuecomment-16228220...
  - wwader 3y ago
    Hey, pick is implemented in jq here https://github.com/jqlang/jq/blob/master/src/builtin.jq#L258-L262 https://github.com/jqlang/jq/blob/master/src/builtin.jq#L258... using various path functions. When the jq manual say "path expression" like the argument to pick you can think of it as just normal jq expression but instead outputting values when path/1 is used it will instead output the paths to those values. The limitation is that you only can index into the input (actually including the null value) and don't construct new literals etc. btw an alternative to pick that might be handy at times is the destruction and shorter object construction syntax: # use {a:{b:1},c:2} as input # . as {a:{$b}} will bind $b to 1 # {$b} is shorthand for {b:$b} # {a} is shorthand for {a:.a} $ jq -n '{a:{b:1},c:2} | (. as {a:{$b}} | {$b}), {a}' { "b": 1 } { "a": { "b": 1 } }
- callalex 3y ago
  I guess I have a terrible imagination. Can someone provide a concrete example of when this should be used?
  - bartvk 3y ago
    I could see this being useful when you coded a utility, which runs off a huge existing JSON as its configuration. You want to go straight for one or two values, not this huge data structure.
  - beardyw 3y ago
    For me, if I am scripting something, do curl, extract value from returned JSON, continue ...
  - 10000truths 3y ago
    If I receive a JSON list of JSON objects from an API call, but only a few of the objects' fields are relevant to me, then I can do something like: curl https://example.com/api/orders | jq '.data[] | pick(.date, .amount)'
    - pcthrowaway 3y ago
      When would this be better than curl https://example.com/api/orders | jq '.data[] | {date, amount}' or curl https://example.com/api/orders | jq '[.data[] | {date, amount}]' if you want it collected back into an array?
      - wwader 3y ago
        In that case it would be very similar but pick is more general then the shorthand object literal syntax, it can be used with nested index lika .a.b.c or any expression like pick(if .a then .b else .c end) but both has it uses i think.
pabs3 3y ago
Great to see the NUL separator work I had a hand in has reached a release. Wish it got to use -0 for easy typing, but with tab completion it shouldn't be too bad.
wwader 3y ago
Note that this is a release candidate so please report any issues!
shortrounddev2 3y ago
Is it not an indictment of bash that, in order to make shells useful to modern data formats, we have to create tools with their own scripting languages and pipes? Since bash can't operate on structured data like jq does, jq has to have pipes inside jq expressions.
- penguin_booze 3y ago
  That's how I'd like it to be. These are domain specific languages, so their own scripting is where I'd see them belong. Command line should stay true to its name: place where you enter your commands (and of course, other things that you can't delegate elsewhere, like variables and control structures).
  - shortrounddev2 3y ago
    yeah but JSON has become to de-facto universal format. At the very least, it would be cool if you could convert that into a common runtime format and pipe structured data from one command to another so you don't have to keep parsing things at every stage, or abandoning the shell entirely and working only in jq
- cryptonector 3y ago
  ksh93 has "compound variables" that let you have arbitrarily deep trees of data. The syntax for that is ugly. It's hard to use. And I've never had luck with not causing ksh93 to crash when trying to use compound variables. I think it'd be very nice to have a bash-like shell with a) jq-like JSON values in the shell (really, parsed, jv-like[0]), b) syntax for using jq programs in ${...}, something like ${var@jq:.jq.program.here}. As for jq having pipes, they're a rather different beast, but I take your point. [0] `jv` is the internal representation of JSON values in jq.
- mike_hock 3y ago
  Yeah, but water is also wet. Shell scripting is beyond salvation. Inconsistent, illogical crap. It's ridiculous that Bash doesn't have first-class JSON support in 2023, but even if it did, that still wouldn't make me want to write Shell.
cryptonector 3y ago
Hey, this is a release candidate. It got made mainly to test the new GitHub Actions release workflow, though it's just as well because we're very close to doing a 1.7. Probably a week away.
jpeeler 3y ago
Some competition for https://github.com/itchyny/gojq https://github.com/itchyny/gojq. I had read somewhere that it was faster than jq - no idea if that's still the case.
- cryptonector 3y ago
  It wouldn't surprise me if it is. Same for jaq and others.

114 points by drewda 3y ago

Great to see. jq is nearly as ubiquitous as bash these days. I’m excited about pick()! Could have used this so many times: > Adds new builtin pick(stream) to emit a projection of the input object or array. @pkoppstein #2656 $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | pick(.a, .b.c, .x)' { "a": 1, "b": { "c": 2 }, "x": null }

wkdneidbwf 3y ago
this looks so, so useful
mpalmer 3y ago
Delighted to see work continue on jq, which I use for altogether way too many things. I ask this as a fan of both the project and the new pick() feature: are there any other functions in jq whose interpretation of their arguments is "non-regular" in the sense the PR author means [here][1]? More concretely, the expression `.a, .b.c, .x` means something very different outside a pick call. It represents a uncollected stream of more than one value, rather than a reduction of an object into a subset of itself. The syntax inside a pick call is (I believe) impossible to implement in jq; it has to be a builtin. There's no way to get "regular jq" to extract the path expressions as strings with which to set an object's keys. Ultimately I do think like the addition, but I recognize the tradeoff of "pure" consistency for syntactical brevity. I'd be curious to hear from the maintainers A) whether I'm off-base about the pathexp syntax limitation and B) if there's some philosophy or design principle guiding why and when they might decide to add specialized interpretations of pathexps. [1]: https://github.com/jqlang/jq/pull/2656#issuecomment-1622822023 https://github.com/jqlang/jq/pull/2656#issuecomment-16228220...
- wwader 3y ago
  Hey, pick is implemented in jq here https://github.com/jqlang/jq/blob/master/src/builtin.jq#L258-L262 https://github.com/jqlang/jq/blob/master/src/builtin.jq#L258... using various path functions. When the jq manual say "path expression" like the argument to pick you can think of it as just normal jq expression but instead outputting values when path/1 is used it will instead output the paths to those values. The limitation is that you only can index into the input (actually including the null value) and don't construct new literals etc. btw an alternative to pick that might be handy at times is the destruction and shorter object construction syntax: # use {a:{b:1},c:2} as input # . as {a:{$b}} will bind $b to 1 # {$b} is shorthand for {b:$b} # {a} is shorthand for {a:.a} $ jq -n '{a:{b:1},c:2} | (. as {a:{$b}} | {$b}), {a}' { "b": 1 } { "a": { "b": 1 } }
callalex 3y ago
I guess I have a terrible imagination. Can someone provide a concrete example of when this should be used?
- bartvk 3y ago
  I could see this being useful when you coded a utility, which runs off a huge existing JSON as its configuration. You want to go straight for one or two values, not this huge data structure.
- beardyw 3y ago
  For me, if I am scripting something, do curl, extract value from returned JSON, continue ...
- 10000truths 3y ago
  If I receive a JSON list of JSON objects from an API call, but only a few of the objects' fields are relevant to me, then I can do something like: curl https://example.com/api/orders | jq '.data[] | pick(.date, .amount)'
  - pcthrowaway 3y ago
    When would this be better than curl https://example.com/api/orders | jq '.data[] | {date, amount}' or curl https://example.com/api/orders | jq '[.data[] | {date, amount}]' if you want it collected back into an array?
    - wwader 3y ago
      In that case it would be very similar but pick is more general then the shorthand object literal syntax, it can be used with nested index lika .a.b.c or any expression like pick(if .a then .b else .c end) but both has it uses i think.

pabs3 3y ago

Great to see the NUL separator work I had a hand in has reached a release. Wish it got to use -0 for easy typing, but with tab completion it shouldn't be too bad.

wwader 3y ago

Note that this is a release candidate so please report any issues!

shortrounddev2 3y ago

Is it not an indictment of bash that, in order to make shells useful to modern data formats, we have to create tools with their own scripting languages and pipes? Since bash can't operate on structured data like jq does, jq has to have pipes inside jq expressions.

penguin_booze 3y ago
That's how I'd like it to be. These are domain specific languages, so their own scripting is where I'd see them belong. Command line should stay true to its name: place where you enter your commands (and of course, other things that you can't delegate elsewhere, like variables and control structures).
- shortrounddev2 3y ago
  yeah but JSON has become to de-facto universal format. At the very least, it would be cool if you could convert that into a common runtime format and pipe structured data from one command to another so you don't have to keep parsing things at every stage, or abandoning the shell entirely and working only in jq
cryptonector 3y ago
ksh93 has "compound variables" that let you have arbitrarily deep trees of data. The syntax for that is ugly. It's hard to use. And I've never had luck with not causing ksh93 to crash when trying to use compound variables. I think it'd be very nice to have a bash-like shell with a) jq-like JSON values in the shell (really, parsed, jv-like[0]), b) syntax for using jq programs in ${...}, something like ${var@jq:.jq.program.here}. As for jq having pipes, they're a rather different beast, but I take your point. [0] `jv` is the internal representation of JSON values in jq.
mike_hock 3y ago
Yeah, but water is also wet. Shell scripting is beyond salvation. Inconsistent, illogical crap. It's ridiculous that Bash doesn't have first-class JSON support in 2023, but even if it did, that still wouldn't make me want to write Shell.

cryptonector 3y ago

Hey, this is a release candidate. It got made mainly to test the new GitHub Actions release workflow, though it's just as well because we're very close to doing a 1.7. Probably a week away.

jpeeler 3y ago

Some competition for https://github.com/itchyny/gojq https://github.com/itchyny/gojq. I had read somewhere that it was faster than jq - no idea if that's still the case.

cryptonector 3y ago
It wouldn't surprise me if it is. Same for jaq and others.