6 ms·

jq - like sed for JSON data

132 points by gnosis 13y ago

deleted 13y ago
[deleted]
NelsonMinar 13y ago
I've been using jq for a few months now while working with big GeoJSON blobs and it's a big help. The query language takes a bit of doing at first, but it's really pretty great once you learn it. curl -s http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.json http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.... | jq '.features[].properties.name'
- dc2447 13y ago
  Personally jgrep seems a better choice here curl foo/bar.json | jgrep -n -s 'property' Or even better curl foo/bar.json | jgrep 'status=500' -n -s 'path' http://jgrep.org/ http://jgrep.org/
laumars 13y ago
Completely unrelated, but I love the typeface he's used on his site. In fact the whole site is laid out really nicely (pretty, yet none of the aesthetics gets in the way of usability. a balance that most sites fail to strike). Anyhow, tangents aside, this looks a really handy tool. Thanks :)
revertts 13y ago
recs has this functionality, and _way_ more; worth giving a shot: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream
- jcrites 13y ago
  I'm a big fan of RecordStream. Everyone should definitely give it a look. However, is it true to say that recs has this functionality? RecordStream is great with sequences of relatively simple JSON structures, where you want to count, group, collate, etc. the sequence. It looks like jq is better able to handle large, complex arbitrary JSON objects, pulling out deeply nested fields and transforming them. For example, how would you implement jq's "recurse" in RecordStream? Now what would be really cool is using the two tools together :-)
doctorpangloss 13y ago
I like the concept, but how does it compare to using node + underscore? Isn't the whole idea of JSON that it exactly serializes most JavaScript objects, and the "right way" to do stuff with it is to just use it like a normal object in your program?
- Xorlev 13y ago
  The spirit of the tool is to make JSON easier to work with in shell flows. There's plenty of times where you'll get a deluge of JSON and need to work with it (for debugging or whatever). You may not have a program to use the JSON in.
- mkehrt 13y ago
  This assumes you're using JSON in Javascript. JSON is a pretty widely spread human-readable data format now outside of javascript, and it's nice to have good tools that aren't tied to a particular language.
nicpottier 13y ago
This is super cool, the language seems both powerful and easy enough to learn. Nicely done.
bsaul 13y ago
Considering the amount of json around, wouldn't it be time to settle on a standard for : - schema definition ( using something like typescript ? ) - data path ( similar to xpath ) - data transformation ( similar to xslt ) i'm pretty sure many people have done their things but i'm still waiting to see a tool like wsdl - based stub generators that would work on many json api. ps : i know that things like jsonpath and wsdl2 exist, but i really feel like those technologies haven't caught widespread adoption.
- haberman 13y ago
  I agree and I believe that the way to accomplish this is to use Protocol Buffer schemas. Protocol Buffers and JSON have an extremely similar data model. The parts of them that people actually use are effectively identical. The actual differences are minor things like: - JSON can have arrays of arrays; with Protocol Buffers you need a message in between. - Protocol Buffers can represent binary data directly; with JSON you have to base64 encode it into a string. - Protocol Buffers are more specific about the size of integer types (int64 and int32 are different); this allows for more efficient in-memory representations but doesn't matter so much on-the-wire. I think the two are a match made in heaven.
  - bsaul 13y ago
    Yes, i also thought of protocol buffer right after sending that comment. Although i wonder : is there any way you could automatically generate a full service/model layer for, let's say angularjs or objective-c, based solely on protocol buffer "standard" ?
  - nly 13y ago
    You're basically describing Piqi: http://piqi.org/ http://piqi.org/
    - haberman 13y ago
      Thanks for the reference, I hadn't seen that. I really believe, though, that the solution to interoperability is to use protocol buffer schemas directly; not to introduce yet another schema language.
guard-of-terra 13y ago
And for xml I highly recommend xmlstarlet. query/transform xml documents from the command line.
evaneykelen 13y ago
For readers who are unfamiliar with sed: don't dimiss jq as yet another json pretty printer. From the jq tutorial "Data in jq is represented as streams of JSON values - every jq expression runs for each value in its input stream, and can produce any number of values to its output stream.". I've played around with jq and it will definitely become a tool in my toolbox. Kudos to the author.
iso8859-1 13y ago
Previously on HN (141 days ago): https://news.ycombinator.com/item?id=4985250 https://news.ycombinator.com/item?id=4985250
- quarterto 13y ago
  Clearly, this is new to enough people that it got to the front page. Merely linking to a previous submission adds nothing here; if you thing the discussion there was informative (clearly not the case here), then say so.
  - iso8859-1 13y ago
    Yeah, I think the previous discussion was informative, it has alternatives and links to ever older relevant posts. Let the karma decide what HN likes. Like you said yourself: https://news.ycombinator.com/item?id=5664116 https://news.ycombinator.com/item?id=5664116
robmil 13y ago
Also available is the brilliant underscore-cli[1], which brings the functional programming of _.js to the command line. There's an examples page, if you're struggling to see how it might be useful[2], but I find myself using 'extract', 'find', and 'pluck' on a daily basis. [1]: https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli [2]: https://github.com/ddopson/underscore-cli/blob/master/Examples.md https://github.com/ddopson/underscore-cli/blob/master/Exampl...
keenerd 13y ago
For a more unix-y version of this, check out Jshon. http://kmkeen.com/jshon/ http://kmkeen.com/jshon/
fabriceleal 13y ago
Some time ago I made this: https://github.com/fabriceleal/Jsoned https://github.com/fabriceleal/Jsoned using node.js. It allows you to use Javascript expressions using a map/filter/reduce paradigm to edit the Json stream, like this: echo "[1,2,3]" | jsoned -m "' _ * 2 '" which will print "[2,4,6]" you can even put the javascript code in files. Check the wiki (incomplete) for documentation: https://github.com/fabriceleal/Jsoned/wiki/Usage https://github.com/fabriceleal/Jsoned/wiki/Usage

132 points by gnosis 13y ago

deleted 13y ago

[deleted]

I've been using jq for a few months now while working with big GeoJSON blobs and it's a big help. The query language takes a bit of doing at first, but it's really pretty great once you learn it. curl -s http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.json http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.... | jq '.features[].properties.name'

dc2447 13y ago
Personally jgrep seems a better choice here curl foo/bar.json | jgrep -n -s 'property' Or even better curl foo/bar.json | jgrep 'status=500' -n -s 'path' http://jgrep.org/ http://jgrep.org/

laumars 13y ago

Completely unrelated, but I love the typeface he's used on his site. In fact the whole site is laid out really nicely (pretty, yet none of the aesthetics gets in the way of usability. a balance that most sites fail to strike). Anyhow, tangents aside, this looks a really handy tool. Thanks :)

revertts 13y ago

recs has this functionality, and _way_ more; worth giving a shot: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream

jcrites 13y ago
I'm a big fan of RecordStream. Everyone should definitely give it a look. However, is it true to say that recs has this functionality? RecordStream is great with sequences of relatively simple JSON structures, where you want to count, group, collate, etc. the sequence. It looks like jq is better able to handle large, complex arbitrary JSON objects, pulling out deeply nested fields and transforming them. For example, how would you implement jq's "recurse" in RecordStream? Now what would be really cool is using the two tools together :-)

doctorpangloss 13y ago

I like the concept, but how does it compare to using node + underscore? Isn't the whole idea of JSON that it exactly serializes most JavaScript objects, and the "right way" to do stuff with it is to just use it like a normal object in your program?

Xorlev 13y ago
The spirit of the tool is to make JSON easier to work with in shell flows. There's plenty of times where you'll get a deluge of JSON and need to work with it (for debugging or whatever). You may not have a program to use the JSON in.
mkehrt 13y ago
This assumes you're using JSON in Javascript. JSON is a pretty widely spread human-readable data format now outside of javascript, and it's nice to have good tools that aren't tied to a particular language.

nicpottier 13y ago

This is super cool, the language seems both powerful and easy enough to learn. Nicely done.

bsaul 13y ago

Considering the amount of json around, wouldn't it be time to settle on a standard for : - schema definition ( using something like typescript ? ) - data path ( similar to xpath ) - data transformation ( similar to xslt ) i'm pretty sure many people have done their things but i'm still waiting to see a tool like wsdl - based stub generators that would work on many json api. ps : i know that things like jsonpath and wsdl2 exist, but i really feel like those technologies haven't caught widespread adoption.

haberman 13y ago
I agree and I believe that the way to accomplish this is to use Protocol Buffer schemas. Protocol Buffers and JSON have an extremely similar data model. The parts of them that people actually use are effectively identical. The actual differences are minor things like: - JSON can have arrays of arrays; with Protocol Buffers you need a message in between. - Protocol Buffers can represent binary data directly; with JSON you have to base64 encode it into a string. - Protocol Buffers are more specific about the size of integer types (int64 and int32 are different); this allows for more efficient in-memory representations but doesn't matter so much on-the-wire. I think the two are a match made in heaven.
- bsaul 13y ago
  Yes, i also thought of protocol buffer right after sending that comment. Although i wonder : is there any way you could automatically generate a full service/model layer for, let's say angularjs or objective-c, based solely on protocol buffer "standard" ?
- nly 13y ago
  You're basically describing Piqi: http://piqi.org/ http://piqi.org/
  - haberman 13y ago
    Thanks for the reference, I hadn't seen that. I really believe, though, that the solution to interoperability is to use protocol buffer schemas directly; not to introduce yet another schema language.

guard-of-terra 13y ago

And for xml I highly recommend xmlstarlet. query/transform xml documents from the command line.

evaneykelen 13y ago

For readers who are unfamiliar with sed: don't dimiss jq as yet another json pretty printer. From the jq tutorial "Data in jq is represented as streams of JSON values - every jq expression runs for each value in its input stream, and can produce any number of values to its output stream.". I've played around with jq and it will definitely become a tool in my toolbox. Kudos to the author.

iso8859-1 13y ago

Previously on HN (141 days ago): https://news.ycombinator.com/item?id=4985250 https://news.ycombinator.com/item?id=4985250

quarterto 13y ago
Clearly, this is new to enough people that it got to the front page. Merely linking to a previous submission adds nothing here; if you thing the discussion there was informative (clearly not the case here), then say so.
- iso8859-1 13y ago
  Yeah, I think the previous discussion was informative, it has alternatives and links to ever older relevant posts. Let the karma decide what HN likes. Like you said yourself: https://news.ycombinator.com/item?id=5664116 https://news.ycombinator.com/item?id=5664116

robmil 13y ago

Also available is the brilliant underscore-cli[1], which brings the functional programming of _.js to the command line. There's an examples page, if you're struggling to see how it might be useful[2], but I find myself using 'extract', 'find', and 'pluck' on a daily basis. [1]: https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli [2]: https://github.com/ddopson/underscore-cli/blob/master/Examples.md https://github.com/ddopson/underscore-cli/blob/master/Exampl...

keenerd 13y ago

For a more unix-y version of this, check out Jshon. http://kmkeen.com/jshon/ http://kmkeen.com/jshon/

fabriceleal 13y ago

Some time ago I made this: https://github.com/fabriceleal/Jsoned https://github.com/fabriceleal/Jsoned using node.js. It allows you to use Javascript expressions using a map/filter/reduce paradigm to edit the Json stream, like this: echo "[1,2,3]" | jsoned -m "' _ * 2 '" which will print "[2,4,6]" you can even put the javascript code in files. Check the wiki (incomplete) for documentation: https://github.com/fabriceleal/Jsoned/wiki/Usage https://github.com/fabriceleal/Jsoned/wiki/Usage