5 ms·

Jshon: A command line program to parse, read and create JSON

39 points by akheron 16y ago

forgotusername 16y ago
Note for simple uses, Python 2.6+ comes with the "json.tool" module (also part of simplejson if you're <= 2.5), which reads JSON on stdin (or argv[1]) before pretty printing it. $ type json json is aliased to `python -m json.tool' $ curl -s twitter.com/users/BarackObama.json|json|grep '"url"' "url": "http://www.barackobama.com",
- keenerd 16y ago
  Please never use grep on json. If you are going to use python as a heavyweight json_reformat, at least have the decency to use all of python's excellent json lib.
timrobinson 16y ago
Jsawk seems more intuitive, as you get to write JavaScript instead of combining command-line flags: https://github.com/micha/jsawk https://github.com/micha/jsawk
- keenerd 16y ago
  Found that, decided I did not want Spidermonkey on my server running untrusted user generated JS. Also, it seemed ironic to have Spidermonkey be a dependency for my "lightweight" shell scripts.
  - akheron 16y ago
    This is what I thought too. JSON is lightweight and so should be the tools for using it.
arethuza 16y ago
I've been using CouchDB a lot over the last six months or so and I've been seeing a lot of JSON on the command line - either through curl or using my own application-specific command line utilities. I must admit that I rather like working with JSON on the command line - it is easy on the eye but allows far more powerful structures than the usual Unix approach of lines with delimited fields (which I've been happily using for 20+ years). Using JSON to extend command line processing looks to me like it is a nice step forward. Things like Microsoft's PowerShell are perhaps conceptually more elegant (where you pass real live objects about) but the practical reality always seems very fussy and tied to a particular programming platform. Other approaches (e.g. passing XML between Unix-style command line programs) isn't very appealing due to the general ugliness and verbosity of most XML.
justincormack 16y ago
There have been some attempts to make a line oriented XML format so that shell tools work; doing the same thing with JSON is possible too; there are a few ways to do it. Have been meaning to give it a go at some point. In my view it makes more sense than building a new set of tools, especially as the Jshon ones do not really seem to compose properly. References for XML: http://www.lbreyer.com/unix_xml-1.html http://www.lbreyer.com/unix_xml-1.html http://www.ofb.net/~egnor/xml2/ http://www.ofb.net/~egnor/xml2/ http://gnosis.cx/publish/programming/xml_matters_17.html http://gnosis.cx/publish/programming/xml_matters_17.html
- keenerd 16y ago
  > the Jshon ones do not really seem to compose properly. Could you elaborate on that a bit?
  - justincormack 16y ago
    Most of the operations do not map any json to json, so you cant pipe them. The whole design principle of the unix command line tools is that they all map lines to lines so anything is composable. So for json I would want a structure match corresponding to grep, filter and map operations, merges etc each of which maps valid json to valid json.
    - keenerd 16y ago
      Almost half* of the commands map json (stdin) to json (stdout). Extract, modify and insert all take and spit out json. Originally (when it only supported single commands) this is how you had to do everything. But that much piping is a real drag. Anyway, show me a shell json util that is more composable or at least provide a suggestion to improving it :-) Grep/filter/map are all stuff you do in the shell, after using Jshon to extract the data from the hard-to-parse-safely json. *The remaining commands are concessions to make line-oriented shell processing easier.
- reirob 16y ago
  I have been using XML Starlet (http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/) for many years in combination with Unix Shell. It is a very nice command line tool allowing to extract data from XML documents (using XPath expressions), to edit XML documents and even to convert XML to PYX format. The only problem it has so far is with big XML files as it loads them in the memory. But there is work going on to allow extract data without loading whole XML file in RAM.
VMG 16y ago
Great tool, was thinking about writing something like that myself. I especially wanted a tool that would print the value of a specific key to STDOUT with a syntax like echo "{ 'a': {'b': {'c": 0 } } }" | json-tool "a.b.c" # echos 0 Maybe your tool does that but I missed it
- keenerd 16y ago
  Glad you like it. I'll be adding a path-like syntax next. This was just a little one-day hackup to learn getopts. You can do similar with -e a -e b -e c. Or just make your own wrapper with jshon $(echo ".a.b.c" | sed 's/./ -e/g')
  - keenerd 16y ago
    jshon $(sed 's/\./ -e/g' <<< ".a.b.c")
SudarshanP 16y ago
http://blogs.fluidinfo.com/terry/2010/11/25/jsongrep-py-python-for-extracting-pieces-of-json-objects/#comment-148295183 http://blogs.fluidinfo.com/terry/2010/11/25/jsongrep-py-pyth... may be interesting...
fictorial 16y ago
> In Aurphan there is an alarming bit of awk: Senator, I know awk, and that is not an alarming bit of awk.
- minimax 16y ago
  It also looks like he swaps out some fairly readable awk for some arguably less readable shell script.
akv 16y ago
It is a bit long, but you can also use Python for parsing the json files from command line: echo '{"a": "hello", "b": "world"}' | python -c"import sys,json;j = json.loads(sys.stdin.read());print j['a']"
- webXL 16y ago
  or in 2.6: echo '{"a": "hello", "b": "world"}' | python -mjson.tool
bguthrie 16y ago
"Bugs: Numerous!" Truth in advertising. I love it.

39 points by akheron 16y ago

Note for simple uses, Python 2.6+ comes with the "json.tool" module (also part of simplejson if you're <= 2.5), which reads JSON on stdin (or argv[1]) before pretty printing it. $ type json json is aliased to `python -m json.tool' $ curl -s twitter.com/users/BarackObama.json|json|grep '"url"' "url": "http://www.barackobama.com",

keenerd 16y ago
Please never use grep on json. If you are going to use python as a heavyweight json_reformat, at least have the decency to use all of python's excellent json lib.

timrobinson 16y ago

Jsawk seems more intuitive, as you get to write JavaScript instead of combining command-line flags: https://github.com/micha/jsawk https://github.com/micha/jsawk

keenerd 16y ago
Found that, decided I did not want Spidermonkey on my server running untrusted user generated JS. Also, it seemed ironic to have Spidermonkey be a dependency for my "lightweight" shell scripts.
- akheron 16y ago
  This is what I thought too. JSON is lightweight and so should be the tools for using it.

arethuza 16y ago

I've been using CouchDB a lot over the last six months or so and I've been seeing a lot of JSON on the command line - either through curl or using my own application-specific command line utilities. I must admit that I rather like working with JSON on the command line - it is easy on the eye but allows far more powerful structures than the usual Unix approach of lines with delimited fields (which I've been happily using for 20+ years). Using JSON to extend command line processing looks to me like it is a nice step forward. Things like Microsoft's PowerShell are perhaps conceptually more elegant (where you pass real live objects about) but the practical reality always seems very fussy and tied to a particular programming platform. Other approaches (e.g. passing XML between Unix-style command line programs) isn't very appealing due to the general ugliness and verbosity of most XML.

justincormack 16y ago

There have been some attempts to make a line oriented XML format so that shell tools work; doing the same thing with JSON is possible too; there are a few ways to do it. Have been meaning to give it a go at some point. In my view it makes more sense than building a new set of tools, especially as the Jshon ones do not really seem to compose properly. References for XML: http://www.lbreyer.com/unix_xml-1.html http://www.lbreyer.com/unix_xml-1.html http://www.ofb.net/~egnor/xml2/ http://www.ofb.net/~egnor/xml2/ http://gnosis.cx/publish/programming/xml_matters_17.html http://gnosis.cx/publish/programming/xml_matters_17.html

keenerd 16y ago
> the Jshon ones do not really seem to compose properly. Could you elaborate on that a bit?
- justincormack 16y ago
  Most of the operations do not map any json to json, so you cant pipe them. The whole design principle of the unix command line tools is that they all map lines to lines so anything is composable. So for json I would want a structure match corresponding to grep, filter and map operations, merges etc each of which maps valid json to valid json.
  - keenerd 16y ago
    Almost half* of the commands map json (stdin) to json (stdout). Extract, modify and insert all take and spit out json. Originally (when it only supported single commands) this is how you had to do everything. But that much piping is a real drag. Anyway, show me a shell json util that is more composable or at least provide a suggestion to improving it :-) Grep/filter/map are all stuff you do in the shell, after using Jshon to extract the data from the hard-to-parse-safely json. *The remaining commands are concessions to make line-oriented shell processing easier.
reirob 16y ago
I have been using XML Starlet (http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/) for many years in combination with Unix Shell. It is a very nice command line tool allowing to extract data from XML documents (using XPath expressions), to edit XML documents and even to convert XML to PYX format. The only problem it has so far is with big XML files as it loads them in the memory. But there is work going on to allow extract data without loading whole XML file in RAM.

VMG 16y ago

Great tool, was thinking about writing something like that myself. I especially wanted a tool that would print the value of a specific key to STDOUT with a syntax like echo "{ 'a': {'b': {'c": 0 } } }" | json-tool "a.b.c" # echos 0 Maybe your tool does that but I missed it

keenerd 16y ago
Glad you like it. I'll be adding a path-like syntax next. This was just a little one-day hackup to learn getopts. You can do similar with -e a -e b -e c. Or just make your own wrapper with jshon $(echo ".a.b.c" | sed 's/./ -e/g')
- keenerd 16y ago
  jshon $(sed 's/\./ -e/g' <<< ".a.b.c")

SudarshanP 16y ago

http://blogs.fluidinfo.com/terry/2010/11/25/jsongrep-py-python-for-extracting-pieces-of-json-objects/#comment-148295183 http://blogs.fluidinfo.com/terry/2010/11/25/jsongrep-py-pyth... may be interesting...

fictorial 16y ago

> In Aurphan there is an alarming bit of awk: Senator, I know awk, and that is not an alarming bit of awk.

minimax 16y ago
It also looks like he swaps out some fairly readable awk for some arguably less readable shell script.

akv 16y ago

It is a bit long, but you can also use Python for parsing the json files from command line: echo '{"a": "hello", "b": "world"}' | python -c"import sys,json;j = json.loads(sys.stdin.read());print j['a']"

webXL 16y ago
or in 2.6: echo '{"a": "hello", "b": "world"}' | python -mjson.tool

bguthrie 16y ago

"Bugs: Numerous!" Truth in advertising. I love it.