14 ms·
Jq - a lightweight and flexible command-line JSON processor
- nekedorff 14y agoVery nice tool! I'm getting syntax errors from the Twitter examples though (and everytime I try to use underscores): <edit>Created an issue on Github</edit> curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true | jq '.results[] | {from_user, text, urls: [.entities.urls[].url]}' error: Invalid character .results[] | {from_user, text, urls: [.entities.urls[].url]} ^ error: syntax error, unexpected IDENT, expecting '}' .results[] | {from_user, text, urls: [.entities.urls[].url]} ^^^^ 2 compile errors % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5970 100 5970 0 0 11969 0 --:--:-- --:--:-- --:--:-- 23050
- philip_roberts 14y agoSame issue here.
- alexchamberlain 14y agoPost it here... https://github.com/stedolan/jq/issues https://github.com/stedolan/jq/issues
- stedolan 14y ago(author here) That bug was fixed a while ago, but I forgot to upload new binaries. Try again, they should be working now.
- alexchamberlain 14y ago> You can download a single binary, scp it to a far away machine, and expect it to work. Not if my machine is x86 and the server is SPARC. Cool utility though.
- evincarofautumn 14y agoGit clone and make, then. As long as you have a C compiler. And if you don’t, you probably aren’t using JSON.
- hnriot 14y agoActually I bet there are a ton of people using JSON that don't have a c compiler. The technologies are spaces by several decades and I would imagine the majority of JSON people are using much higher level languages than c. If I want to munge JSON I find it easier to do so in python, just convert it back to data structures and manipulate them and serialized it back to JSON.
- evincarofautumn 14y agoI would usually use some other language with a JSON library as well, but this utility is conveniently one-function, after Unix fashion. I like composability. Also, a C compiler is usually on hand whether you use it or not; even if none is, they’re not exactly hard to obtain.
- ville 14y ago"they're not exactly hard to obtain" Except, of course, in Mac OS X where you have to download (a couple of gigabytes) Xcode from Apple and install it to get GCC :)
- codebearded 14y ago<The technologies are spaces by several decades this intrigues me. while the widespread use of JSON as JSON is perhaps what differentiates it from what i'm about to describe i propose that the concept is an inevitable or obvious means of data representation/manipulation. As an arguably irrelevant anecdote to support this proposition I submit that prior to my personal introduction to the existence of JSON I was organizing data in strings and in txt files in a format nearly identical to basic JSON and writing simple functions to essentially do the same thing. I submit that I am not special or particularly clever and therefore the baller-coders that cometh before me certainly had to have their own types of crap like this. If i didn't have a tablet in class i probably would have had their names in my notes, but i digress. tl;dr - While the advent of JSON's incarnation and that of C are several decades apart the underlying concepts are not. tl;dr;dr - just realized your second sentence kinda says this.
- evincarofautumn 14y agoWe periodically have to munge JSON at work—will definitely be trying this out.
- Draiken 14y agoVery nice tool, congrats! This is something everyone ends up needing at least one time, and we always end up using a language specific tool. Nothing beats the command line piping
- deleted 14y ago[deleted]
- nsp 14y agoD. DQ,-"l
- metabrew 14y agoSee also, jsawk: https://github.com/micha/jsawk https://github.com/micha/jsawk
- tmcw 14y agoSee also, underscore-cli https://github.com/ddopson/underscore-cli https://github.com/ddopson/underscore-cli
- almog 14y agoGot the following compile error when running make (Lion + XCode 4.5.1): bison -W -d parser.y -v --report-file=parser.gen.info -o parser.gen.c bison: invalid option -- W Try `bison --help' for more information. make: *** [parser.gen.c] Error 1
- stedolan 14y ago(author here) I haven't tried to build on a mac in a while, bison there seems to support fewer options (must be an older version). For now, I've just checked in the autogenerated parser, so that bison won't have to run when you build master. git-pull and try again :)
- almog 14y agoWorks great now, thank you:)
- atdt 14y agoIt'd be great if the help text contained some examples rather than simply a pointer to the website, and if you added a man page. When you're in the middle of a terminal session, switching to a browser can be really disruptive. Also, if you tag releases with a version number and make them available as downloads on GitHub, you'll have an easier time getting Jq into Homebrew, the OS X package manager. I wrote a formula for you, but AFAIK you'll need to add in a real version number before submitting a pull request to Homebrew: https://gist.github.com/3928074 https://gist.github.com/3928074
- alexchamberlain 14y agoArch PKGBUILD would be much easier with a version number too...
- stedolan 14y agoThey exist now!
- littlemerman 14y agoThis is awesome. Thank you.
- deleted 14y ago[deleted]
- ruggeri 14y agoThanks; very nicely done. :-)
- xargs12 14y agoBuilt for Windows using cygwin, works a treat.
- MatthewPhillips 14y agoBuilds with MinGW as well, although I did have to manually edit a header file because I couldn't get sed to work properly.
- 055static 14y agoWow, this compiles fast. And the Flex and Bison files are there. This seems hackable. Nice work! My only questions are (they are always the same with any purported sed/awk "replacement"): 1. what was the problem you were trying to solve where sed and awk failed you, and 2. does this program operate line by line or does it read entire files into memory? I had to deal with some JSON a while ago and threw together some sed like this just so I could read it: sed ' s/,/&\ /g; /^$/d; s/^[{][^}]/\ &/g; /\"/s/,/<##eol##>/g; s/ *//; ' |tr '\012' '\040' |sed ' s/<##eol##>/\ /g; s/\[/\ &\ /; s/\]/\ &\ /; s/, /, /g; ' But any difficuly I have dealing with JSON I attribute to the pervasive use of JSON, not sed.
- stedolan 14y agoThanks! 1. I think your example shows sed/awk's failings with JSON data :) I don't want to write a JSON parser by hand every time I want to pull a field out of an object, and parsing recursive structures with regexes is never a good plan. 2. It reads JSON items from stdin into memory, one at a time. So if the input is a single giant JSON value, it all goes into memory, but if it's a series of whitespace-separated values they'll be processed sequentially. It's cat-friendly: if you do cat a | jq foo; cat b | jq foo then it's the same as doing cat a b | jq foo
- 055static 14y ago1. But those are general statements. Opinions. What I mean is give me a specific case. A specific example, a specific block of JSON and a specific task. Once I have that, then I can ask myself "Is this something I would ever need to do or that I have to do on a regular basis?" Sometimes I need to write one-off filters. There is just no getting around it. I have to choose a utility that gives maximum flexibility and is not too verbose; I don't like to type. Lots of people like Perl, and other similar scripting languages for writing one-off filters. But Perl, _out of the box_, is not a line-by-line filter. It's unlike sed/awk; it needs more memory. That brings us to #2. 2. If I understand correctly, jq is reading the entire JSON block into memory. This is what separates your program and so many other sed/awk "replacements" from sed and awk, the programs they purport to "replace". sed/awk don't read entire files into memory, they operate line-by-line and use a reasonably small buffer. Any sed/awk "replacement" would have to match that functionality. Given that sed/awk don't read in an entire structure (JSON, XML, etc.) before processing it, they are ideal for memory constrained environments. (As long as you don't overfill their buffers, which rarely happens in my experience.) Anyway, so far I like this program. Best JSON filter I've seen yet (because I can hack on the lexer and parser you provided). Well done.
- vhost- 14y agoI vote for a man page!
- codereflection 14y agohttp://stedolan.github.com/jq/manual/ http://stedolan.github.com/jq/manual/
- bryogenic 14y agoWhen doing the tasks this seems to be targeting, Jansson has worked well for me. http://www.digip.org/jansson/ http://www.digip.org/jansson/
- trustfundbaby 14y agoFYI if you have python installed (and most do) you can just do curl 'http://search.twitter.com/search.json?q=json&rpp=5&include_entities=true http://search.twitter.com/search.json?q=json&rpp=5&i... | python -mjson.tool
- draegtun 14y agoAlso see App::PipeFilter - https://metacpan.org/module/App::PipeFilter https://metacpan.org/module/App::PipeFilter The module is a framework/collection of shell pipelines which include JSON processing: curl -s 'http://api.duckduckgo.com/?q=poe&o=json' | jsonpath -o '$..Topics.*.FirstURL' -o '$..Topics.*.Text' | grep -i perl | jmap -i col0 -o url -i col1 -o title | json2yaml Output: title: Perl Object Environment, a library for event driven multitasking for the Perl programming language url: http://duckduckgo.com/Perl_Object_Environment Other structured data can be added to this module (Looks like Ingy maybe adding YAML - https://github.com/ingydotnet/app-pipefilter https://github.com/ingydotnet/app-pipefilter)
- ohaal 14y agoSee also jsontool, http://trentm.com/json/ http://trentm.com/json/