10 ms·

Show HN: jql – Easier jq Alternative with a Lispy Syntax Written in Go

217 points by cube2222 7y ago

cube2222 7y ago
Hey, author here. Christmas Eve, 22 pm, after having eaten with my family, I was browsing HN (because what else could you be doing then), and stumbled upon this comment: https://news.ycombinator.com/item?id=21860107 https://news.ycombinator.com/item?id=21860107 It inspired me to create an alternative to jq with more of a Lispy syntax, as I think the original is awesome but also fairly cryptic for anything more advanced than single field selection. Overall it was a fun few-day project, and was also very educational in terms of writing a parser in Go. (I used goyacc before in the sql parser of OctoSQL[1], however, that one is copied from vitess, so I’ve never built one from scratch, only customised an existing one. It’s really pleasant overall and the code is very simple, so I encourage you to take a look[2]. I'd love to hear any feedback, comments or potential improvements you can think of. [1]:https://github.com/cube2222/octosql https://github.com/cube2222/octosql [2]:https://github.com/cube2222/jql/tree/master/jql/parser https://github.com/cube2222/jql/tree/master/jql/parser
- daxelrod 7y ago
  Nicely done! I scratched a similar itch by creating Jowl[1], which uses JavaScript one-liners with the Lodash library to transform JSON. Like you, I wanted syntax that was more familiar to me than jq's, and like you, I wanted to use a language that was built for data structure transformation. However, I offloaded all of the actual hard work to an existing JavaScript runtime, and to an existing library for data structure transformation, meaning I could punt on the parser and language design; the hard parts that you've taken on. I have this bookmarked to read through in more detail later this week. [1]: https://github.com/daxelrod/jowl https://github.com/daxelrod/jowl
  - cube2222 7y ago
    Sounds great! I'll make sure to check out out! My approach was caused by the fact that I wanted to have a minimal language which only contains what really is necessary and is as regular as possible, to be very simple as a result.
- downerending 7y ago
  Looks great! As an old Lisper, I did get a wry chuckle out of the fact that you managed to avoid Lisp for a tool that uses Lispy syntax. :-)
  - cube2222 7y ago
    Thanks! Yup, I felt the irony myself creating it, as I actually love clojure! But I'm most comfortable with Go, and it results with a single binary for any of the major platforms with very quick startup time (as illustrated by the small benchmark), so those were huge advantages.
    - useragent86 7y ago
      I love the idea of a Lispy version of jq, but as a Lisper, the examples feel bewildering to me. The fact that (elem) takes an optional second argument, whose default value is 'identity, seems familiar, but then I try to understand this example, and it just doesn't make sense to me: (elem "countries" (elem (keys) (elem "name"))) I think part of the problem is, IIUC, this is constructing a pipeline, like (->) in Clojure, but is using function call syntax rather than pipeline syntax, and also mixing in a function call with (keys). I feel like it's a very non-Lispy language masquerading as Lisp by wearing parens. Instead, if you wrote it as real Lisp function calls, and provided a real pipeline macro, it would be much more Lispy and, I think, much easier to understand. For example, instead of: (elem "countries" (elem (keys) (elem "name"))) Either of these alternative syntaxes: (-> "countries" (key "name")) (key "name" ("countries")) Would return the same result: ["Poland", "United States", "Germany"] While being more concise and more Lispy. I also wouldn't mind using an anaphoric 'mapcar, something like: (-> "countries" (map-> (key "name")) Otherwise it may be unclear which functions implicitly map across entries and which don't. My two cents. Thanks for sharing your work.
      - cube2222 7y ago
        Thanks for the feedback! Have you checked out the type cheatsheet? (keys) is a function call! The function you pass as last to elem is kind of a continuation. It's a function which transforms the output before returning. All you're doing writing a jql query is composing one big function. With values in function call position, it's just that (keys) gets evaluated to a value (the keys of the map) and that is used to index the json. That's why keys wouldn't work, you need the parentheses to make it a function call. This may sound complex at first but I find it gets intuitive quickly. EDIT: Check out this comment thread on reddit as I think the commenter may have had similar problems: https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_query_processor_with_an_easier_lispy/fckwj7t/ https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_que...
- jsd1982 7y ago
  Looks good but there seems to be a missing explanation gap between switching from `(elem "countries")` and dropping the `elem`s to just `("countries")`. Why is `elem` able to be dropped? Is it the default function or something? I didn't follow that big jump in the readme.
  - cube2222 7y ago
    It's explained right below, I think I messed up the order there. Is this understandable? "You can see that elem is the most used function, and in fact it's what you'll usually be using when munging data, so there's a shortcut. If you put a value in function name position, it implicitly converts it to an elem." EDIT: Fixed the order.
  - aasasd 7y ago
    This feature can probably be thought of similarly to Clojure's map access via keywords: if you write `(:something map-var)`, you'll get the value of the field ‘:something’ in the ‘map-var’ variable. I.e. keywords can be ‘executed’, and such calls get translated to map access.
    - cube2222 7y ago
      Indeed, it just extends to values, and works recursively if it's an array or map.
- yamafaktory 7y ago
  Hi, awesome work! Love the idea of using S-expression for the query syntax! Funny fact: we share the same project name - even though the core language and the query syntax are different - see https://github.com/yamafaktory/jql https://github.com/yamafaktory/jql.
  - cube2222 7y ago
    Hey, looks really nice! I see we had similar goals of simplicity in mind, though with very different approaches. Sorry for the name duplication! I thought I'm sharing the name only with the jira API jql. EDIT: I'd love to see some benchmarks. I expect your jql to greatly outperform mine and jq, considering it's written in Rust!
    - yamafaktory 7y ago
      No worries regarding the naming, it's what makes open-source friendly! You can see some benches here e.g. https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-start-after_success-1 https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-st... but those actually more like regression tests that are triggered for pull requests to compare them against master (and I should use Github actions probably here now instead of Travis). Thanks for your reply!
- ball_of_lint 7y ago
  To me, pipe feels far more intuitive than callbacks as a way to write complex queries. Using it right away might make the first few examples much easier to read.
  - cube2222 7y ago
    I suppose to each their own. I like how the query matches the json structure well.
onionisafruit 7y ago
Nice project. I'm a heavy jq user and still find myself regularly tinkering around with queries on https://jqplay.org https://jqplay.org. It would be nice if there were a similar page for jql, even if it is just a local server.
- cube2222 7y ago
  I was thinking of an interactive terminal app, but that would be much more work. Though you can surely put a oneliner together which achieves the same effect using fzf, as per: https://paweldu.dev/posts/fzf-live-repl/ https://paweldu.dev/posts/fzf-live-repl/
  - preek 7y ago
    Here's an interactive implementation (in Emacs): https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq
- cube2222 7y ago
  Done: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'
xvilka 7y ago
There is more universal alternative to jq that supports also protobuf, msgpack, cbor, etc - rq[1]. [1] https://github.com/dflemstr/rq https://github.com/dflemstr/rq
- cube2222 7y ago
  Thanks, didn't know about it, looks interesting! Though from the tutorial I don't really understand how I can use it to transform data, other than converting between formats.
  - dan-robertson 7y ago
    Looks like query support was dropped.
mharju 7y ago
Looks cool! I'm using https://github.com/borkdude/babashka https://github.com/borkdude/babashka with with https://github.com/borkdude/jet https://github.com/borkdude/jet and a few aliases along with curl for the same effect. The syntax of jq is so hard to remember a familiar lisp always beats it :)
- cube2222 7y ago
  Nice, this looks great, and I love Clojure too, so I'll make sure to check it out!
EdSchouten 7y ago
So I guess we all know https://jsonnet.org https://jsonnet.org, right? It's advertised as a data templating language. Interestingly enough, you can also use it as a querying tool. The first four examples on the jql page can be translated to Jsonnet as follows: $ jsonnet -e "(import 'test.json').countries[0]" $ jsonnet -e "(import 'test.json').countries[0:2]" $ jsonnet -e "[x.name for x in (import 'test.json').countries]" $ jsonnet -e "std.objectFields((import 'test.json').countries[0])" With that in mind, I never bothered to learn how to use a tool like jq, rq, jql, etc.
- cube2222 7y ago
  This is actually the best alternative to jq for me I've seen so far! (other than jql obviously ;) ) Thanks for posting!
- hnlmorg 7y ago
  Shameless plug but I've also written a language (murex) for shell scripting which can be used for this too: open test.json -> [ countries ] open test.json -> [[ /countries/0 ]] open test.json -> [ countries ] -> [ 0 2 ] Single brackets only return the next level deep but can retrieve multiple items (and negative integers to count from the end of an array). Double brackets are to specify a path. So [[/foo/bar]] is literally the same as [foo]->[bar] Lastly the 4th example: open test.json -> [ countries ] -> foreach c { echo $c[name] } Unfortunately this one doesn't output it in JSON. If you needed that you could chain another command to reformat it: open test.json -> [ countries ] -> foreach c { echo $c[name] } -> cast str -> format json This works because murex can auto-convert between lists, JSON, YAML, CSV and a few other structured formats. However it's fair to say it is a lot more verbose than jsonnet and jql in that last example. Github repo: https://github.com/lmorg/murex https://github.com/lmorg/murex I've been using this as my primary shell for a few years now and, like yourself, I've never bothered to learn jq because of that. NB all of the above example are running from inside the murex interactive command line (like bash). If you wanted to run it like jq/jql then you'd need to do the same sort of thing as bash: murex -c 'open test.json -> [ countries ]'
- jaib8 7y ago
  I haven't used jsonnet before but is it possible to read via stdin? for eg to parse a json response from a rest endpoint in jq I would do something like: $ curl "..." | jq .abc
markandrewj 7y ago
This is fairly short talk, but it might interesting to some people. https://youtu.be/PS_9pyIASvQ https://youtu.be/PS_9pyIASvQ [Serious Programming with jq?! A practical and Purely Functional Programming Language!]
- cube2222 7y ago
  Thanks for posting! I'll make sure to watch it!
  - markandrewj 7y ago
    No problem. I hope it is helpful and interesting for you.
ldd 7y ago
only tangentially related, I made a VS Code extension to help out with jq development[0]. Also, jq supports modules. which is awesome! [0]https://marketplace.visualstudio.com/items?itemName=ldd-vs-code.jq-preview https://marketplace.visualstudio.com/items?itemName=ldd-vs-c...
dan-robertson 7y ago
I tend end to struggle with this sort of lispy syntax as I use a regular shell (rather than one in emacs) which makes balancing parens hard, and the syntax is otherwise verbose. If I’m using a query tool like this it will be interactively writing a one-liner so I want syntax that’s easy to get right (balancing parens by hand is hard), easy in the usual case (eg I think it should accept some way to use symbols to name fields in a record because typing and balancing double quotes all of the time is a waste of time) and easy to append to (I will run a query, press up, move to the end of the query, and modify. In this case I think moving to the end involves counting parens). Perhaps I would want some implicit pipe or threading at the top level (which could be turned off with some option for scripts I guess). Another helpful feature could be adding the magic closing paren: ] which will close all open parens to the top level. E.g. these two lines are equivalent: (foo (bar (baz) whizz)) (foo (bar (baz) whizz] A completely different (and less powerful) paradigm I like is one of match-format where the pattern looks a lot like what it’s trying to match. In this case most functions and fancy transformations are impossible. So you could write something like this to extract the list of countries: j '{countries: %%}' Or to extract each country’s name: j '{countries: [ .*, { name: %% }, .*]}' (Presumably you would have syntax for trying to match against each array element). (I also would make commas optional but I’m trying to make this syntax obvious). Or to list the keys of the 0th element: j '{countries: [ { %%: . }, .* ]}' Or to get name and population: j '{countries: [ .*, { name: %name, population: %pop }]}' The output might look like: { "name": "Poland", "population": "38000000" } ———————— An alternative query tool which I haven’t really seen before would be one which is interactive rather than repeatedly run: you pipe your json (or whatever) into it and it spins up some kind of (curses) gui shoeing you your data with a nice interface to interactively write your query and see its results. It could finish by outputting a command line to run the query non-interactively. The closest thing to this I know of is a small programming by example demo that came out of Microsoft research. The goal was to figure out how to extract some desired data from some big blob of json (or maybe html, I don’t remember). The user would see the json in some web gui and could click on the data they wanted and the app would try to figure out the “most obvious” query to get it. If they clicked on the country name, I think it would probably suggest a query for all the countries’ names (but maybe that would be the second option). I think it would also try to be clever about grouping, so if the user picked name and population it would hopefully figure out that they should come in pairs rather than being two independent lists that might not be the same length.
moondev 7y ago
Personally I like https://github.com/mikefarah/yq https://github.com/mikefarah/yq It also converts between yaml and json with ease for writing and reading
michaelmrose 7y ago
If this was lispy then (elem "countries" (elem 0) Would be an error. Expression evaluation ought to be inside out.
- cube2222 7y ago
  Why? It is, it's just that everything returns a function! The topmost function will in the end be called with the json as an argument. Check out these parts of the README: https://github.com/cube2222/jql#attention https://github.com/cube2222/jql#attention https://github.com/cube2222/jql#type-cheatsheet https://github.com/cube2222/jql#type-cheatsheet (elem 0) returns a function which takes a JSON array as input and returns its first element.
  - michaelmrose 7y ago
    Can you find many lisps or even any lisps that evaluate like that? A familiar syntax that actually works different from any lisp in existence may be more hindrance than help.
    - scarejunba 7y ago
      It's continuation-passing style.
    - kazinator 7y ago
      You can find Lisp macros which evaluate like that, more or less. Programming with combinators over functions is a thing also.
    - cube2222 7y ago
      It’s first class functions. Clojure has them. It's just that the query ends up having a type JSON -> JSON, not just JSON.
      - michaelmrose 7y ago
        Take (elem "countries" (elem 0)) Inside out elem 0 has to return a function since it doesn't have enough to do anything. I'm not sure why 0 couldn't be a key necessitating a different function to differentiate but whatever. Elem countries has to return json and one can imagine using the json in the first position like a map in clojure and taking a fn in following position. This would mean that on net it would be exactly the same as. (elem 0 (elem countries))
lcfcjs2 7y ago
Syntax is too confusing, I'll stick with jq thanks
cookiecaper 7y ago
To be frank the ascendance of jq has always seemed weird to me. There are many alternatives that felt much more natural, as this thread demonstrates. JSONata remains pretty obscure but it's been my favorite JSON query/mangling DSL for a while now: https://github.com/jsonata-js/jsonata https://github.com/jsonata-js/jsonata. jmespath is also quite natural, but I think it was hurt by reference implementation in Python, meaning it's quite slow. http://jmespath.org/ http://jmespath.org/
- rajangdavis 7y ago
  jq is simple to install, can handle large datasets (have used it to parse GB's of JSON data), and works well with bash/shell.
  - cookiecaper 7y ago
    Yeah -- I've ended up defaulting to jq just because of its ubiquity, but many of these other tools have similar attributes without the esoteric nature of the jq DSL. It'd be great if something more natural came along and actually started to displace it.
    - deleted 7y ago
      [deleted]
    - rajangdavis 7y ago
      I've never felt it was an unnatural API, but it can be hard to grok/remember. I was primarily using jq either to extract data from large JSON files or to concatenate a bunch of smaller JSON files into one, large file. I think it can get a little weird as far as error handling goes, but I feel it's been pretty approachable.
pololee 7y ago
I have used fx for a few days and enjoyed it. https://github.com/antonmedv/fx https://github.com/antonmedv/fx Highlights - interactive mode - Use full power of JavaScript. $ curl ... | fx '.filter(x => x.startsWith("a"))' - Access all lodash (or ramda, etc) methods by using .fxrc file. $ curl ... | fx '_.groupBy("commit.committer.name")' '_.mapValues(_.size)'
- cube2222 7y ago
  I've seen fx recently and it looks very interesting. jql is obviously not as feature-rich, but you can achieve interactivity with fzf! echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'
preek 7y ago
If you're into jq, here's an interface so you can have live feedback (in Emacs): https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq
- cube2222 7y ago
  You can probably also use an analogue to: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}' Something close to: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jq {q}'
spikepuppet 7y ago
I'm super excited to sink my teeth into this. I usually have to use jq in intense bursts and no matter what i end up back in the manual because it just get's so damn confusing. From a quick look on the repo, this seems so much simpler. Thanks!!!
- cube2222 7y ago
  I'm happy that you're so enthusiastic! Make sure to let me know what you think about it after having tried it out!
fiatjaf 7y ago
Shameless plug to my list of interesting `jq` stuff: https://github.com/fiatjaf/awesome-jq https://github.com/fiatjaf/awesome-jq It even features a complete rewrite of the actual jq in Go.
- bloopernova 7y ago
  Thank you very much for shamelessly plugging your list, it's great! I had no idea about jq-mode for Emacs, it even has org-babel support. So cool!
murat124 7y ago
echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}' Trivial note. echo instead of echo '' or echo "". $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $? $> 0
- cimnine 7y ago
  $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $? $> 0 Trivial note. `; echo $?` instead of `&& echo $? || echo $?` $> [[ "$(echo)" == "$(echo '')" ]]; echo $? $> 0
bloopernova 7y ago
Good Lawd'amercy. jq and lisp? In all seriousness, thank you for writing this. I've had trouble wrapping my small brain around some of jq's more advanced syntax, so being able to look at things from a different direction seems like a great idea to me.
- cube2222 7y ago
  Thanks! Let me know if it was understandable/confusing overall!
  - bloopernova 7y ago
    It was very well laid out and understandable. I found myself wishing for a parenthesis-matching capability in my zsh though! I also liked the informal/joking conversational style of the tutorial/walkthrough. It helped to put me at ease while learning something new. And having it written in Go, and with Go's simple installation method, makes me way more likely to use it in the future :) Nice job!
    - cube2222 7y ago
      Very glad to hear that! Feel free to open issues if you have problems or feature requests! There's an issue for at least adding trailing and starting parentheses automatically, so I'll try to add at least the trailing ones.
dfc 7y ago
Documentation Suggestion: I'm not familiar with fzf. So having the first example the README use fzf but not subsequent examples is a little confusing. Maybe start with a couple small examples and then use fzf towards the end?
- cube2222 7y ago
  I've added this after comments here asked for it, but I may move it towards a small "interactivity" chapter towards the end. It just evaluates jql continuously while you can write and edit your query.
aasasd 7y ago
Nice. However, a part of my problem with jq is that changing data structure also gets awkward pretty quick. So I hoped to hijack some small Lisp that's still likely to have a suite of functional tools―`map` and such. This would also rely on the fact that I can represent JSON's maps and arrays in most such Lisps. Afaict jql doesn't have that. So far I'm planning to just fire up Hy the next time I need anything in this vein. Since it's on Python, both JSON parsing/encoding and functional stuff should be there out of the box. Alternatively, I still can try using ClojureScript with Lumo. But I don't expect too fast startup with either of them, compared to, say, Fennel/Lua. BTW, there's a Lisp that translates to Go: https://github.com/jcla1/gisp https://github.com/jcla1/gisp. Though not too popular, and I've heard an opinion that it's toy-like in its capabilities.
- cube2222 7y ago
  Indeed, I know I didn't really solve your problem (and create your idea), as I decided to go in a different direction. Thanks for the inspiration though!

217 points by cube2222 7y ago

Hey, author here. Christmas Eve, 22 pm, after having eaten with my family, I was browsing HN (because what else could you be doing then), and stumbled upon this comment: https://news.ycombinator.com/item?id=21860107 https://news.ycombinator.com/item?id=21860107 It inspired me to create an alternative to jq with more of a Lispy syntax, as I think the original is awesome but also fairly cryptic for anything more advanced than single field selection. Overall it was a fun few-day project, and was also very educational in terms of writing a parser in Go. (I used goyacc before in the sql parser of OctoSQL[1], however, that one is copied from vitess, so I’ve never built one from scratch, only customised an existing one. It’s really pleasant overall and the code is very simple, so I encourage you to take a look[2]. I'd love to hear any feedback, comments or potential improvements you can think of. [1]:https://github.com/cube2222/octosql https://github.com/cube2222/octosql [2]:https://github.com/cube2222/jql/tree/master/jql/parser https://github.com/cube2222/jql/tree/master/jql/parser

daxelrod 7y ago
Nicely done! I scratched a similar itch by creating Jowl[1], which uses JavaScript one-liners with the Lodash library to transform JSON. Like you, I wanted syntax that was more familiar to me than jq's, and like you, I wanted to use a language that was built for data structure transformation. However, I offloaded all of the actual hard work to an existing JavaScript runtime, and to an existing library for data structure transformation, meaning I could punt on the parser and language design; the hard parts that you've taken on. I have this bookmarked to read through in more detail later this week. [1]: https://github.com/daxelrod/jowl https://github.com/daxelrod/jowl
- cube2222 7y ago
  Sounds great! I'll make sure to check out out! My approach was caused by the fact that I wanted to have a minimal language which only contains what really is necessary and is as regular as possible, to be very simple as a result.
downerending 7y ago
Looks great! As an old Lisper, I did get a wry chuckle out of the fact that you managed to avoid Lisp for a tool that uses Lispy syntax. :-)
- cube2222 7y ago
  Thanks! Yup, I felt the irony myself creating it, as I actually love clojure! But I'm most comfortable with Go, and it results with a single binary for any of the major platforms with very quick startup time (as illustrated by the small benchmark), so those were huge advantages.
  - useragent86 7y ago
    I love the idea of a Lispy version of jq, but as a Lisper, the examples feel bewildering to me. The fact that (elem) takes an optional second argument, whose default value is 'identity, seems familiar, but then I try to understand this example, and it just doesn't make sense to me: (elem "countries" (elem (keys) (elem "name"))) I think part of the problem is, IIUC, this is constructing a pipeline, like (->) in Clojure, but is using function call syntax rather than pipeline syntax, and also mixing in a function call with (keys). I feel like it's a very non-Lispy language masquerading as Lisp by wearing parens. Instead, if you wrote it as real Lisp function calls, and provided a real pipeline macro, it would be much more Lispy and, I think, much easier to understand. For example, instead of: (elem "countries" (elem (keys) (elem "name"))) Either of these alternative syntaxes: (-> "countries" (key "name")) (key "name" ("countries")) Would return the same result: ["Poland", "United States", "Germany"] While being more concise and more Lispy. I also wouldn't mind using an anaphoric 'mapcar, something like: (-> "countries" (map-> (key "name")) Otherwise it may be unclear which functions implicitly map across entries and which don't. My two cents. Thanks for sharing your work.
    - cube2222 7y ago
      Thanks for the feedback! Have you checked out the type cheatsheet? (keys) is a function call! The function you pass as last to elem is kind of a continuation. It's a function which transforms the output before returning. All you're doing writing a jql query is composing one big function. With values in function call position, it's just that (keys) gets evaluated to a value (the keys of the map) and that is used to index the json. That's why keys wouldn't work, you need the parentheses to make it a function call. This may sound complex at first but I find it gets intuitive quickly. EDIT: Check out this comment thread on reddit as I think the commenter may have had similar problems: https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_query_processor_with_an_easier_lispy/fckwj7t/ https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_que...
jsd1982 7y ago
Looks good but there seems to be a missing explanation gap between switching from `(elem "countries")` and dropping the `elem`s to just `("countries")`. Why is `elem` able to be dropped? Is it the default function or something? I didn't follow that big jump in the readme.
- cube2222 7y ago
  It's explained right below, I think I messed up the order there. Is this understandable? "You can see that elem is the most used function, and in fact it's what you'll usually be using when munging data, so there's a shortcut. If you put a value in function name position, it implicitly converts it to an elem." EDIT: Fixed the order.
- aasasd 7y ago
  This feature can probably be thought of similarly to Clojure's map access via keywords: if you write `(:something map-var)`, you'll get the value of the field ‘:something’ in the ‘map-var’ variable. I.e. keywords can be ‘executed’, and such calls get translated to map access.
  - cube2222 7y ago
    Indeed, it just extends to values, and works recursively if it's an array or map.
yamafaktory 7y ago
Hi, awesome work! Love the idea of using S-expression for the query syntax! Funny fact: we share the same project name - even though the core language and the query syntax are different - see https://github.com/yamafaktory/jql https://github.com/yamafaktory/jql.
- cube2222 7y ago
  Hey, looks really nice! I see we had similar goals of simplicity in mind, though with very different approaches. Sorry for the name duplication! I thought I'm sharing the name only with the jira API jql. EDIT: I'd love to see some benchmarks. I expect your jql to greatly outperform mine and jq, considering it's written in Rust!
  - yamafaktory 7y ago
    No worries regarding the naming, it's what makes open-source friendly! You can see some benches here e.g. https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-start-after_success-1 https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-st... but those actually more like regression tests that are triggered for pull requests to compare them against master (and I should use Github actions probably here now instead of Travis). Thanks for your reply!
ball_of_lint 7y ago
To me, pipe feels far more intuitive than callbacks as a way to write complex queries. Using it right away might make the first few examples much easier to read.
- cube2222 7y ago
  I suppose to each their own. I like how the query matches the json structure well.

onionisafruit 7y ago

Nice project. I'm a heavy jq user and still find myself regularly tinkering around with queries on https://jqplay.org https://jqplay.org. It would be nice if there were a similar page for jql, even if it is just a local server.

cube2222 7y ago
I was thinking of an interactive terminal app, but that would be much more work. Though you can surely put a oneliner together which achieves the same effect using fzf, as per: https://paweldu.dev/posts/fzf-live-repl/ https://paweldu.dev/posts/fzf-live-repl/
- preek 7y ago
  Here's an interactive implementation (in Emacs): https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq
cube2222 7y ago
Done: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'

xvilka 7y ago

There is more universal alternative to jq that supports also protobuf, msgpack, cbor, etc - rq[1]. [1] https://github.com/dflemstr/rq https://github.com/dflemstr/rq

cube2222 7y ago
Thanks, didn't know about it, looks interesting! Though from the tutorial I don't really understand how I can use it to transform data, other than converting between formats.
- dan-robertson 7y ago
  Looks like query support was dropped.

mharju 7y ago

Looks cool! I'm using https://github.com/borkdude/babashka https://github.com/borkdude/babashka with with https://github.com/borkdude/jet https://github.com/borkdude/jet and a few aliases along with curl for the same effect. The syntax of jq is so hard to remember a familiar lisp always beats it :)

cube2222 7y ago
Nice, this looks great, and I love Clojure too, so I'll make sure to check it out!

EdSchouten 7y ago

So I guess we all know https://jsonnet.org https://jsonnet.org, right? It's advertised as a data templating language. Interestingly enough, you can also use it as a querying tool. The first four examples on the jql page can be translated to Jsonnet as follows: $ jsonnet -e "(import 'test.json').countries[0]" $ jsonnet -e "(import 'test.json').countries[0:2]" $ jsonnet -e "[x.name for x in (import 'test.json').countries]" $ jsonnet -e "std.objectFields((import 'test.json').countries[0])" With that in mind, I never bothered to learn how to use a tool like jq, rq, jql, etc.

cube2222 7y ago
This is actually the best alternative to jq for me I've seen so far! (other than jql obviously ;) ) Thanks for posting!
hnlmorg 7y ago
Shameless plug but I've also written a language (murex) for shell scripting which can be used for this too: open test.json -> [ countries ] open test.json -> [[ /countries/0 ]] open test.json -> [ countries ] -> [ 0 2 ] Single brackets only return the next level deep but can retrieve multiple items (and negative integers to count from the end of an array). Double brackets are to specify a path. So [[/foo/bar]] is literally the same as [foo]->[bar] Lastly the 4th example: open test.json -> [ countries ] -> foreach c { echo $c[name] } Unfortunately this one doesn't output it in JSON. If you needed that you could chain another command to reformat it: open test.json -> [ countries ] -> foreach c { echo $c[name] } -> cast str -> format json This works because murex can auto-convert between lists, JSON, YAML, CSV and a few other structured formats. However it's fair to say it is a lot more verbose than jsonnet and jql in that last example. Github repo: https://github.com/lmorg/murex https://github.com/lmorg/murex I've been using this as my primary shell for a few years now and, like yourself, I've never bothered to learn jq because of that. NB all of the above example are running from inside the murex interactive command line (like bash). If you wanted to run it like jq/jql then you'd need to do the same sort of thing as bash: murex -c 'open test.json -> [ countries ]'
jaib8 7y ago
I haven't used jsonnet before but is it possible to read via stdin? for eg to parse a json response from a rest endpoint in jq I would do something like: $ curl "..." | jq .abc

markandrewj 7y ago

This is fairly short talk, but it might interesting to some people. https://youtu.be/PS_9pyIASvQ https://youtu.be/PS_9pyIASvQ [Serious Programming with jq?! A practical and Purely Functional Programming Language!]

cube2222 7y ago
Thanks for posting! I'll make sure to watch it!
- markandrewj 7y ago
  No problem. I hope it is helpful and interesting for you.

ldd 7y ago

only tangentially related, I made a VS Code extension to help out with jq development[0]. Also, jq supports modules. which is awesome! [0]https://marketplace.visualstudio.com/items?itemName=ldd-vs-code.jq-preview https://marketplace.visualstudio.com/items?itemName=ldd-vs-c...

dan-robertson 7y ago

I tend end to struggle with this sort of lispy syntax as I use a regular shell (rather than one in emacs) which makes balancing parens hard, and the syntax is otherwise verbose. If I’m using a query tool like this it will be interactively writing a one-liner so I want syntax that’s easy to get right (balancing parens by hand is hard), easy in the usual case (eg I think it should accept some way to use symbols to name fields in a record because typing and balancing double quotes all of the time is a waste of time) and easy to append to (I will run a query, press up, move to the end of the query, and modify. In this case I think moving to the end involves counting parens). Perhaps I would want some implicit pipe or threading at the top level (which could be turned off with some option for scripts I guess). Another helpful feature could be adding the magic closing paren: ] which will close all open parens to the top level. E.g. these two lines are equivalent: (foo (bar (baz) whizz)) (foo (bar (baz) whizz] A completely different (and less powerful) paradigm I like is one of match-format where the pattern looks a lot like what it’s trying to match. In this case most functions and fancy transformations are impossible. So you could write something like this to extract the list of countries: j '{countries: %%}' Or to extract each country’s name: j '{countries: [ .*, { name: %% }, .*]}' (Presumably you would have syntax for trying to match against each array element). (I also would make commas optional but I’m trying to make this syntax obvious). Or to list the keys of the 0th element: j '{countries: [ { %%: . }, .* ]}' Or to get name and population: j '{countries: [ .*, { name: %name, population: %pop }]}' The output might look like: { "name": "Poland", "population": "38000000" } ———————— An alternative query tool which I haven’t really seen before would be one which is interactive rather than repeatedly run: you pipe your json (or whatever) into it and it spins up some kind of (curses) gui shoeing you your data with a nice interface to interactively write your query and see its results. It could finish by outputting a command line to run the query non-interactively. The closest thing to this I know of is a small programming by example demo that came out of Microsoft research. The goal was to figure out how to extract some desired data from some big blob of json (or maybe html, I don’t remember). The user would see the json in some web gui and could click on the data they wanted and the app would try to figure out the “most obvious” query to get it. If they clicked on the country name, I think it would probably suggest a query for all the countries’ names (but maybe that would be the second option). I think it would also try to be clever about grouping, so if the user picked name and population it would hopefully figure out that they should come in pairs rather than being two independent lists that might not be the same length.

moondev 7y ago

Personally I like https://github.com/mikefarah/yq https://github.com/mikefarah/yq It also converts between yaml and json with ease for writing and reading

michaelmrose 7y ago

If this was lispy then (elem "countries" (elem 0) Would be an error. Expression evaluation ought to be inside out.

cube2222 7y ago
Why? It is, it's just that everything returns a function! The topmost function will in the end be called with the json as an argument. Check out these parts of the README: https://github.com/cube2222/jql#attention https://github.com/cube2222/jql#attention https://github.com/cube2222/jql#type-cheatsheet https://github.com/cube2222/jql#type-cheatsheet (elem 0) returns a function which takes a JSON array as input and returns its first element.
- michaelmrose 7y ago
  Can you find many lisps or even any lisps that evaluate like that? A familiar syntax that actually works different from any lisp in existence may be more hindrance than help.
  - scarejunba 7y ago
    It's continuation-passing style.
  - kazinator 7y ago
    You can find Lisp macros which evaluate like that, more or less. Programming with combinators over functions is a thing also.
  - cube2222 7y ago
    It’s first class functions. Clojure has them. It's just that the query ends up having a type JSON -> JSON, not just JSON.
    - michaelmrose 7y ago
      Take (elem "countries" (elem 0)) Inside out elem 0 has to return a function since it doesn't have enough to do anything. I'm not sure why 0 couldn't be a key necessitating a different function to differentiate but whatever. Elem countries has to return json and one can imagine using the json in the first position like a map in clojure and taking a fn in following position. This would mean that on net it would be exactly the same as. (elem 0 (elem countries))

lcfcjs2 7y ago

Syntax is too confusing, I'll stick with jq thanks

cookiecaper 7y ago

To be frank the ascendance of jq has always seemed weird to me. There are many alternatives that felt much more natural, as this thread demonstrates. JSONata remains pretty obscure but it's been my favorite JSON query/mangling DSL for a while now: https://github.com/jsonata-js/jsonata https://github.com/jsonata-js/jsonata. jmespath is also quite natural, but I think it was hurt by reference implementation in Python, meaning it's quite slow. http://jmespath.org/ http://jmespath.org/

rajangdavis 7y ago
jq is simple to install, can handle large datasets (have used it to parse GB's of JSON data), and works well with bash/shell.
- cookiecaper 7y ago
  Yeah -- I've ended up defaulting to jq just because of its ubiquity, but many of these other tools have similar attributes without the esoteric nature of the jq DSL. It'd be great if something more natural came along and actually started to displace it.
  - deleted 7y ago
    [deleted]
  - rajangdavis 7y ago
    I've never felt it was an unnatural API, but it can be hard to grok/remember. I was primarily using jq either to extract data from large JSON files or to concatenate a bunch of smaller JSON files into one, large file. I think it can get a little weird as far as error handling goes, but I feel it's been pretty approachable.

pololee 7y ago

I have used fx for a few days and enjoyed it. https://github.com/antonmedv/fx https://github.com/antonmedv/fx Highlights - interactive mode - Use full power of JavaScript. $ curl ... | fx '.filter(x => x.startsWith("a"))' - Access all lodash (or ramda, etc) methods by using .fxrc file. $ curl ... | fx '_.groupBy("commit.committer.name")' '_.mapValues(_.size)'

cube2222 7y ago
I've seen fx recently and it looks very interesting. jql is obviously not as feature-rich, but you can achieve interactivity with fzf! echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'

preek 7y ago

If you're into jq, here's an interface so you can have live feedback (in Emacs): https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq

cube2222 7y ago
You can probably also use an analogue to: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}' Something close to: echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jq {q}'

spikepuppet 7y ago

I'm super excited to sink my teeth into this. I usually have to use jq in intense bursts and no matter what i end up back in the manual because it just get's so damn confusing. From a quick look on the repo, this seems so much simpler. Thanks!!!

cube2222 7y ago
I'm happy that you're so enthusiastic! Make sure to let me know what you think about it after having tried it out!

fiatjaf 7y ago

Shameless plug to my list of interesting `jq` stuff: https://github.com/fiatjaf/awesome-jq https://github.com/fiatjaf/awesome-jq It even features a complete rewrite of the actual jq in Go.

bloopernova 7y ago
Thank you very much for shamelessly plugging your list, it's great! I had no idea about jq-mode for Emacs, it even has org-babel support. So cool!

murat124 7y ago

echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}' Trivial note. echo instead of echo '' or echo "". $> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $? $> 0

cimnine 7y ago
$> [[ "$(echo)" == "$(echo '')" ]] && echo $? || echo $? $> 0 Trivial note. `; echo $?` instead of `&& echo $? || echo $?` $> [[ "$(echo)" == "$(echo '')" ]]; echo $? $> 0

bloopernova 7y ago

Good Lawd'amercy. jq and lisp? In all seriousness, thank you for writing this. I've had trouble wrapping my small brain around some of jq's more advanced syntax, so being able to look at things from a different direction seems like a great idea to me.

cube2222 7y ago
Thanks! Let me know if it was understandable/confusing overall!
- bloopernova 7y ago
  It was very well laid out and understandable. I found myself wishing for a parenthesis-matching capability in my zsh though! I also liked the informal/joking conversational style of the tutorial/walkthrough. It helped to put me at ease while learning something new. And having it written in Go, and with Go's simple installation method, makes me way more likely to use it in the future :) Nice job!
  - cube2222 7y ago
    Very glad to hear that! Feel free to open issues if you have problems or feature requests! There's an issue for at least adding trailing and starting parentheses automatically, so I'll try to add at least the trailing ones.

dfc 7y ago

Documentation Suggestion: I'm not familiar with fzf. So having the first example the README use fzf but not subsequent examples is a little confusing. Maybe start with a couple small examples and then use fzf towards the end?

cube2222 7y ago
I've added this after comments here asked for it, but I may move it towards a small "interactivity" chapter towards the end. It just evaluates jql continuously while you can write and edit your query.

aasasd 7y ago

Nice. However, a part of my problem with jq is that changing data structure also gets awkward pretty quick. So I hoped to hijack some small Lisp that's still likely to have a suite of functional tools―`map` and such. This would also rely on the fact that I can represent JSON's maps and arrays in most such Lisps. Afaict jql doesn't have that. So far I'm planning to just fire up Hy the next time I need anything in this vein. Since it's on Python, both JSON parsing/encoding and functional stuff should be there out of the box. Alternatively, I still can try using ClojureScript with Lumo. But I don't expect too fast startup with either of them, compared to, say, Fennel/Lua. BTW, there's a Lisp that translates to Go: https://github.com/jcla1/gisp https://github.com/jcla1/gisp. Though not too popular, and I've heard an opinion that it's toy-like in its capabilities.

cube2222 7y ago
Indeed, I know I didn't really solve your problem (and create your idea), as I decided to go in a different direction. Thanks for the inspiration though!