4 ms·

Hey, author here. Christmas Eve, 22 pm, after having eaten with my family, I was browsing HN (because what else could you be doing then), and stumbled upon this

Hey, author here. Christmas Eve, 22 pm, after having eaten with my family, I was browsing HN (because what else could you be doing then), and stumbled upon this comment: https://news.ycombinator.com/item?id=21860107 https://news.ycombinator.com/item?id=21860107 It inspired me to create an alternative to jq with more of a Lispy syntax, as I think the original is awesome but also fairly cryptic for anything more advanced than single field selection. Overall it was a fun few-day project, and was also very educational in terms of writing a parser in Go. (I used goyacc before in the sql parser of OctoSQL[1], however, that one is copied from vitess, so I’ve never built one from scratch, only customised an existing one. It’s really pleasant overall and the code is very simple, so I encourage you to take a look[2]. I'd love to hear any feedback, comments or potential improvements you can think of. [1]:https://github.com/cube2222/octosql https://github.com/cube2222/octosql [2]:https://github.com/cube2222/jql/tree/master/jql/parser https://github.com/cube2222/jql/tree/master/jql/parser

daxelrod 7y ago
Nicely done! I scratched a similar itch by creating Jowl[1], which uses JavaScript one-liners with the Lodash library to transform JSON. Like you, I wanted syntax that was more familiar to me than jq's, and like you, I wanted to use a language that was built for data structure transformation. However, I offloaded all of the actual hard work to an existing JavaScript runtime, and to an existing library for data structure transformation, meaning I could punt on the parser and language design; the hard parts that you've taken on. I have this bookmarked to read through in more detail later this week. [1]: https://github.com/daxelrod/jowl https://github.com/daxelrod/jowl
- cube2222 7y ago
  Sounds great! I'll make sure to check out out! My approach was caused by the fact that I wanted to have a minimal language which only contains what really is necessary and is as regular as possible, to be very simple as a result.
downerending 7y ago
Looks great! As an old Lisper, I did get a wry chuckle out of the fact that you managed to avoid Lisp for a tool that uses Lispy syntax. :-)
- cube2222 7y ago
  Thanks! Yup, I felt the irony myself creating it, as I actually love clojure! But I'm most comfortable with Go, and it results with a single binary for any of the major platforms with very quick startup time (as illustrated by the small benchmark), so those were huge advantages.
  - useragent86 7y ago
    I love the idea of a Lispy version of jq, but as a Lisper, the examples feel bewildering to me. The fact that (elem) takes an optional second argument, whose default value is 'identity, seems familiar, but then I try to understand this example, and it just doesn't make sense to me: (elem "countries" (elem (keys) (elem "name"))) I think part of the problem is, IIUC, this is constructing a pipeline, like (->) in Clojure, but is using function call syntax rather than pipeline syntax, and also mixing in a function call with (keys). I feel like it's a very non-Lispy language masquerading as Lisp by wearing parens. Instead, if you wrote it as real Lisp function calls, and provided a real pipeline macro, it would be much more Lispy and, I think, much easier to understand. For example, instead of: (elem "countries" (elem (keys) (elem "name"))) Either of these alternative syntaxes: (-> "countries" (key "name")) (key "name" ("countries")) Would return the same result: ["Poland", "United States", "Germany"] While being more concise and more Lispy. I also wouldn't mind using an anaphoric 'mapcar, something like: (-> "countries" (map-> (key "name")) Otherwise it may be unclear which functions implicitly map across entries and which don't. My two cents. Thanks for sharing your work.
    - cube2222 7y ago
      Thanks for the feedback! Have you checked out the type cheatsheet? (keys) is a function call! The function you pass as last to elem is kind of a continuation. It's a function which transforms the output before returning. All you're doing writing a jql query is composing one big function. With values in function call position, it's just that (keys) gets evaluated to a value (the keys of the map) and that is used to index the json. That's why keys wouldn't work, you need the parentheses to make it a function call. This may sound complex at first but I find it gets intuitive quickly. EDIT: Check out this comment thread on reddit as I think the commenter may have had similar problems: https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_query_processor_with_an_easier_lispy/fckwj7t/ https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_que...
jsd1982 7y ago
Looks good but there seems to be a missing explanation gap between switching from `(elem "countries")` and dropping the `elem`s to just `("countries")`. Why is `elem` able to be dropped? Is it the default function or something? I didn't follow that big jump in the readme.
- cube2222 7y ago
  It's explained right below, I think I messed up the order there. Is this understandable? "You can see that elem is the most used function, and in fact it's what you'll usually be using when munging data, so there's a shortcut. If you put a value in function name position, it implicitly converts it to an elem." EDIT: Fixed the order.
- aasasd 7y ago
  This feature can probably be thought of similarly to Clojure's map access via keywords: if you write `(:something map-var)`, you'll get the value of the field ‘:something’ in the ‘map-var’ variable. I.e. keywords can be ‘executed’, and such calls get translated to map access.
  - cube2222 7y ago
    Indeed, it just extends to values, and works recursively if it's an array or map.
yamafaktory 7y ago
Hi, awesome work! Love the idea of using S-expression for the query syntax! Funny fact: we share the same project name - even though the core language and the query syntax are different - see https://github.com/yamafaktory/jql https://github.com/yamafaktory/jql.
- cube2222 7y ago
  Hey, looks really nice! I see we had similar goals of simplicity in mind, though with very different approaches. Sorry for the name duplication! I thought I'm sharing the name only with the jira API jql. EDIT: I'd love to see some benchmarks. I expect your jql to greatly outperform mine and jq, considering it's written in Rust!
  - yamafaktory 7y ago
    No worries regarding the naming, it's what makes open-source friendly! You can see some benches here e.g. https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-start-after_success-1 https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-st... but those actually more like regression tests that are triggered for pull requests to compare them against master (and I should use Github actions probably here now instead of Travis). Thanks for your reply!
ball_of_lint 7y ago
To me, pipe feels far more intuitive than callbacks as a way to write complex queries. Using it right away might make the first few examples much easier to read.
- cube2222 7y ago
  I suppose to each their own. I like how the query matches the json structure well.

daxelrod 7y ago

Nicely done! I scratched a similar itch by creating Jowl[1], which uses JavaScript one-liners with the Lodash library to transform JSON. Like you, I wanted syntax that was more familiar to me than jq's, and like you, I wanted to use a language that was built for data structure transformation. However, I offloaded all of the actual hard work to an existing JavaScript runtime, and to an existing library for data structure transformation, meaning I could punt on the parser and language design; the hard parts that you've taken on. I have this bookmarked to read through in more detail later this week. [1]: https://github.com/daxelrod/jowl https://github.com/daxelrod/jowl

cube2222 7y ago
Sounds great! I'll make sure to check out out! My approach was caused by the fact that I wanted to have a minimal language which only contains what really is necessary and is as regular as possible, to be very simple as a result.

downerending 7y ago

Looks great! As an old Lisper, I did get a wry chuckle out of the fact that you managed to avoid Lisp for a tool that uses Lispy syntax. :-)

cube2222 7y ago
Thanks! Yup, I felt the irony myself creating it, as I actually love clojure! But I'm most comfortable with Go, and it results with a single binary for any of the major platforms with very quick startup time (as illustrated by the small benchmark), so those were huge advantages.
- useragent86 7y ago
  I love the idea of a Lispy version of jq, but as a Lisper, the examples feel bewildering to me. The fact that (elem) takes an optional second argument, whose default value is 'identity, seems familiar, but then I try to understand this example, and it just doesn't make sense to me: (elem "countries" (elem (keys) (elem "name"))) I think part of the problem is, IIUC, this is constructing a pipeline, like (->) in Clojure, but is using function call syntax rather than pipeline syntax, and also mixing in a function call with (keys). I feel like it's a very non-Lispy language masquerading as Lisp by wearing parens. Instead, if you wrote it as real Lisp function calls, and provided a real pipeline macro, it would be much more Lispy and, I think, much easier to understand. For example, instead of: (elem "countries" (elem (keys) (elem "name"))) Either of these alternative syntaxes: (-> "countries" (key "name")) (key "name" ("countries")) Would return the same result: ["Poland", "United States", "Germany"] While being more concise and more Lispy. I also wouldn't mind using an anaphoric 'mapcar, something like: (-> "countries" (map-> (key "name")) Otherwise it may be unclear which functions implicitly map across entries and which don't. My two cents. Thanks for sharing your work.
  - cube2222 7y ago
    Thanks for the feedback! Have you checked out the type cheatsheet? (keys) is a function call! The function you pass as last to elem is kind of a continuation. It's a function which transforms the output before returning. All you're doing writing a jql query is composing one big function. With values in function call position, it's just that (keys) gets evaluated to a value (the keys of the map) and that is used to index the json. That's why keys wouldn't work, you need the parentheses to make it a function call. This may sound complex at first but I find it gets intuitive quickly. EDIT: Check out this comment thread on reddit as I think the commenter may have had similar problems: https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_query_processor_with_an_easier_lispy/fckwj7t/ https://www.reddit.com/r/golang/comments/ehnsz5/jql_json_que...

jsd1982 7y ago

Looks good but there seems to be a missing explanation gap between switching from `(elem "countries")` and dropping the `elem`s to just `("countries")`. Why is `elem` able to be dropped? Is it the default function or something? I didn't follow that big jump in the readme.

cube2222 7y ago
It's explained right below, I think I messed up the order there. Is this understandable? "You can see that elem is the most used function, and in fact it's what you'll usually be using when munging data, so there's a shortcut. If you put a value in function name position, it implicitly converts it to an elem." EDIT: Fixed the order.
aasasd 7y ago
This feature can probably be thought of similarly to Clojure's map access via keywords: if you write `(:something map-var)`, you'll get the value of the field ‘:something’ in the ‘map-var’ variable. I.e. keywords can be ‘executed’, and such calls get translated to map access.
- cube2222 7y ago
  Indeed, it just extends to values, and works recursively if it's an array or map.

yamafaktory 7y ago

Hi, awesome work! Love the idea of using S-expression for the query syntax! Funny fact: we share the same project name - even though the core language and the query syntax are different - see https://github.com/yamafaktory/jql https://github.com/yamafaktory/jql.

cube2222 7y ago
Hey, looks really nice! I see we had similar goals of simplicity in mind, though with very different approaches. Sorry for the name duplication! I thought I'm sharing the name only with the jira API jql. EDIT: I'd love to see some benchmarks. I expect your jql to greatly outperform mine and jq, considering it's written in Rust!
- yamafaktory 7y ago
  No worries regarding the naming, it's what makes open-source friendly! You can see some benches here e.g. https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-start-after_success-1 https://travis-ci.org/yamafaktory/jql/jobs/618374730#fold-st... but those actually more like regression tests that are triggered for pull requests to compare them against master (and I should use Github actions probably here now instead of Travis). Thanks for your reply!

ball_of_lint 7y ago

To me, pipe feels far more intuitive than callbacks as a way to write complex queries. Using it right away might make the first few examples much easier to read.

cube2222 7y ago
I suppose to each their own. I like how the query matches the json structure well.