34 ms·
Q: A faster re-implementaiton of jq written in Reason Native/OCaml
- jakuboboza 6y agoDo we need to make jq faster ? Anyone has issues with current speed ? Is there any specific reason other than "because we can" ?
- phonebucket 6y agoI can't answer for the OP, but "because we can" is a valid enough reason (pun unintended) for me. IMO, an individual dev making a fast useful tool should always be welcomed as a feat of worthy hacking.
- ludamad 6y agoIf we were at a board meeting deciding how to spend my time, no. If it is done? Why not
- mumblemumble 6y agoAccording to the "Purpose" section of the readme, it doesn't look like beating jq's speed was ever a goal. It was meant to be a learning exercise. But if I had done something like that, and then serendipitously discovered that I was exceeding the original's performance, I certainly wouldn't be shy about it. Also, this comes across as armchair criticism purely for the sake of armchair criticism. My own experience has been that, when I'm doing ETL that involves wrangling JSON, the "wrangling JSON" bit of it is almost always the bottleneck. So any improvement is more than welcome and deserves to be cheered. Even if it's an improvement on something that's already the current fastest way to do it.
- davesnx 6y agoI'm not sure at which "we" do you refer. Reimplementing a piece of software that is 12 years old which mimics their UX and improves performance and error messages it's more than welcome in my opinion. My purpose was to learn the OCaml stack of writting compilers, so I personally found that I "needed" a language already created. Thanks for raising those concerns
- specialist 6y agoBatting practice. We'd all be better off if plebes grew their skills by reimplementing common tools.
- StavrosK 6y agoThis looks nice, but I was a bit dismayed at "friends don't let friends curl | bash, to install this run curl | bash".
- faitswulff 6y agoThat is pretty amusing. I’ve seen some bootstrap scripts that pipe the curled output to the terminal for approval before executing it. That seems like an ergonomic alternative to curl | bash. It would be at least as useful as the terms of service warnings before you install something, anyway.
- konjin 6y agoI remember one of the first times I tried installing Linux software in the wild. The bash script asked for your password, sent it to their server using curl then returned you the script with the password hard coded into it, run itself with sudo, all over unencrypted http. I was 17 but even then I stopped to think if this was a good idea. It wasn't.
- davesnx 6y agoThere's 4 alternatives to install query-json. Before doing any curl | bash, check what's on the install command, that's the entire point of it.
- andylynch 6y agoThis looks interesting, but could be confusing given the programming language of the same name (https://code.kx.com/q/ https://code.kx.com/q/)
- dkdk8283 6y agoIs jq slow? I have only worked with datasets up to 1mb but I’ve never had a performance issue that wasn’t attributed to my error.
- minaguib 6y agoYes I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions. jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.
- nicoburns 6y agojq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file). The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.
- davesnx 6y agoRight, q doesn't support streaming so it will manage a 90GB JSON. I should specify that on the performance section, Thanks!
- arethuza 6y agoOut of interest, what created a 90GB JSON file?
- lllr_finger 6y agoNot OP, but I routinely call a specific HTTP API for millions of entities or pull down entire Kafka topics - all in JSON format. For various reasons those are the canonical sources of data and/or the most performant, so I end up ripping through GBs and GBs of JSON when troubleshooting/reporting on things.
- nicoburns 6y agoA Firebase Realtime Database backup file
- toastal 6y agoAre we sure it should get a single-letter 'q' binary name though? Docs seem to point that it's short for 'query-json'? Why not call it 'query-json' and let the user decide that as a shell alias or whatever. Even the ubiquitous 'ls' and 'cd' are two characters.
- mritchie712 6y agoor at least qj
- davesnx 6y agoI don't want to fight for name, q was a shortener. Happy to rename it to qj instead, but the option of renaming the binary it's a good workaround.
- toastal 6y agoGiven these are two of the least-used characters in English and basically never appear in this sequence, it would certainly help for all the reasons people are mentioning. It'd at least follow suit with like 'rg' for 'ripgrep'.
- rurban 6y agoI made the same argument to him on reddit. q even exists already. He replied and thought about qj
- Hnrobert42 6y agoYeah. Can you imagine trying to do a web search for ‘q’?
- ksmg 6y agoHm, I thought q is synonym for querying CSV files https://harelba.github.io/q/ https://harelba.github.io/q/
- mkesper 6y agoI'd long for such a tool with a better comprehensible query language.
- rbrtdrmpc- 6y agothis
- cube2222 6y agoIf so, and for anybody else having this wish, check out jql[0], I've created it exactly for this reason, to have the most common jq operations available in a more uniform and easier to use interface. [0]: https://github.com/cube2222/jql https://github.com/cube2222/jql
- davesnx 6y agoNice! I will try to bring it to the brenchmark, thanks for sharing
- peterohler 6y agoThe oj command in https://github.com/ohler55/ojg https://github.com/ohler55/ojg uses JSONPath as the query and filter language. Maybe it is more in line with what you are looking for.
- diggan 6y agoGive jet a try! Uses a lightweight query language over EDN. If you're familiar with Clojure, it'll be very natural to use and if you're not familiar with Clojure, the query language used is very easy to pickup :) https://github.com/borkdude/jet/blob/master/doc/query.md https://github.com/borkdude/jet/blob/master/doc/query.md
- JamesSwift 6y agoFor anything moderately complex I iterate using https://jqplay.org/ https://jqplay.org/. Life is much better since I started doing that.
- 6y ago
- YesThatTom2 6y agoGreat! Now improve the syntax!
- dividedbyzero 6y agoHow, though? I agree that jq's syntax isn't exactly the most straightforward, and it gets raised as a point of criticism anytime jq is mentioned, but its scripting language seems like a pretty good compromise between compactness and rich features. Replacing that with, say, traditional command line flags would make it a lot less useful for me, I'd probably have to build much longer pipe-chains to do things that are relatively simple and readable jq snippets (if one knows the syntax.) Using an established scripting language in its place would make it pretty much just python -c/ruby -e or whatever with some pre-loaded functions, but what's the point? You can always just write a quick python/ruby/whatever script, jq to me is an alternative for cases where a script feels unnecessary. It would also mean everything gets more verbose, so less of my jq transformations can be inlined without loss of readability. Aligning it to more established languages would probably cause confusion as well in those cases where it doesn't match the reference language 1:1. Looks like javascript, writes like javascript, but only for a tiny subset of the language, etc. Doing this only for a few function names or syntax constructs still results in a pretty unique and unusual language that will require people to reference the docs a lot, just now lots of existing scripts break.
- davesnx 6y agoJust because jq is very well stablished doesn't mean their APIs are well designed and we shoudn't improved because will break existing scripts. There're a lot of quirks from the usage of it and people struggling with learning such a great tool, so in the area of query-json it will try to make a better interface for users.
- aasasd 6y agoPerhaps one of these might work for you: https://news.ycombinator.com/item?id=24470715 https://news.ycombinator.com/item?id=24470715
- JamesSwift 6y ago
- jamil7 6y agoAs an outsider I get very confused by the Reason / Reason Native / OCaml / Bucklescript / Rescript?! ecosystem. What does it mean for it to be written in Reason Native/OCaml?
- davesnx 6y agoRight, the explanation of Reason - BuckleScript - OCaml is always nebulous. I used Reason to compile to Native, so using OCaml's stdlib and OCaml's dependencies and compiling it with OCaml, but my source code is written in Reason syntax.
- rashkov 6y agoThat means it produces a native binary (for example, a .exe file on windows platforms), so ultimately you're aiming to run the program in a terminal. This is the normal way for OCaml to operate. In this case the author is using Reason as an alternative syntax to OCaml. Reason resembles javascript a little more, and some people find that nicer to work with. So the idea is that you write Reason code, then translate it into OCaml code using the Reason tools, and then ultimately you compile it down to a native binary. If instead you want to write a web-app which runs in a web browser or node.js, then you'd need to compile it to Javascript, which is what bucklescript helps you do. Where does Rescript come in? As explained above, Reason can be used for writing either native apps or javascript apps. However, it's hard to evolve the syntax of Reason in a way which satisfies both aims. So they've now split the work -- going forward, Reason will specialize on native, and Rescript will specialize on javascript apps. Their syntax is expected to diverge from each other, in order to support those aims as best as they can.
- jamil7 6y agoThank you for the detailed answer! I check in on the status of the related projects from time to time and was often confused by the relationship between the components.
- layoutIfNeeded 6y agoUmm... There's already a language called Q for array processing.
- davesnx 6y agoWill rename it to query-json. Thaaanks!
- gkfasdfasdf 6y agoCurious, any description as to why it's faster? Something intrinsic to Reason Native/OCaml? Architectural changes? Reduced feature set?
- tyingq 6y agoJq appears to have its own hand written json parser and requires flex/bison. I suspect something about the hand written parser is slow for large data sets. I was somewhat surprised it didn't use an existing json parser library.
- brundolf 6y agoI'm doubly surprised that such a popular utility uses bison; generated parsers tend to be slower than handwritten parser, and JSON isn't exactly the world's hardest language to parse
- masklinn 6y agojq’s in C do probably not anything intrinsic. Not to mention I don’t think the ocaml compiler is an optimisation beast.
- dubcanada 6y agoWell it is missing a ton of jq functionality, it's possible that in that list is something causing the performance degradation.
- vasergen 6y agoThe speed is not concern for me. I am wondering if there something better than `jq` in terms of syntax. Whenever I want to get something more that just prettify json output in the console or simply get value by specific field name I have a problem, for me it is just difficult to remember jq syntax without looking into history. As well have in my notes links to examples like this one https://mosermichael.github.io/jq-illustrated/dir/content.html https://mosermichael.github.io/jq-illustrated/dir/content.ht...
- davesnx 6y agoOne of the main ideas of query-json is to provide excelent errors. So, it would teach you by using the tool. and there are a few techniques to "discover" the schema of the json file, I trend to read with '.' or 'keys' and later keep going. I'm planning to implement a flag where each operation prints the internal state of the json, so you would see what are the "pipes". I will pick a few of your cheatsheet to implement next in q, Thanks!
- mumblemumble 6y agoFor my part, I've always wanted a tool that just replicates PostgreSQL's JSON syntax. That way I can have only one syntax to remember.
- bradly 6y agoCheck out jql [https://github.com/cube2222/jql https://github.com/cube2222/jql] and oj [https://github.com/ohler55/ojg https://github.com/ohler55/ojg]
- vasergen 6y agodefinitively will take a look, I've never heard of `jql` before, thanks
- aasasd 6y agoMight want to take a look at some of these alternatives: https://news.ycombinator.com/item?id=24470715 https://news.ycombinator.com/item?id=24470715
- jeffbee 6y ago1) refuses to operate on stdin; requires a filename argument, which is so irritating. 2) doesn't accept values that jq accepts % time jq -r '[expression]' < parcels | wc 365 1454 7978 jq -r < parcels 1.39s user 0.00s system 99% cpu 1.390 total wc 0.00s user 0.00s system 0% cpu 1.390 total % time ~/.yarn/bin/q '[expression]' parcels | wc q: internal error, uncaught exception: Yojson.Json_error("Line 56, bytes -1-32:\nJunk after end of JSON value: '{\n \"OBJECTID\": 155303,\n \"BOOK\"'")
- f311a 6y agoThat's super weird, I think most people use jq for bash pipelines.
- jeffbee 6y agoYes, I don't understand how people end up with assertions that the filename is a require argument. At least we've got /dev/stdin or /proc/self/fd/0 as workarounds.
- benibela 6y agofiles might be faster, because you can mmap them?
- f311a 6y agoMost people work on compressed JSON lines files. Sometimes they are stored on s3. Files do not give flexibility. When using jq, I can do a lot of things: aws s3 cp s3://bucket/file.json.gz - | zcat | head | jq .field | sort
- jerf 6y agoAnother common thing you can do is accept a generic stream as input, but have some code that penetrates the abstraction a bit to see what kind of stream it is, and if it is a file or something, do something special with it to go even faster. This way, you start with something maximally useful up front, and easy to use, but you can optimize things based on details as you go. That's how Go's static file web server works. It serves streams, but if you happen to io.Copy to that stream with something that is also an ∗os.File on Linux, it can use the sendfile call in the kernel instead. (A downside of making it so transparent is that if you wrap that stream with something you may not realize that you've wrecked the optimization because it no longer unwraps to an ∗os.File but whatever your wrapper is, but, well, nothing's perfect.)
- muktabh 6y agoSlightly out of context here, I find the entire stack of bsb, bsb-native, ocaml and esy pretty cool. However, I just dont find enough resources, good tutorials etc on Google search. Is there a good set of beginner tutorials anyone can point to ? Thanks in advance.
- davesnx 6y agoThe documentation is a problem in the OCaml world and a problem with Reason Native as well. I found myself pretty lost some times, esy.sh should be a initial point in contact for most of Reason related stuff. Menhir/sedlex and others are pretty high accessibility barrier for new commers. One of the nice things about all of it it's the discord, it's friendly and always helpful. Hope it helps, just let me know if there's any specific!
- smabie 6y agoJust ditch Reason and use OCaml. There's a lot more documentation and the syntax is better.
- jonemi 6y agoI used to be a regular user of jq, but I was never parsing very large JSON. I now do what I used to do with jq in my browser's developer tools console. Map and filter are far more familiar than jq's syntax where I found myself referring to the documentation most of the time. I'm sure other people have use cases where the browser wouldn't meet their needs, but for me, I find jq unnecessary.
- choward 6y agoWriting a script? I'm not going to have my script open a web browser so I can attempt to interact with a web console.
- jonemi 6y agoWhen it got to the point when I needed a script, I just preferred Python. I can understand how some might prefer jq and a shell script, I just realized it wasn't worth it for my particular needs.
- Borkdude 6y agoIf you're into Clojure, check out https://github.com/borkdude/jet https://github.com/borkdude/jet
- iLemming 6y agoI use jet all the time when I need to quickly examine a json snippet in Emacs. I would use <C-u M-|> (shell-command-on-region with a prefix) and execute jet to convert selected json part to EDN. That cuts out all the visual noise. EDN is much more concise, cleaner and easier to read. I'd use it even if I don't write Clojure.
- as-j 6y ago> Aside from that, q isn't feature parity with jq which is ok at this point, but jq contains a ton of functionality that query-json misses and some of the jq operations aren't native, are builtin with the runtime. In order to do a proper comparision all of this above would need to take into consideration. > The report shows that q is between 2x and 5x faster than jq in all operations tested and same speed (~1.1x) with huge files (> 100M). While faster for somethings....that's a pretty large set of caveats!
- davesnx 6y agoAdding most of the jq operations shoudn't affect performance at all, in fact If I endup implementing streaming could be even faster. I have a issue to improve performance where I can push this forward: https://github.com/davesnx/query-json/issues/7 https://github.com/davesnx/query-json/issues/7 But sure, are caveats!
- aasasd 6y agoFor everyone pining for a Jq with a different syntax: I have a bunch of links to alternatives collected, you might want to try some of them (some may be for different things than JSON): https://github.com/fiatjaf/awesome-jq https://github.com/fiatjaf/awesome-jq https://github.com/TomConlin/json2xpath https://github.com/TomConlin/json2xpath https://github.com/antonmedv/fx https://github.com/antonmedv/fx https://github.com/fiatjaf/jiq https://github.com/fiatjaf/jiq https://github.com/simeji/jid https://github.com/simeji/jid https://github.com/jmespath/jp https://github.com/jmespath/jp https://github.com/cube2222/jql https://github.com/cube2222/jql https://jsonnet.org https://jsonnet.org https://github.com/borkdude/jet https://github.com/borkdude/jet https://github.com/jzelinskie/faq https://github.com/jzelinskie/faq https://github.com/dflemstr/rq https://github.com/dflemstr/rq Personally I think that next time I might just fire up Hy and use its functional capabilities.
- vips7L 6y agoDon't forget powershell's Convert-FromJson :) https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertfrom-json?view=powershell-7 https://docs.microsoft.com/en-us/powershell/module/microsoft...
- melbourne_mat 6y agoThat is so not jq! I've really been pining for jq on my current Windows project :-(
- vips7L 6y agoWhat do you feel is missing?
- mkesper 6y agoDoes it still convert only two levels by default?
- cristoperb 6y agoMy go-to for simple queries is https://github.com/tidwall/jj https://github.com/tidwall/jj It is not nearly as expressive as jq, but it is faster for my use cases (written in golang).
- RMPR 6y agoUpcoming q-rs a rewrite of q in Rust :p
- davesnx 6y agoI hope so!
- yahyaheee 6y agoI’m with Q!
- davesnx 6y ago+1
- deleted 6y ago[deleted]
- riston 6y agoWould be good if someone adds an explanation why this new approach is better, is it that the OCaml is faster, more efficient algorithms were used, etc?
- davesnx 6y agoI tried to explain it on the Performance section and on the report https://github.com/davesnx/query-json#performance https://github.com/davesnx/query-json#performance https://github.com/davesnx/query-json/blob/master/benchmarks/report.md https://github.com/davesnx/query-json/blob/master/benchmarks... But all explanations aren't based by any evidence, just asumptions.
- tus88 6y agoIsn't JQ written in C? I doubt LISP is going to be faster.
- davesnx 6y agoYes, jq is written in C. Where LISP comes from?
- Ericson2314 6y agoThis is funny because Stephen Dolan, the original jq author, works on OCaml itself.
- davesnx 6y agoExactly! I wanted to contact him
- nikolay 6y agoJMESPath is the only viable alternative, which probably has a wider footprint than even jq as it's part of AWS CLI.
- acdha 6y agoIt's definitely popular but “only viable alternative” is a bit strong: that's only if you need compatibility with particular tools which support only one of the two formats. There's no reason why anyone who doesn't like those tools couldn't create a different syntax to scratch whatever particular itch they have.
- nikolay 6y agoIt's embeddable and available as a library for all languages [0]. Everything else is nothing but an CLI tool pretty much, which further limits its adoption. [0]: https://github.com/jmespath https://github.com/jmespath
- acdha 6y agoWell, there is XPath 3.1 if you want standards[1] but my point was simply that it depends on whether your question is “I need compatibility with existing jq scripts”, “I need an embeddable library I can integrate in other programs”, or “I want to process JSON for my own usage”. For example, someone who works with a lot of Python might prefer something like https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello to write comprehensions using the full capabilities of Python, especially since that would provide a direct path to using the final expressions in a Python program or even embedded in one of the environments where Python is used as a scripting language. Is that a viable alternative? The answer depends entirely on who's asking. 1. https://www.w3.org/TR/xpath-31/#id-introduction https://www.w3.org/TR/xpath-31/#id-introduction
- skywhopper 6y agoThis is cool, but I’m not sure it’s fair to claim it’s “faster” yet when it doesn’t do 95% of what jq does—-particularly the command line options. If it’s still faster when you can match 80% of the functionality, then it might be a claim worth making.
- davesnx 6y agoExactly I didn't claim to be faster in all the cases, since there's no feature parity and I won't make it that way. For the set of operations that I implement it it's faster, that's true.
- heycosmo 6y agoIn case anyone is interested in yet another alternative, I have this old, unpolished project: https://github.com/bauerca/jv https://github.com/bauerca/jv It is a JSON parser in C without heap allocations. The query language is piddly, but the tool can be useful for grabbing a single value from a very large JSON file. I don't have time for it, but someone could fork and make it a real deal.
- brundolf 6y agoI'd love to hear some speculation - from the author or otherwise - as to why a fresh OCaml implementation would so dramatically outperform a mature C implementation
- davesnx 6y agoThere are a few good asumtions about why is faster, there are just speculations since I didn't profile jq or query-json. The feature that I think penalizes a lot jq is "def functions", the capacity of define any function that can be available during run-time. This creates a few layers, one of the difference is the interpreter and the linker, the responsible for getting all the builtin functions and compile them have them ready to use at runtime. The other pain point is the architecture of the operations on top of jq, since it's a stack based. In query-json it's a piped recursive operations. Aside from the code, the OCaml stack, menhir has been proved to be really fast when creating those kind of compilers. I will dig more into performance and try to profile both tools in order to improve mine. Thanks