18 ms·
A Faster Alternative to Jq
- furryrain 6mo agoIf it's easier to use than jq, they should sell the tool on that.
- deleted 6mo ago[deleted]
- Bigpet 6mo agoWhen initially opening the page it had broken colors in light mode. For anyone else encountering it: switch to dark mode and then back to light mode to fix it.
- vladvasiliu 6mo agoLooks fine to me on Edge/Windows.
- keysersoze33 6mo agoI had the same problem (brave browser)
- qwe----3 6mo agoWhite text with light background, yeah.
- jvdvegt 6mo agoFine in Firefox on Android. Note that the scales of the charts are all different, which makes them hard to compare. Also, there are lots of charts without comparison so the numbers mean nothing...
- youngtaff 6mo agoBroken on iOS Safari too
- CodeCompost 6mo agoI suspect the website is vibe-coded, like the tool itself.
- jmalicki 6mo agoI can forgive vibe code... It needs to execute if it works it's fine. Unedited vibe documentation is unforgivable.
- merlindru 6mo agothis is a bad faith take. i think the website is really cool and doesn't reek of slop at all. what makes you think differently?
- xeyownt 6mo agoSame. And who cares if it's vibe-coded or not. Since when do we care more on the how than on the what? Are people looking at how a tool was coded before using it, as if it would accelerate confidence?
- 1123581321 6mo agoIt’s a heuristic to approach a program a bit warily as the length of the documentation likely outpaces how thoroughly it was designed and tested.
- g947o 6mo agoI would not be surprised at all if it's vibe coded. I have seen exactly the same thing myself. I gave instruction to Claude to add a toggle button to a website where the value needs to be stored in local storage. It is a very straightforward change. Just follow exactly how it is done for a different boolean setting and you are set. An intern can do that on the first day of their job. Everything is done properly except that on page load, the stored setting is not read. Which can be easily discovered if the author, with or without AI tools, has a test or manually goes through the entire workflow just once. I discovered the problem myself and fixed it. Setting all of that aside -- even if this is not AI coded, at the least it shows the site owner doesn't have the basic care for its visitors to go through this important workflow to check if everything works properly.
- shellac 6mo agoI think this has just been fixed. A bit of dark mode was leaking into light in the css.
- xyst 6mo agoModern programmers these days just give a shit about user experience. Better to just load up in reader mode.
- micahkepe 6mo agoHey OP here! Sorry about this this is just laziness on my part because I never use light mode so I forget to test haha, will push a fix!
- micahkepe 6mo agoShould be fixed now! Let me now :)
- Kovah 6mo agoI wonder so often about many new CLI tools whose primary selling point is their speed over other tools. Yet I personally have not encountered any case where a tool like jq feels incredibly slow, and I would feel the urge to find something else. What do people do all day that existing tools are no longer enough? Or is it that kind of "my new terminal opens 107ms faster now, and I don't notice it, but I simply feel better because I know"?
- Jakob 6mo agoSpeed is a quality in itself. We are so bugged down by slow stuff that we often ignore that and don’t actively search for another. But every now and then a well-optimised tool/page comes along with instant feedback and is a real pleasure to use. I think some people are more affected by that than others. Obligatory https://m.xkcd.com/1205 https://m.xkcd.com/1205
- Imustaskforhelp 6mo agoI am not sure if it was simon or pg who might've quoted this but I remembered a quote about that a 2 magnitude order in speed (quantity) is a huge qualititative change in it of itself.
- n_e 6mo agoI process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.
- eru 6mo agoThis reminds me of someone who wrote a regex tool that matches by compiling regexes (at runtime of the tool) via LLVM to native code. You could probably do something similar for a faster jq.
- nchmy 6mo agoThis isn't for you then > The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation. Mind me asking what sorts of TB json files you work with? Seems excessively immense.
- keysersoze33 6mo agoI was a bit skeptical at first, but after reading more into jsongrep, it's actually very good. Only did a very quick test just now, and after stumbling over slightly different syntax to jq, am actually quite impressed. Give it a try
- carlmr 6mo agoWhat were your syntax stumbling blocks? I must be honest I've used jq enough but can never remember the syntax. It's one of the worst things about jq IMO (not the speed, even though I'm a fan of speedups). There's something ungrokkable about that syntax for me.
- keysersoze33 6mo agoJust the basic things, like viewing the complete json (with syntax highlighting) to then determine the filter, that is '.' becomes '**'
- silverwind 6mo agoEffort would be better investigated making `jq` itself faster.
- ifh-hn 6mo agoI learned a number of data processing cli tools: jq, mlr, htmlq, xsv, yq, etc; to name a few. Not to the level of completing advent of code or anything, but good enough for my day to day usage. It was never ending with the amount of formats I needed to extract data from, and the different syntax's. All that changed when I found nushell though, its replaced all of these tools for me. One syntax for everything, breath of fresh air!
- joknoll 6mo agoSame here, nushell is awesome! It helped me to automate so many more things than I did with any other shell. The syntax is so much more intuitive and coherent, which really helps a lot for someone who always forgot how to write ifs or loops in bash ^^
- igorramazanov 6mo agoSame! Nushell replaced almost all of them Had to spend some efforts to set up completions, also there some small rough edges around commands discoverability, but anyway, much better than the previous oh-my-zsh setup Ideally, wish it also had a flag to enforce users to write type annotations + compiling scripts as static binaries + a TUI library, and then I'd seriously consider it for writing small apps, but I like and appreciate it in the current state already
- rlonstein 6mo ago+1. I switched to using Nushell as my daily driver around mid-2023 (0.84.0?) and use it in preference to other interactive tools. I do keep at hand jq, yq, and mlr because I need to exchange stuff with colleagues who don't use Nu.
- ndyg 6mo agoSomething I find myself saying a lot, Nushell is a better `jq` than `jq`
- steelbrain 6mo agoSurprised to see that there's no official binaries for arm64 darwin. Meaning macOS users will have to run it through the Rosetta 2 translation layer.
- QuantumNomad_ 6mo agoI’d install it via cargo anyway and that would build it for arm64. If the arm64 version was on homebrew (didn’t check if it is but assume not because it’s not mentioned on the page), I’d install it from there rather than from cargo. I don’t really manually install binaries from GitHub, but it’s nice that the author provides binaries for several platforms for people that do like to install it that way.
- maleldil 6mo agoYou can use cargo-binstall to retrieve Github binary releases if there are any.
- baszalmstra 6mo agoReally? That is your response? This is an high quality article from someone who spend a lot of time implementing a cool tool and also sharing the intricate inner workings of it. And your response is, "eh there are no official binaries for my platform". Give them some credit! Be a little more constructive!
- coldtea 6mo agoHis response at least fits the discussion and is relevant to the tool, not generic hollier-than-thou scolding. To address the concern, anyway, I'm sure it would soon be available in brew as an arm binary.
- alexellisuk 6mo agoJust hit this too: https://news.ycombinator.com/item?id=47542182 https://news.ycombinator.com/item?id=47542182 The reason I was interested, was adding the new tool to arkade (similar to Brew, but more developer/devops focused - downloads binaries) The agent found no Arm binaries.. and it seemed like an odd miss for a core tool https://x.com/alexellisuk/status/2037514629409112346?s=20 https://x.com/alexellisuk/status/2037514629409112346?s=20
- maxloh 6mo agoFrom their README [0]: > Jq is a powerful tool, but its imperative filter syntax can be verbose for common path-matching tasks. jsongrep is declarative: you describe the shape of the paths you want, and the engine finds them. IMO, this isn't a common use case. The comparison here is essentially like Java vs Python. Jq is perfectly fine for quick peeking. If you actually need better performance, there are always faster ways to parse JSON than using a CLI. [0]: https://github.com/micahkepe/jsongrep https://github.com/micahkepe/jsongrep
- quotemstr 6mo agoReminder you can also get DuckDB to slurp the JSON natively and give you a much more expressive query model than anything jq-like.
- hackrmn 6mo agoHaving used `jq` and `yq` (which followed from the former, in spirit), I have never had to complain about performance of the _latter_ which an order of magnitude (or several) _slower_ than the former. So if there's something faster than `jq`, it's laudable that the author of the faster tool accomplished such a goal, but in the broader context I'd say the performance benefit would be required by a niche slice of the userbase. People who analyse JSON-formatted logs, perhaps? Then again, newline-delimited JSON reigns supreme in that particular kind of scenario, making the point of a faster `jq` moot again. However, as someone who always loved faster software and being an optimisation nerd, hat's off!
- alcor-z 6mo ago[dead]
- bungle 6mo agoIntegrating with server software, the performance is nice to have, as you can have say 100 kRPS requests coming in that need some jq-like logic. For CLI tool, like you said, the performance of any of them is ok, for most of the cases.
- robmccoll 6mo agojq is probably faster than storage, the network, compression, or something else in your stack and not your bottleneck.
- mroche 6mo ago> Having used `jq` and `yq` If you don't mind me asking, which yq? There's a Go variant and a Python pass-through variant, the latter also including xq and tomlq.
- hackrmn 6mo agoIndeed, thanks for spotting that, as I myself remember discovering there's at least two. Thing is, I had learned and started with Mike Farah's `yq`, not the pass-through-to-`jq` variant written in Python that's often more easily (read: system package manager) available. Both semantics and syntax are a bit different between the two. A bit of a fun fact: there's a quote by Farah where he said that the language and semantics of the tool he was writing, didn't really "click in" until he was well into writing it :-) I myself have been on occasion pulling my hair out trying to wield `yq`'s language, there's some inconsistencies here and there which I think are related to the novel nature of the language (not novel to everyone but it's uncommon even for those well versed with e.g. SQL). `jq` suffers from similar woes, but to a lesser degree.
- adastra22 6mo agoThe fastest alternative to jq is to not use JSON.
- deleted 6mo ago[deleted]
- 1vuio0pswjnm7 6mo agoThe unlimited memory required for JSON is poor design netstrings has no such issues
- jiehong 6mo agoFirst of all, congratulations! Nice tool! Second, some comments on the presentation: the horizontal violin graphs are nice, but all tools have the same colours, and so it's just hard to even spot where jsongrep is. I'd recommend grouping by tool and colour coding it. Besides, jq itself isn't in the graphs at all (but the title of the post made me think it would be!). Last, xLarge is a 190MiB file. I was surprised by that. It seems too low for xLarge. I daily check 400MiB json documents, and sometimes GiB ones.
- micahkepe 6mo agoHey thank you! OP here, yes I was struggling to find large enough documents to run the benchmarks on, the range currently on the benchmark data is ~106 B - ~190MB, which I think covers the majority of quick task workloads, but would love to have large documents, if there's an public ones you can thinking of I'd like to know!
- jiehong 6mo agoThe US government tend to offer big public json document [0], such as crime rates [1], or others. [0]: https://catalog.data.gov/dataset/?res_format=JSON https://catalog.data.gov/dataset/?res_format=JSON [1]: https://catalog.data.gov/dataset/crimes-2001-to-present https://catalog.data.gov/dataset/crimes-2001-to-present
- coldtea 6mo agoSpeed is good! Not a big fan of the syntax though.
- 1a527dd5 6mo agoI appreciate performance as much as the next person; but I see this endless battle to measure things in ns/us/ms as performative. Sure there are 0.000001% edge cases where that MIGHT be the next big bottleneck. I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart. 9/10 whatever tooling you are using now will be perfectly fine. Example; I use grep a lot in an ad hoc manner on really large files I switch to rg. But that is only in the handful of cases.
- dalvrosa 6mo agoFair, but agentic tooling can benefit quite a lot from this Opencode, ClaudeCode, etc, feel slow. Whatever make them faster is a win :)
- jamespo 6mo agoIt's not running jq locally that's causing that
- httpsterio 6mo agoThe 2ms it takes to run jq versus the 0.2ms to run an alternative is not why your coding agent feels slow.
- jmalicki 6mo agoStill, jq is run a whole lot more than it used to be due to coding agents, so every bit helps. The vast majority of Linux kernel performance improvement patches probably have way less of a real world impact than this.
- PunchyHamster 6mo ago> The vast majority of Linux kernel performance improvement patches probably have way less of a real world impact than this. unlikely given that the number they are multiplying by every improvement is far higher than "times jq is run in some pipeline". Even 0.1% improvement in kernel is probably far far higher impact than this
- bouk 6mo agoI highly recommend anyone to look at jq's VM implementation some time, it's kind of mind-blowing how it works under the hood: https://github.com/jqlang/jq/blob/master/src/execute.c https://github.com/jqlang/jq/blob/master/src/execute.c It does some kind of stack forking which is what allows its funky syntax
- vbezhenar 6mo agoLooks like naive implementation of homemade bytecode interpreter. What's so mind blowing about that? Maybe I missed something.
- functional_dev 6mo agoThe backtracking implementation in jq is really the secret sauce for how it handles those complex filters without getting bogged down
- wolfi1 6mo agoforgive me my rant, but when I see "just install it with cargo" I immediately lose interest. How many GB do I have to install just to test a little tool? sorry, not gonna do that
- onedognight 6mo agoHaving the equivalent jq expression in these examples might help to compare expressiveness, and it might help me see if jq could “just” use a DFA when a (sub)query admits one. grep, ripgrep, etc change algorithms based on the query and that makes the speed improvements automatic.
- stuaxo 6mo agoNice. Some bits of the site are hard to read "takes a query and a JSON input" query is in white and the background of the site is very light which makes it hard to read.
- enricozb 6mo agoI am excited for some alternative syntax to jq's. I haven't given much thought to how I'd write a new JSON query syntax if I were writing things from scratch, but I personally never found the jq syntax intuitive. Perhaps I haven't given it enough effort to learn properly.
- s_dev 6mo agoYou don't learn it properly. It's not supposed to be intuitive, it's supposed to be concise at the cost of it being intuitive. Would be like somebody saying typing words in to Google is more intuitive than writing regex. jq is supposed to fit in to other bash scripts as a one liner. That's it's super power. I know very few people who write regex on the fly either (unless you were using it everyday) they check the documentation and flesh it out when they need it. Just use Claude to generate the jq expression you need and test it.
- PUSH_AX 6mo agoIs Jq slow?
- PunchyHamster 6mo agono
- marxisttemp 6mo agoMany Useless Uses of cat in this documentation. You never need to do `cat file | foo`, you can just do `<file foo`. cat is for concatenating inputs, you never need it for a single input.
- norenh 6mo agoAs someone who worked with Unix/Linux and command line arguments for 30 years and still "abuse" cat like the documentation, I regularly hear this complaint. Yes, "cmd <file" is more efficient for the computer but not for the reader in many cases. I read from left to the right and the pipeline might be long or "cmd" might have plenty of arguments (or both). Having "cat file | cmd" immediately gives me the context for what I am working with and corresponds well with "take this file, do this, then that, etc" with it) and makes it easier for me to grok what is happening (the first operation will have some kind of input from stdin). Without that, the context starts with the (first) operation like in the sentence "do this operation, on this file (,then this, etc)". I might not be familiar with it or knowing the arguments it expects. At least for me, the first variant comes more naturally and is quicker to follow (in most cases), so unless it is performance sensitive that is what I end up with (and cat is insanely fast for most cases).
- jasomill 6mo agoIf left-to-right is your main concern, observe that the post you replied to uses <file command which is equivalent to command <file
- Asmod4n 6mo agoYou could just take simdjson, use its ondemand api and then navigate it with .at_path(_with_wildcard) (https://github.com/simdjson/simdjson/blob/master/doc/basics.md#jsonpath https://github.com/simdjson/simdjson/blob/master/doc/basics....) The whole tool would be like a few dozen lines of c++ and most likely be faster than this.
- luc4 6mo agoSince the query compilation needs exponential time, I wonder how large the queries can be before jsongrep becomes slower than all the other tools. In that regard, I think the library could benefit from some functionality for query compilation at compile-time.
- mitul005 6mo ago[flagged]
- rswail 6mo agoJust about to read, but I had to change to dark mode to be able to see the examples, which are bold white on a white background.
- leontloveless 6mo ago[dead]
- sirfz 6mo agoNowadays I'd just use clickhouse-local / chdb / duckdb to query json files (and pretty much any standard format files)
- peterohler 6mo agoAnother alternative is oj, https://github.com/ohler55/ojg https://github.com/ohler55/ojg. I don't know how the performance compares to jq or any others but it does use JSONPath as the query language. It has a few other options for making nicely formatted JSON and colorizing JSON.
- Voranto 6mo agoQuick question: Isn't the construction of a NFA - DFA a O(2^n) algorithm? If a JSON file has a couple hundred values, its equivalent NFA will have a similar amount, and the DFA will have 2^100 states, so I must be missing something.
- functional_dev 6mo agotheory is one thing but the cpu cache is the real bottleneck here... here is a small visual breakdown of how these arrays look in memory and why pointer chasing is so expensive compared to the actual logic: https://vectree.io/c/json-array-memory-indexing https://vectree.io/c/json-array-memory-indexing basically the double jump to find values in the heap is what slows down these tools most
- Voranto 6mo agoI can see that in practice the bottleneck isn't the automata construction, I'm just curious of how the construction is approached with such a super-exponential conversion algorithm
- Jenk 6mo agoI switched to Jaq[0] a while back for the 'correctness' sake rather than performance. But Jaq also claims to be more performant than jq. [0]: https://github.com/01mf02/jaq https://github.com/01mf02/jaq
- password4321 6mo agoThank you for the recommendation. It looks like jaq has already progressed much further in the right direction than jsongrep has just started in the not-quite-as-right direction.
- jeffbee 6mo agoI keep an eye on jaq, but there are some holes in the story. jaq 3.0 is faster than Linux distro builds of jq, but jq built correctly is faster than jaq. As far as I can tell the performance reputation of jq is caused by bad distro packaging.
- ontouchstart 6mo agoEverything can be written in JavaScript will be written in JavaScript. Everything can be rewritten in Rust will be written in Rust.
- arjie 6mo agoThank you. Very cool. Going to try embedding this into my JSON viewer. One thing I’ve struggled with is that live querying in the UI is constrained by performance.
- alexellisuk 6mo agoQuick comment for the author. Just added this new tool to arkade, along with the existing jq/yq. No Arm64 for Darwin.. seriously? (Only x86_64 darwin.. it's a "choice") No Arm64 for Linux? For Rust tools it's trivial to add these. Do you think you can do that for the next release? https://github.com/micahkepe/jsongrep/releases/tag/v0.7.0 https://github.com/micahkepe/jsongrep/releases/tag/v0.7.0
- deleted 6mo ago[deleted]
- vindin 6mo agoThe data viz of the benchmarks is really rough. I think you’d get a lot of leverage out of rebuilding it and using colors and/or shapes to extract additional dimensions. Nobody wants to scan through raw file paths as labels to try and figure out what the hell the results are
- skywhopper 6mo agoIf the author cares, I can’t read everything on this page. The command snippets have a “BASH” pill in the top left that covers up the command I’m supposed to run. And then there are, I guess topic headings or something that are white-on-white text, so honestly I don’t know what they say or what they are.
- micahkepe 6mo agoOP here: sorry about that, the light mode inconsistencies should be fixed now. Will continue to work on making the site design better as well!
- regus 6mo agoJq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.
- NSPG911 6mo agoI also genuinely hate using jq. It is one of the only things that I rely heavily on AI.
- amelius 6mo agoAt that point why don't we ask the AI directly to filter through our data? The AI query language is much more powerful.
- latexr 6mo agoBecause the output you get can have hallucinations, which don’t happen with a deterministic tool. Furthermore, by getting the `jq` command you get something which is reusable, fast, offline, local, doesn’t send your data to a third-party, doesn’t waste a bunch of tokens, … Using an LLM to filter the data is worse in every metric.
- amelius 6mo agoYou can use a local LLM and you can ask it to use tools so it is faster.
- kelvinjps10 6mo agoThere is hardware that is able to run jq but no a local AI model that's powerful enough to make the filtering reliable. Ex a raspberry pi
- deleted 6mo ago[deleted]
- sigseg1v 6mo ago
- throwawaypath 6mo agoAfter reading the title, I was worried that this wasn't written in Rust!
- deleted 6mo ago[deleted]
- VHRanger 6mo agoIf rust is not in the HN title and fire emojis in the readme, it doesn't come from the Rust region of France. It's just sparkling memory safe high performance software
- tehnub 6mo agoI've been using jj, which apparently is also faster than jq https://github.com/tidwall/jj https://github.com/tidwall/jj
- swah 6mo agojj was already taken by jujutsu, unfortunately.
- hilti 6mo agoI'm glad you adjusted the CSS while I was typing my comment. I needed to switch to dark mode to be able to read highlighted words. Nice write up. I will try out your tool.
- mlmonkey 6mo agoLOL ... came here to grips about that! Also "jg" reads very similar to "jq", and initially I thought he was talking about "jq" all along, and I was like: where can I see the "jasongrep" examples? Threw me off for a minute.
- mlmonkey 6mo agoMinor suggestion: often I just want to extract one field, whose name I know exactly. I see that `jg` has an option `-F` like this: $ cat sample.json | jg -F name I would humbly suggest that a better syntax would be: $ cat sample.json | jg .name for a leaf node named "name"; or $ cat sample.json | jg -F .name. for any node named "name".
- 1vuio0pswjnm7 6mo agoOne problem I have not seen addressed by jq or alterataives, perhaps this one addresses it, is "JSON-like" data. That is, JSON that is not contained in a JSON file For example, web pages sometimes contain inline "JSON". But as this is not a proper JSON file, jq-style utilties cannot process it The solution I have used for years is a simple utility written in C using flex^1 (a "filter") that reformats "JSON" on stdin, regardless of whether the input is a proper JSON file or not, into stdout that is line-delimited, human-readable and therefore easy to process with common UNIX utilities The size of the JSON input does not affect the filter's memory usage. Generally, a large JSON file is processed at the same speed with the same resource usage as a small one The author here has provided musl static-pie binaries instead of glibc. HN commenters seeking to discredit musl often claim glibc is faster Personally I choose musl for control not speed 1. jq also uses flex
- 1vuio0pswjnm7 6mo ago*alternatives
- commers148 6mo ago[flagged]
- jrhey 6mo agoSince when was jq considered slow?
- allknowingfrog 6mo agoI deal with a fair amount of newline-delimited JSON in my day job, where each line in the file is a complete JSON object. I've seen this referred to as "jsonl", and it's not entirely uncommon for logs and other kinds of time-series data dumps. Do any of the popular JSON CLI tools work with this format? I didn't see any mention of it here.
- soleveloper 6mo agoI already can't remember jq syntax. Naming this jg just means I'll type one, instinctively use the other's syntax, and get an error anyway. It's a DX trap. But I will admit, the new syntax makes a lot more sense.
- ryguz 6mo ago[dead]
- vismit2000 6mo agoTable of contents seems inspired by the famous ripgrep post from 2016: https://burntsushi.net/ripgrep/ https://burntsushi.net/ripgrep/
- imranstrive7 6mo ago[dead]
- damotiansheng 6mo ago[dead]
- micahkepe 6mo agoOP jsongrep author here: v0.8.0 now has multi format support for serializable formats![^1] [1]: https://github.com/micahkepe/jsongrep/releases/tag/v0.8.0 https://github.com/micahkepe/jsongrep/releases/tag/v0.8.0
- Self-Perfection 6mo agoI think that in most cases jq is launched to extract value from relatively small JSON document, for which raw parsing speed is not affect much. jq is just really slow to start. Version 1.6 was especially abysmally slow to start, 10x times slower than 1.5: https://github.com/jqlang/jq/issues/1826 https://github.com/jqlang/jq/issues/1826 So any replacement candidate should also benchmark like hyperfine "jq .a <<< '{"a": 10 }'" . This oneliner does not work but should illustrate the idea. Also please just use jshon if you need to just extract specific value from some small JSON. jshon uses way less resources by any conceivable metric.