7 ms·
yq: command-line YAML, JSON, XML, CSV and properties processor
- notorandit 4y agoThis is really a great tool! Thanks Mike!
- user3939382 4y agoI recently dug into the docs of jq and was surprised to find that, contrary to my prior belief based on shallow experience with it, jq’s expression aren’t merely a path syntax but apparently a turing complete language. I was blown away. I wish MySQL and AWS could have figured out a way to adopt it, or a subset of it, rather than each using different ones. Now I have varying levels of knowledge for 4-5 variations of JSON path semantics/standards, it’s annoying.
- WesolyKubeczek 4y agoThere has been a jq written in jq here on the front page.
- nequo 4y agoUser odnoletkov[1] has solved several years’ worth of Advent of Code in jq: https://github.com/odnoletkov/advent-of-code-jq https://github.com/odnoletkov/advent-of-code-jq [1] https://news.ycombinator.com/user?id=odnoletkov https://news.ycombinator.com/user?id=odnoletkov
- alecthomas 4y agoThat is simultaneously impressive and insane. Respect.
- justin_oaks 4y agoThe JSON expressions used by MySQL and the AWS command line tool (JMESpath) are extremely limited compared to jq. So not only do we end up learning multiple JSON path variations, but most of them are nearly useless for anything but the simplest use cases. I appreciate the intention of including JMESpath in awscli, but I quickly dropped it in favor of piping the JSON results to jq.
- mdaniel 4y agoI have a similar complaint but I'd guess there are (at least) two problems standing in the way of awscli getting jq language support: a python impl of the language with a license that awscli tolerates, and awscli being (in general) very conservative about changes. There are innumerable open issues about quality of life improvements that are "thank you for your input" and I'd expect that change to be similarly ignored
- thayne 4y agoNot to mention that JMESpath appears to be abandoned. There is a fork (https://github.com/jmespath-community/jmespath.spec https://github.com/jmespath-community/jmespath.spec), but it seems unlikely to be used by the aws cli (https://github.com/aws/aws-cli/issues/7396 https://github.com/aws/aws-cli/issues/7396). Although, for that matter jq is semi-abandoned itself.
- deleted 4y ago[deleted]
- dragonwriter 4y ago> I wish MySQL and AWS could have figured out a way to adopt it, or a subset of it, rather than each using different ones. For AWS CLI, you can just output unfiltered JSON and pipe the results through jq; the filtering is client-side anyway, so it’s not like you are losing anything doing external filtering vs. filtering within the AWS CLI.
- eru 4y agoIt's not just a Turing complete language, but a well designed one, too.
- voytec 4y agoThe yaml document from hell[1] needed three changes ("*.html", "*.png", "!.git") to be parsed by yq at all. "Norway problem" is not a problem as no was converted to a quoted string. Unquoted strings in "allow_postgres_versions" part were not quoted by yq. [1] https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...
- MuffinFlavored 4y agoi wonder if the people who worked on the YAML spec regret not string wrapping stuff? i’m sure it was by design “at the time”?
- tyingq 4y agoThe spec says this: "The plain (unquoted) style has no identifying indicators and provides no form of escaping. It is therefore the most readable, most limited and most context sensitive style." Which reads to me that they expected people to treat the plain style as a convenience that had notable downsides. Edit: Note that this is the "plain" style, where there's also single and double quoted styles.
- tgv 4y agoA reasonable choice (in contrast to "no" or "null"). YAML is simply not a format for all use cases. It's good enough for many tasks, and more readable than most other formats where it fits.
- dragonwriter 4y ago> YAML is simply not a format for all use cases. Maybe, but unquoted strings not being the right choice for all use cases (or, similarly, structure-by-indentation not being) doesn’t show that, since YAML supports unquoted and quoted strings, and supports both indent-sensitive “block style” and delimiter-based “flow style”.
- VWWHFSfQ 4y agoLooks very cool! I don't care so much about YAML, but I do a ton of processing of JSON and csv/tsv. Any word on the performance relative to jq and xsv [1]? [1] https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
- snacktaster 4y agoBeing written in Go, I would be very surprised if it's anywhere close to as fast as either of those tools.
- VWWHFSfQ 4y agoI kind of suspected as much. But would still like to see some actual benchmarks.
- 0cf8612b2e1e 4y agoI am all for faster tools, but I am curious as to your use case where the jq speed would be limiting. I only ever cleanup a maximum of a few megabytes at a time, where the jq response is close enough to instant that it has never been a concern.
- VWWHFSfQ 4y agoI typically work with multi-gigabyte JSON and CSV files. I just did a quick test with yq and it's only about 30% faster than just using Python's csv and json libraries. Whereas the same thing is 1,200% faster with jq and xsv. It's just my use-case though, so YMMV.
- encryptluks2 4y agoThere is also another tool named yq that is Python based, and passes everything through jq. The Go-based yq is pretty awesome, but does have some limitations.
- nivekastoreth 4y agothe python yq is by far my preferred utility. i love native jq and the fact it simply wraps it means every thing just works, which is not the experience i’ve had with the tool mentioned in the topic
- ehPReth 4y agohonourable mention for HTML: https://github.com/mgdm/htmlq https://github.com/mgdm/htmlq
- deleted 4y ago[deleted]
- theonemind 4y agogojq works great with YAML and reimplements jq itself in Go. I use gojq with --yaml-input or --yaml-output (or sometimes both) and flip back and forth between JSON and YAML promiscuously and have 100% jq UI compat, which helps because I use jq a lot. First thing I looked at on yq is '-s', which is 'slurp' for jq, but different for yq. Slightly altered semantics would just trip me up, and it seems like you can make a nearly straight bijection between YAML and JSON so you can just do exactly the same things with either one (with some minor exceptions.) https://github.com/itchyny/gojq https://github.com/itchyny/gojq
- 0cf8612b2e1e 4y agogojq does not preserve key order or offer option to sort keys. Which is a non-starter for me. The majority of my jq use is to cleanup API responses for easier human review.
- karmakaze 4y agoThey should be feature requests to gojq. There must be libraries for maps with sorted keys or preserving insertion order to use in place of the std 'map'.
- tenken 4y agoyup. this. It's portable! ... But nerfed.
- quinncom 4y agoYou might enjoy the httpie cli, which is better than curl for testing APIs for many reasons, one of which is automatic pretty printed and colorized text response output. https://httpie.io/cli https://httpie.io/cli
- pokstad 4y agoI believe gojq is also bundled with Benthos. Benthos is a great Swiss army CLI tool for various data manipulations.
- filereaper 4y ago
- CathalMullan 4y agoAnother tool in this space is Dasel[1], which can handle querying/modifying JSON, YAML, TOML, XML and CSV files. [1] https://github.com/TomWright/dasel https://github.com/TomWright/dasel
- AndyKluger 4y agoI do prefer that to jq syntax, as well as alternatives jello and yamlpath.
- EdwardDiego 4y ago`yq` is invaluable when working in the K8s world, love it.
- worldsavior 4y agoLooks cool, but disappointing it's written in Go. Go is fast, but not as fast as Rust or C. I'm sure with large streams, you probably can see the difference in time if it's written in Rust or Go.
- ihucos 4y agoCan we stop this programming language bashing.
- avgcorrection 4y agoAs they say: always measure when it comes to performance. Unless it’s a programming langauge that you don’t like. Then you don’t even have to run the program.
- worldsavior 4y agoI don't understand why my comment is being flagged. I like the language, just thought it can be even faster in other languages, just a thought.
- rgoulter 4y agoThere's a difference between: "I have a <very large> YAML file, and it takes yq <some long time> to parse, whereas <some workflow I use written in rust/c> takes <much less time>". "Go is slower than Rust, the author should have written it in that". It's likely for most YAML documents you encounter, the difference between using Go and using Rust or C is negligible. -- Though, if this isn't true, some numbers would be useful, too. A comment like "Go is a bad language to use" is just a thought; but it's also a low-effort dismissal of something someone has put effort into, and of a tool that's quite useful.
- daurnimator 4y agoI personally find the yq tool from https://github.com/kislyuk/yq https://github.com/kislyuk/yq much more useful: it has all the same options and formats as `jq` (as it's really a wrapper around jq). Rather than the `yq` in the OP here where only partial functionality exists.
- Scubabear68 4y agoLately I have had to do a lot of flat file analysis and tools along these lines have been a godsend. Will check this out. My go to lately has been csvq (https://mithrandie.github.io/csvq/ https://mithrandie.github.io/csvq/). Really nice to be able run complicated selects right over a CSV file with no setup at all.
- scarface74 4y agoSide rant: Every normal yaml processor I’ve tried struggles with CloudFormation. I end up using the cfn-flip command line program/Python module to deal with CFT Yaml.
- raydiatian 4y agoI think my dream is yq but with JSONata and an interactive editor at the command line. I love yq and jq, but imo the core feature they’re missing is queryability. The problem is that afaia the jq syntax doesn’t support things like “where value = x”. There’s another lesser known but imo better querylang called JSONata [0], which is basically a querying and reshaping syntax for structured data. I’m working on this in my spare time but if any know of one that exists so that I don’t have to GO (lang) down a rabbit hole, please do share. [0] https://jsonata.org/ https://jsonata.org/
- slgurtman 4y agojq and yq both support filtering with select operator
- raydiatian 4y agoOkay I must have forgotten this, now that you point it out. But that’s nowhere near as elegant as compact compared with how JSONata handles it. The ideal tool probably lets you choose. Xpath, jq, jsonata, etc
- wiradikusuma 4y agoAnother worthy alternative: https://www.brimdata.io/blog/introducing-zq/ https://www.brimdata.io/blog/introducing-zq/
- mattewong 4y agoAnother for the mix: https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv