5 ms·
"gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like t
by simonw 4y ago
"gojq does not keep the order of object keys" is a bit disappointing.
I care about key order purely for cosmetic reasons: when I'm designing JSON APIs I like to put things like the "id" key first in an object layout, and when I'm manipulating JSON using jq or similar I like to maintain those aesthetic choices.
I know it's bad to write code that depends on key order, but it's important to me as a way of keeping JSON as human-readable as possible.
After all, human readability is one of the big benefits of JSON over various other binary formats.
- silverwind 4y agoAgree, this is deterring me from this tool. Many languages/tools nowadays guarantee object key order which is convenient in many ways.
- lapser 4y agoFor what it's worth, JSON Objects are not guaranteed to be ordered. Maps in many different languages are implemented without an order.
- hyperpallium2 4y agojq used to do this, but changed to preserve key order.
- haasted 4y agoI bet it's an artifact of Go having a randomized iteration order over maps [0]. Getting a deterministic ordering requires extra work. [0] https://stackoverflow.com/questions/9619479/go-what-determines-the-iteration-order-for-map-keys https://stackoverflow.com/questions/9619479/go-what-determin...
- simonw 4y agoI used to have the exact same problem with Python, until Python 3.7 made maintaining sort order a feature of the language: https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/ https://softwaremaniacs.org/blog/2020/02/05/dicts-ordered/
- c2h5oh 4y agoGo actually went in the other direction for a bunch of reasons (e.g. hash collision dos) and made key order quasi-random when iterating. Small maps used to maintain order, but a change was made to randomize that so people didn't rely on that and get stung when their maps got larger: https://github.com/golang/go/issues/6719 https://github.com/golang/go/issues/6719
- tialaramex 4y agoRight, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. It's like if you did such a bad job making a drag racer that the street legal model of the same car was substantially faster over a quarter mile despite also having much better handling and reliability. In some communities the reaction would have been to write a good unordered dict which would obviously be even faster, but since nobody is exactly looking for the best possible performance from Python, they decided that ordered behaviour was worth the price, and it's not as though existing Python programmers could complain since it was faster than what they'd been tolerating previously. Randomizing is the other choice if you actually want your maps to be fast and want to resist Hyrum's law, but see the absl experience - they initially didn't bother to randomize tiny maps but then the order of those tiny maps changed for technical reasons and... stuff broke. Because hey, in testing I made six of this tiny map, they always had the same order therefore (ignoring the documentation imploring me not to) I shall assume the order is always the same...
- alecthomas 4y ago> Right, the startling thing about Python's previous dict was that it was so terrible that the ordered dict was actually significantly faster. I've never heard that before and it would be really surprising, given that Python's builtin dict is used for everything from local symbol to object field lookup. Do you have more information?
- aaronbee 4y ago
- deleted 4y ago[deleted]
- vips7L 4y agoDoes Go not have more than one Map implementation in the standard library?
- deleted 4y ago[deleted]
- esprehn 4y agoIt does not. Maps are not even a real interface you can implement, it's compiler magic encoded in the language spec: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implements-maps-efficiently-without-generics https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem... This is all fallout of not having generics.
- xmonkee 4y agoI fucking hate this so much. Honestly Go isn't a bad language, but I dunno why these kind of things just piss me off.
- hxtk 4y agoIt seems as though a lot of people view it as hypocritical, e.g., generics for me but not for thee (dated example since there are now generics for everyone). The fact that they needed to make a map a part of the language in order to allow it to be generic and statically-typed proves that generics are useful and should therefore have been a language feature much earlier than they became one. There are a variety of things that the standard library or compiler deal with using weird workarounds that seem to indicate missing language features. The thing is, the features are only "missing" if the language is designed to do the things those features permit. So the counterargument is that Go is a very opinionated language designed to do solve a few classes of problem very easily, like writing database-backed web services, and the reason the standard library or compiler teams have to do weird hacks at times is because Go wasn't made for writing those things, and designing to make those use cases easy would pollute the language from the perspective of someone using it for its intended purpose.
- Someone 4y agoNo, it isn’t. “gojq does not keep the order of object keys” isn’t about ordering keys consistently across runs, it’s about keeping them in the order of the input file.
- akpa1 4y agoWhich it can't do because, as mentioned, Go randomly iterates over maps. That's the data structure that most would use to load arbitrary input files into the program.
- Someone 4y agoIf you have a hammer in your toolbox, it doesn’t mean you have to use it in every job. It golangs maps don’t do what you want, pick a different data structure. This is like claiming that, because updating native integers isn’t guaranteed to be atomic in a language, you can’t do multi-threaded programming.
- rfiat 4y agoGP is correct, per the README: > gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have keys_unsorted function and --sort-keys (-S) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. And later in the same file: gojq does not support some functions intentionally; <snip> --sort-keys, -S (sorts by default because map[string]interface{} does not keep the order),
- Someone 4y agoNo, they aren’t. haasted replied to simonw’s > "gojq does not keep the order of object keys" is a bit disappointing with > “I bet it's an artifact of Go having a randomized iteration order over maps. Getting a deterministic ordering requires extra work.” But deterministic iteration order doesn’t imply that the order of keys is kept the same. There are map implementations that keep iteration follow insertion order, but the canonical map does not guarantee that. https://en.wikipedia.org/wiki/Associative_array#Hash_table_implementations https://en.wikipedia.org/wiki/Associative_array#Hash_table_i...: “The most frequently used general purpose implementation of an associative array is with a hash table: an array combined with a hash function that separates each key into a separate "bucket" of the array“ Such implementations iterate over maps in order of hash value (and hash collisions may or may not follow (reverse) insertion order)
- zxcvbn4038 4y agoYeah, this is a deal breaker. While technically the key order doesn’t matter, in the real world it really does matter. People have to read this stuff. People have to be able to differentiate between actual changes and stuff moving around just because. Luckily it’s a solved problem and you can write marshalers that preserve order, but it’s extra work and generally specific to an encoding format. It would be nice to have ordered maps in the base library as an option.
- cerved 4y agointo it's not about code, it's about predicable and consistent layout so that you can easily diff
- 41b696ef1113 4y agoIt does not even provide a `--sort-keys` option. That's like 90% of the the reason I ever lean on jq - to standardize API output for my human brain.
- latchkey 4y agoBest 3rd party Map library that I've found. https://github.com/cornelk/hashmap https://github.com/cornelk/hashmap
- Lurkars 4y agoThis. For one project, I even write a tool to reorder keys to a specific order. And of course this has no technically reason. But I used JSON here for the human readability and that non-technical people have best changes to understand and change the data. And therefore starting with id and name on top is important than with an huge array of data.
- xorcist 4y agoOrdered keys in json is not only for cosmetic reasons. If this ever touches disk you want the ability to diff them or stash them in git without the whole file changing with every update.