10 ms·
Show HN: Gojay – Performant JSON encoder/decoder for Golang
- edhelas 8y agoHow does it compare with encoding the same kind of data in a XML stream?
- laumars 8y agoThat would depend on the XML interface you used. I don't really rate the XML parser in the Go core library much. But I think as much if that is my own personal biased against XML.
- dullgiulio 8y agoEncoding XML to a structure makes little sense, as there is not a direct map between tags, attributes and struct fields. Something like a DOM makes lots more sense, but it is also much more verbose. This is the big reason to prefer JSON over XML.
- laumars 8y agoYeah, that was the point I was hinting at :)
- nieksand 8y agoThe fastest Go JSON parser I know of is https://github.com/buger/jsonparser https://github.com/buger/jsonparser. I've used that one in production quite successfully. I don't see that in the comparative benchmarks for Gojay.
- mintcrate 8y agoThat project is not an encoder/decoder - it just returns values of keys. They are not comparable as the use cases are entirely different.
- nieksand 8y agoWhen it comes to parsing JSON, they are indeed not doing the same thing. Buger's parser is not unmarshalling into an annotated struct. But often that struct isn't your end goal in life, such as when your data is being transformed, dispatched on, or being moved into another yet data structure. In that case unmarshalling to an intermediate structure, while convenient, is also extra computational work. Buger's parser isn't the first thing I'd reach for, but if JSON parsing performance is an issue, ObjectEach() can be used in many places where you'd otherwise invoke json.Unmarshal.
- alex_dev 8y agoHence why the above poster said different use cases and why comparing between them is ridiculous. Using ObjectEach() for every field to reach parity with json.Unmarshal is unconscionable.
- latch 8y ago> I don't see that in the comparative benchmarks for Gojay. Look again, JasonParser is included in some, but not all, of the benchmarks. JasonParser [apparently] allocates no memory, which, to me, seems pretty compelling.
- nieksand 8y agoYou are right. I totally missed it in the first few tables. The lack of allocs for JsonParser was definitely a big win for my project. When processing at high rps (40k msg/sec), GC is near the top of all my performance profiles. Driving allocs down with Buger's parser was very helpful for getting the throughput up.
- Twirrim 8y agoThere's an important distinction between the decoding speed of documents and overall throughput. Wonder what can be done to realistically generate something approximating a real world benchmark to cover the throughput scenario.
- zerr 8y agoWhat's the project? Seems like a language with a GC is not a good choice, is it?
- nieksand 8y agoEssentially a gloried batcher / data transformer consuming from Kafka. Even with the GC overhead, I will still able to hit my performance goal of 40k msg/sec per AWS c3.4xl. In a vacuum I would have picked Rust over Go for this. But Go is widely used at my work, whereas I'm the only Rust guy.
- woolvalley 8y agoHow much of a difference you think rust would of made? Have you made something similar?
- tomohawk 8y agoCheck the source to the benchmarks https://github.com/francoispqt/gojay#benchmark-results https://github.com/francoispqt/gojay#benchmark-results
- segmondy 8y agoIs it? Kubernete switched from the native go Json parser to codec for performance increase. http://ugorji.net/blog/go-codecgen http://ugorji.net/blog/go-codecgen
- laumars 8y agoThis looks perfect for a project I'm working on which is a UNIX / Linux $SHELL that makes heavy use of JSON pipelining.
- agumonkey 8y agostructured shell, go on
- zbentley 8y agohttps://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell
- laumars 8y agoPowershell is object oriented and, in my opinion at least, an absolute pig for doing quick one liners (overly verbose syntax, pipelines bork if types mismatch even when the data getting passed is still essentially just textual). I wanted something that was still in the same realm of typical UNIX shells (even with Bash comparability where sensible) but with an awareness of complex data formats. However everyone has their own opinions and preferences. I wrote my shell to scratch my own personal itch and if others like / use it that is a bonus.
- laumars 8y agohttps://github.com/lmorg/murex https://github.com/lmorg/murex It's not without its bugs but I now use it everyday as my primary shell. Documentation is about 30% there but that's something I'm actively working on at the moment. Happy to receive any feedback, possitive or negative :)
- tptacek 8y agoPerformance is a real issue for Go's standard JSON, but this is a lot of extra boilerplate code to have to write (I'd probably codegen most of this if I had to), so I'd assume the reasonable strategy would be to implement with encoding/json, profile, and then just GoJay the hotspots.
- segmondy 8y agoThat's exactly what go-codec does
- laumars 8y agoIt looks like this has a Marshal and Unmarshal function just like it's core library counterpart. So I'd guess you might be able to use this as a drop in replacement. However I'm yet to prove that theory.
- tptacek 8y agoencoding/json's Marshal and Unmarshall uses reflect and struct tags. This library doesn't: you have to define a function to make your struct satisfy an interface.
- phamilton 8y agoMacros would solve this problem nicely if they we're available. I sure wish first class macros were on the golang roadmap.
- tptacek 8y agoIf you want to code in a language with macros, pick a language with macros. They tend not to look like Go (or C, or C++), because part of what makes macros effective is that coding in those languages is pretty close to working with their ASTs. Writing a Lisp-like macro for Go sounds like it would actually be more annoying than simply writing a codegen for it.
- 8y ago
- BillinghamJ 8y agoWould be handy if it included a utility to generate the (un)marshaling functions, so that can just be done as a build step. So far we’ve used https://github.com/mailru/easyjson https://github.com/mailru/easyjson for that, but the code generation step is unbelievably slow - multiple minutes for about 100 structs.
- francoisllm 8y agoIt's actually the next milestone, a generator for structs, maps and slices. Until now the goal was to make it ready for some high traffic services in production. These traffic receive JSON with chinese, vietnamese character so we needed to make sure it works well first (boilerplate was not an issue).
- anothergoogler 8y agoGo is pretty good about Unicode, did you run into issues with Chinese and Vietnamese with standard encoding/json package or with other third-party Go JSON libraries?
- francoispqr 8y agoNope, it's quite easy to integrate unicode parsing :) just need to check for "\u1234" strings in JSON as it is valid, also need to check for utf16 surrogates. Standard package does it already just had to implement it in Gojay.
- dtolnay 8y agoTo see how this fares against native code, I ported their benchmark to Rust's JSON library https://serde.rs/ https://serde.rs/. Disclaimer: I am a maintainer of Serde. Numbers and graphs: https://github.com/serde-rs/json-benchmark/tree/gojay#serde https://github.com/serde-rs/json-benchmark/tree/gojay#serde Rust source code: https://github.com/serde-rs/json-benchmark/blob/gojay/src/lib.rs https://github.com/serde-rs/json-benchmark/blob/gojay/src/li... TL;DR GoJay ranges from 20% slower to 2.7x slower depending on workload.
- IshKebab 8y agoGo is "native code" but anyway nice work!
- Groxx 8y agoAs always, it's a range. Go has a runtime which you cannot control which schedules your code when it feels like it and a garbage collector. Rust has neither. Sure, Go isn't parsing and interpreting its files at runtime. But neither does Python, so I'm not sure that's a meaningful line to draw.
- abiox 8y ago> Sure, Go isn't parsing and interpreting its files at runtime afaik, referring to something as "native code" just indicates it isn't compiled to/executing as bytecode. not anything to do with gc, runtime, etc.
- Groxx 8y agoDepends on who you ask. As demonstrated by literally every commenter in this thread so far. Besides, define bytecode. Once it has run through a JIT and diverged from the on-disk representation, is it now native? How is it distinguishable from a binary that detects your CPU architecture and executes different branches of code? What about other forms of self-modifying code? There are cases where an easy argument can be made (e.g. Java, which has a separately-supplied VM to run your bytecode... but then where's the line with DLLs?), but there isn't an unambiguous line in the sand here. At any line you can produce a new system that straddles it (e.g. a JAR which ships with its own VM. the VM is native code, is the binary now native or not?), and often there are already widely-used examples.
- krylon 8y agoI have been using ffjson[1] in a couple of private projects, which generates code to encode/decode data, which supposedly is faster than the json library from Go's standard library. Has somebody any first-hand expertise how this compares to the competition? (I could, of course, do my own benchmarks, but so far I have not had any performance issues on this end, so it has not been a pressing need. The only problem I have run into with ffjson is that various linters will bark at the generated code, but again, that has not been a big problem.) [1] https://github.com/pquerna/ffjson https://github.com/pquerna/ffjson
- pquerna 8y ago(ffjson author here) The main feature that ffjson has that most of the non-stdlib JSON libraries is stdlib compatibility. Eg, the same struct-tags and interfaces used in stdlib are used by ffjson. It's just trying move most of the reflection / allocations / etc to a `go generate` step vs runtime. If you abandon trying to stay consistent with the stdlib JSON, and make new APIs/interfaces for propagating the encoder or decoder state, as Gojay has, it will undoubtedly be faster than ffjson or stdlib.
- krylon 8y agoThank you very much!
- throwme_1980 8y agoyet another boiler plate library, native lib works fine for 99.99% of use cases. for edge cases i wouldn't be using go anyways if speed was my thing.
- segmondy 8y agoHow does this compare to http://ugorji.net/blog/go-codecgen http://ugorji.net/blog/go-codecgen ?
- francoispqr 8y agoIt's faster :)
- atefatef 8y agoszss