6 ms·

Show HN: ffjson: faster json serialization in Go

83 points by pquerna 13y ago

hannibalhorn 13y ago
The implementation of this is pretty interesting, in that it generates code that imports your code, compiles it, and then uses reflection to generate the serialization code. And in the end, that worked out better for the author than using the AST.
- lucian1900 13y ago
  If anything, that would be an indication that the tooling is nowhere near good enough. Any reflection should be doable entirely with data the compiler has anyway.
  - timtadh 13y ago
    Actually there is a tool for this. I think the author was simply un-aware of it. It is called oracle* and is able to answer this typing question and many more. *: https://code.google.com/p/go/source/browse/cmd/oracle/main.go?repo=tools https://code.google.com/p/go/source/browse/cmd/oracle/main.g...
  - hierro 13y ago
    The tooling is excellent, but the author wasn't aware of it. In fact, the stdlib and the go.types repository provide almost anything required to write a front-end for a Go compiler.
kevrone 13y ago
Seems like this approach might also work for building type-specific collection implementations as well!
knodi 13y ago
This is a good example of why everyone should avoid using reflect package as much as possible. I use reflect for quick development and then remove it before production roll out.
- Dobbs 13y ago
  I've had to use reflect due to missing functions in an upstream API. We could have forked the upstream package or used reflect in one function. We decided to put in a patch and use reflect until it is accepted.
haberman 13y ago
As someone who has been working on parsing/serialization for many years, I can absolutely confirm that generating schema-specific code will beat generic code almost every time. The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. And you pay this compile-time overhead for every message type you want to manipulate. The pain of this generated code was one of Kenton Varda's motivations for creating Cap'n Proto after working on Protocol Buffers for years. Unlike Protocol Buffers, Cap'n Proto doesn't need to generate parsing/serialization code because its serialization format also works as an in-memory format. I have taken a somewhat different approach to the problem, with my Protocol Buffer / JSON / etc. serialization framework upb (https://github.com/haberman/upb https://github.com/haberman/upb). Instead of using static code generation, I use a JIT approach where I generate specialized code on-the-fly. This approach is particularly important when you are wrapping the library in an interpreted language like Python/Ruby/Lua/JavaScript, because users of these languages don't have a compile cycle at all, so adding one is a large inconvenience. My library isn't ready for prime time but I'm hoping to have something ready to use this year.
- throwaway812 13y ago
  The last time I looked at upb, it only supported parsing, which was a deal-breaker. Have you implemented serializing since then? It's hard to tell from the git log, which is mostly "dump from google internal sources XX-YY-ZZZZ". I'm very excited about upb! Thanks for your work on it over the years. Do you have any tasks that an outside contributor could help with? Thanks!
  - haberman 13y ago
    Still no serialization, sorry. This is getting close to being my top priority though. It's a hard problem to design the kind of generic visitor I want while being speed-competitive with the fastest stuff that's out there. Probably can't use a lot of outside help, but thanks for asking. What's taking so long is a very intense process of refining the design. I am often reminded of this bit from "Worse is Better": "The right thing takes forever to design, but it is quite small at every point along the way." This is exactly the approach I am taking. It's taken forever to design, but over time it becomes more and more capable without the core growing in size very much. I hope that this intensive design work will pay off soon. Thanks for your interest and I hope to soon be able to reward it with a highly compelling parsing/serialization experience!
hierro 13y ago
This is way more complicated that it needs to be. Use code.google.com/p/go.tools/go/types to process the AST and you'll get basically the same information that the compiler sees. With that you can generate the code pretty easily. For comparison, our JSON code generator implementation is just ~350 lines and supports having different JSON representations for the same type, which vary depending on the container type. Also, if you want to make the serialization faster you need to understand exactly what makes encoding/json slow (hint: is not only reflect) and remove all reasonable bottlenecks. You state that megajson does not support the MarshallJSON interface like that's a bad thing, but I'm pretty sure that's deliberate because it's indeed a feature. When encoding/json encounters a type which implements MarshalJSON it does the following: 1 - Call MarshallJSON to obtain its JSON representation as []byte 2 - Validate the produced JSON using a slower-than-the-bad's-guy-horse function based state machine which processes each character individually 3 - Copy the []byte returned by MarshallJSON to its own buffer Unsurprisingly (after reading encoding/json's code, of course) having a MarshalJSON method is way slower than letting encoding/json use reflection if the JSON you're generating is anything but trivial and without almost any nesting, because it avoids extra allocations, copies and the validation step.
chimeracoder 13y ago
Shameless plug, but this looks like the exact inverse of gojson[0], which generates code (struct definitions) based on sample JSON. I originally wrote it when writing a client library to interface with a third-party API; it saves a lot of effort compared to typing out struct definitions manually, and a lot of type assertions compared to using map[string]interface{} everywhere. [0] https://github.com/ChimeraCoder/gojson https://github.com/ChimeraCoder/gojson
mwsherman 13y ago
On the .Net side, Jil is doing similar, creating a custom static serializer per type. It’s able to do the code generation at runtime by emitting IL: https://github.com/kevin-montrose/Jil https://github.com/kevin-montrose/Jil
United857 13y ago
Looking at the name, at first I thought ffmpeg author and general programming god Fabrice Bellard had come up with it.
otterley 13y ago
Good stuff, there. Feature request: optionally emit canonicalized (key-sorted) JSON.
AYBABTME 13y ago
I've made a library this weekend that doesn't need code generation to achieve 2x improvement[1] over the standard library. While the OP only implemented the encoding part, I only implemented the decoding part =): https://github.com/aybabtme/fatherhood https://github.com/aybabtme/fatherhood So I guess they overlap nicely in that. [1]:https://github.com/aybabtme/fatherhood#performance https://github.com/aybabtme/fatherhood#performance
eikenberry 13y ago
You can get near the same speedup by just avoiding reflection. You unmarshal into an interface{} then pull the data out manually using type assertions as necessary. In my last project I think I got about a 1.6-1.7 speedup this way.
- alecthomas 13y ago
  Nice hack (in the hacker sense), but not exactly convenient.

83 points by pquerna 13y ago

The implementation of this is pretty interesting, in that it generates code that imports your code, compiles it, and then uses reflection to generate the serialization code. And in the end, that worked out better for the author than using the AST.

lucian1900 13y ago
If anything, that would be an indication that the tooling is nowhere near good enough. Any reflection should be doable entirely with data the compiler has anyway.
- timtadh 13y ago
  Actually there is a tool for this. I think the author was simply un-aware of it. It is called oracle* and is able to answer this typing question and many more. *: https://code.google.com/p/go/source/browse/cmd/oracle/main.go?repo=tools https://code.google.com/p/go/source/browse/cmd/oracle/main.g...
- hierro 13y ago
  The tooling is excellent, but the author wasn't aware of it. In fact, the stdlib and the go.types repository provide almost anything required to write a front-end for a Go compiler.

kevrone 13y ago

Seems like this approach might also work for building type-specific collection implementations as well!

knodi 13y ago

This is a good example of why everyone should avoid using reflect package as much as possible. I use reflect for quick development and then remove it before production roll out.

Dobbs 13y ago
I've had to use reflect due to missing functions in an upstream API. We could have forked the upstream package or used reflect in one function. We decided to put in a patch and use reflect until it is accepted.

haberman 13y ago

As someone who has been working on parsing/serialization for many years, I can absolutely confirm that generating schema-specific code will beat generic code almost every time. The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. And you pay this compile-time overhead for every message type you want to manipulate. The pain of this generated code was one of Kenton Varda's motivations for creating Cap'n Proto after working on Protocol Buffers for years. Unlike Protocol Buffers, Cap'n Proto doesn't need to generate parsing/serialization code because its serialization format also works as an in-memory format. I have taken a somewhat different approach to the problem, with my Protocol Buffer / JSON / etc. serialization framework upb (https://github.com/haberman/upb https://github.com/haberman/upb). Instead of using static code generation, I use a JIT approach where I generate specialized code on-the-fly. This approach is particularly important when you are wrapping the library in an interpreted language like Python/Ruby/Lua/JavaScript, because users of these languages don't have a compile cycle at all, so adding one is a large inconvenience. My library isn't ready for prime time but I'm hoping to have something ready to use this year.

throwaway812 13y ago
The last time I looked at upb, it only supported parsing, which was a deal-breaker. Have you implemented serializing since then? It's hard to tell from the git log, which is mostly "dump from google internal sources XX-YY-ZZZZ". I'm very excited about upb! Thanks for your work on it over the years. Do you have any tasks that an outside contributor could help with? Thanks!
- haberman 13y ago
  Still no serialization, sorry. This is getting close to being my top priority though. It's a hard problem to design the kind of generic visitor I want while being speed-competitive with the fastest stuff that's out there. Probably can't use a lot of outside help, but thanks for asking. What's taking so long is a very intense process of refining the design. I am often reminded of this bit from "Worse is Better": "The right thing takes forever to design, but it is quite small at every point along the way." This is exactly the approach I am taking. It's taken forever to design, but over time it becomes more and more capable without the core growing in size very much. I hope that this intensive design work will pay off soon. Thanks for your interest and I hope to soon be able to reward it with a highly compelling parsing/serialization experience!

hierro 13y ago

This is way more complicated that it needs to be. Use code.google.com/p/go.tools/go/types to process the AST and you'll get basically the same information that the compiler sees. With that you can generate the code pretty easily. For comparison, our JSON code generator implementation is just ~350 lines and supports having different JSON representations for the same type, which vary depending on the container type. Also, if you want to make the serialization faster you need to understand exactly what makes encoding/json slow (hint: is not only reflect) and remove all reasonable bottlenecks. You state that megajson does not support the MarshallJSON interface like that's a bad thing, but I'm pretty sure that's deliberate because it's indeed a feature. When encoding/json encounters a type which implements MarshalJSON it does the following: 1 - Call MarshallJSON to obtain its JSON representation as []byte 2 - Validate the produced JSON using a slower-than-the-bad's-guy-horse function based state machine which processes each character individually 3 - Copy the []byte returned by MarshallJSON to its own buffer Unsurprisingly (after reading encoding/json's code, of course) having a MarshalJSON method is way slower than letting encoding/json use reflection if the JSON you're generating is anything but trivial and without almost any nesting, because it avoids extra allocations, copies and the validation step.

chimeracoder 13y ago

Shameless plug, but this looks like the exact inverse of gojson[0], which generates code (struct definitions) based on sample JSON. I originally wrote it when writing a client library to interface with a third-party API; it saves a lot of effort compared to typing out struct definitions manually, and a lot of type assertions compared to using map[string]interface{} everywhere. [0] https://github.com/ChimeraCoder/gojson https://github.com/ChimeraCoder/gojson

mwsherman 13y ago

On the .Net side, Jil is doing similar, creating a custom static serializer per type. It’s able to do the code generation at runtime by emitting IL: https://github.com/kevin-montrose/Jil https://github.com/kevin-montrose/Jil

United857 13y ago

Looking at the name, at first I thought ffmpeg author and general programming god Fabrice Bellard had come up with it.

otterley 13y ago

Good stuff, there. Feature request: optionally emit canonicalized (key-sorted) JSON.

AYBABTME 13y ago

I've made a library this weekend that doesn't need code generation to achieve 2x improvement[1] over the standard library. While the OP only implemented the encoding part, I only implemented the decoding part =): https://github.com/aybabtme/fatherhood https://github.com/aybabtme/fatherhood So I guess they overlap nicely in that. [1]:https://github.com/aybabtme/fatherhood#performance https://github.com/aybabtme/fatherhood#performance

eikenberry 13y ago

You can get near the same speedup by just avoiding reflection. You unmarshal into an interface{} then pull the data out manually using type assertions as necessary. In my last project I think I got about a 1.6-1.7 speedup this way.

alecthomas 13y ago
Nice hack (in the hacker sense), but not exactly convenient.