4 ms·
I'm hitting the same kind of problem writing serde code in go, trying to marshal/unmarshal data to/from structs. The biggest problem is how SLOW reflect is, req
by kstenerud 6y ago
I'm hitting the same kind of problem writing serde code in go, trying to marshal/unmarshal data to/from structs. The biggest problem is how SLOW reflect is, requiring all sorts of complicated caching to ensure the least number of reflect calls are made in aggregate (have a look at the JSON codec in the standard library for an idea).
The other terrible part is io.Writer. Since these are interfaces, the compiler can't tell what the actual implementation will do with the []byte parameter, even though we humans know perfectly well. So if you try to write from a stack allocated buffer, the compiler will hoist it into the heap, every time (and allocations are one of the slowest parts of go). I started off by writing a complicated heap-based buffering mechanism until finally I said to myself "nah... it can't be THAT bad - they've got to have some kind of mitigation for this!" So I changed everything to very simple writer.Write, and the serialization performance dropped by more than half (from 1700 ns/op to 3500 ns/op). Allocations went from 5 per op to 37 per op!
Would be nice to have a "nostore" modifier for slices to tell the compiler that the receiver is not allowed to store it (thus preventing this constant stack-to-heap copying).
- marcus_holmes 6y agoI've arrived at the opinion that any use of reflect is a red flag that the author is trying to do something that Go wasn't designed to do and the code will probably be extremely flaky. It's not a 100% match, but it's a good heuristic in most cases. As my experience with Go has increased, the number of times I need to touch reflect has decreased.
- christophilus 6y agoHow do you marshal/unmarshal JSON?
- nvarsj 6y agoYou write a custom marshal/unmarshal method for every type, of course.
- kstenerud 6y agoThat's what it does already. But you can't dynamically create a value of a decoded type without reflection, nor can you examine an arbitrary object for encoding without reflection.
- christophilus 6y agoWait. Is that what the built in utility does? I thought it was reflection based. How does it do this without generators?
- kstenerud 6y agoIt's a mix of cached builders and reflection. It uses reflection to build an appropriate builder object for that specific type. Note that it is still using reflection in the builder though.
- marcus_holmes 6y agoI wasn't really referring to the std lib. Though, to be honest, given the amount of hassles I've had marshaling and unmarshaling json in Go, I could well include it ;)