4 ms·

Big fan of protobuf here. Awesome work. We did something along those lines for our product except we used Boost.Fusion to generate the serialization code which

Big fan of protobuf here. Awesome work. We did something along those lines for our product except we used Boost.Fusion to generate the serialization code which enables us to skip the interface compilation phase altogether. Basically you have any structure, you Boost.Fusion adapt it and bam!, cross-platform binary serialization at compile-time. The best is that this approach works for everything. Want JSON? Bam! You got JSON. One trick we used is to zero-copy the big chunks of memory. Don't know if that's the case here. It's tricky though because if you do asynchronous I/O later you have to make sure the original buffer stays alive long enough. (ps: getters and setters, this is soooo last century ;) )

kentonv 14y ago
Thanks! Yeah, getters and setters are old school, but still necessary in C++, especially in this case since the underlying data isn't necessarily in native format. In more modern languages I'd hope to use properties. :) I do hope to write a JSON transcoder as well, so that for those cases where you really need to send/receive JSON, you can do that, while still using the same protocol spec.
- skybrian 14y ago
  An important gotcha for JSON: it's not enough just to follow the JSON spec. If you want the data to be usable in a browser, you also have to take into account JavaScript's limitations. In particular, the JSON spec says that arbitrary-precision numbers are allowed, but if you want to round-trip a 64-bit integer without losing precision, you'd better put quotes around it.
- onedognight 14y ago
  > Yeah, getters and setters are old school, but still necessary in C++ Why isn't operator= usable in this case?
  - kentonv 14y ago
    Because you need an lvalue to assign to, which turns out to be pretty painful when the underlying data is not (quite) in native format. The lvalue can't just be a reference to the underlying data, so instead you need to set up some parallel set of objects somewhere that wrap the data, and each of these objects probably needs to contain a pointer, so they may end up being bigger than the actual data they are wrapping. It really isn't worth the effort.
- shin_lao 14y ago
  I wasn't thinking about properties. We've used a C++er approach (I feel the stench of Java in you ;) ): objects are stored natively in memory and only when serialized - if needed - transcoding is applied just before, on the flight, on local copies. Since the objects for which representation is different are small (actually, only integers), it's insanely fast and almost no memory is wasted. It's not quite "0 ms transcoding" but quite and most importantly it's 0 ms read/write access time (you spend more time accessing the object in memory than serializing it).

kentonv 14y ago

Thanks! Yeah, getters and setters are old school, but still necessary in C++, especially in this case since the underlying data isn't necessarily in native format. In more modern languages I'd hope to use properties. :) I do hope to write a JSON transcoder as well, so that for those cases where you really need to send/receive JSON, you can do that, while still using the same protocol spec.

skybrian 14y ago
An important gotcha for JSON: it's not enough just to follow the JSON spec. If you want the data to be usable in a browser, you also have to take into account JavaScript's limitations. In particular, the JSON spec says that arbitrary-precision numbers are allowed, but if you want to round-trip a 64-bit integer without losing precision, you'd better put quotes around it.
onedognight 14y ago
> Yeah, getters and setters are old school, but still necessary in C++ Why isn't operator= usable in this case?
- kentonv 14y ago
  Because you need an lvalue to assign to, which turns out to be pretty painful when the underlying data is not (quite) in native format. The lvalue can't just be a reference to the underlying data, so instead you need to set up some parallel set of objects somewhere that wrap the data, and each of these objects probably needs to contain a pointer, so they may end up being bigger than the actual data they are wrapping. It really isn't worth the effort.
shin_lao 14y ago
I wasn't thinking about properties. We've used a C++er approach (I feel the stench of Java in you ;) ): objects are stored natively in memory and only when serialized - if needed - transcoding is applied just before, on the flight, on local copies. Since the objects for which representation is different are small (actually, only integers), it's insanely fast and almost no memory is wasted. It's not quite "0 ms transcoding" but quite and most importantly it's 0 ms read/write access time (you spend more time accessing the object in memory than serializing it).