4 ms·

I have to question the value of avoiding deserialization. With no type tag space reserved in structures or pointers, are there any strongly-typed language runti

I have to question the value of avoiding deserialization. With no type tag space reserved in structures or pointers, are there any strongly-typed language runtimes (i.e., not C) which can actually use this as the definitive in-memory representation? If you have to wrap proxy objects around a Cap'n Proto blob, the cost of continual indirections will quickly exceed the cost of just deserializing into native objects. I'm glad to see efforts to move away from encoding the schema (ids and types) into every field of every message, though. I can't remember the last time I found a use for self-describing data of a type that was completely unknown when I wrote the code. And the suggested packing format looks feasible, where too many protocol designers think everyone can afford to gzip everything by default (and make a general-purpose library re-deduce where the redundancy is in your schema, when certain bits are inevitably constant for every message you could ever send).

kentonv 14y ago
This will tend to depend on the details of the language. In Java, there is java.nio.ByteBuffer which (I think) can basically do everything that is done with pointer arithmetic in the C++ implementation. In, say, Python, it may make more sense to decode the message to Py objects upfront. You could use Python's struct packing library to do this fairly efficiently, and you could lazily decode sub-objects. If you are using Python, you probably aren't going for blazing speed anyway. Another possibility in many languages is to write bindings that call into C++, though that obviously has a bunch of trade-offs. Finally, if you're communicating with browser-side Javascript, you probably just want to transcode the object to/from JSON on the server side. I intend to provide libraries that make that easy.
barrkel 14y ago
Avoiding serialization is a big win for short-lived object graphs like those seen when processing an http request. About 6 years ago, I developed a similar technique used in a .net app server to minimize server load for relatively chatty AJAX apps. It's good to see the same idea polished up further coming around again.

kentonv 14y ago

This will tend to depend on the details of the language. In Java, there is java.nio.ByteBuffer which (I think) can basically do everything that is done with pointer arithmetic in the C++ implementation. In, say, Python, it may make more sense to decode the message to Py objects upfront. You could use Python's struct packing library to do this fairly efficiently, and you could lazily decode sub-objects. If you are using Python, you probably aren't going for blazing speed anyway. Another possibility in many languages is to write bindings that call into C++, though that obviously has a bunch of trade-offs. Finally, if you're communicating with browser-side Javascript, you probably just want to transcode the object to/from JSON on the server side. I intend to provide libraries that make that easy.

barrkel 14y ago

Avoiding serialization is a big win for short-lived object graphs like those seen when processing an http request. About 6 years ago, I developed a similar technique used in a .net app server to minimize server load for relatively chatty AJAX apps. It's good to see the same idea polished up further coming around again.