3 ms·
What always trips me up about capnproto is that it's billed as a serialization library, but what it is is an in-memory storage layout, and "serialization" is mo
by makmanalp 9y ago
What always trips me up about capnproto is that it's billed as a serialization library, but what it is is an in-memory storage layout, and "serialization" is mostly just dumping memory into a file, right? (which is cool)
What confuses me is, then what are the costs of migrating to this system? Am I essentially dumping my programming language's object model for my capnproto implementation's? When can this be annoying? Or does it vary from implementation to implementation?
In a similar tangent - how similar is this to apache arrow, not because of the columnar analytics part, but could I expect to just dump a bunch of data in shared memory and read it from another process to eliminate IPC serialization/copy costs?
- kentonv 9y agoGenerally I'd recommend using Cap'n Proto in much the same way as you'd use Protobuf. It's not intended that your in-memory state be in Cap'n Proto objects, only the messages you intend to transmit, or data stored on disk.
- makmanalp 9y agoWait, but then aren't you merely shifting the serialization cost instead into building capnproto objects? (and perhaps that's more efficient somehow?) It seemed to make more sense when you already have your data as capnproto objects, versus creating objects only to send and discard them, which is similar to regular old serialization again.
- kentonv 9y agoWith Protobuf, you still have the cost of constructing the objects, and the cost of then serializing them. Cap'n Proto removes the latter cost. Serializing is generally the much more expensive step. It also turns out Cap'n Proto reduces the building cost by a fair amount, because the arena-style allocation needed to support zero-copy output also happens to be a lot cheaper and more cache-friendly, but that's somewhat of an accident. For message-passing scenarios, Cap'n Proto is an incremental improvement over Protobufs -- faster, but still O(n), since you have to build the messages. For loading large data files from disk, though, Cap'n Proto is a paradigm shift, allowing O(1) random access.
- makmanalp 9y agoI understand much better now - thank you very much! And congratulations on the release!
- na_ka_na 9y agoWhy is not intended that in-memory state be in Cap'n Proto / Protobuf objects? What are the down-sides?
- rkv 9y agoAfter integrating protobufs in my application for messaging I decided to use a separate schema for storing the current state of the program. Ie. When state changes, the protobuf is updated and written to disk. When the program restarts, the state file is loaded into memory. I have not run into any problems doing this. Edit: Your question is addressed here: https://news.ycombinator.com/item?id=14249367 https://news.ycombinator.com/item?id=14249367
- na_ka_na 9y agoThanks, but I don't follow how that comment addresses my question. Is it that cost of constructing Cap'n Proto / Protobuf is quite a bit higher than constructing objects defined natively?
- deleted 9y ago[deleted]
- kentonv 9y ago> Is it that cost of constructing Cap'n Proto / Protobuf is quite a bit higher than constructing objects defined natively? I discussed in more detail in reply to your first post, but just to be really clear on this: No. In fact, for deeply-nested object trees, constructing a Cap'n Proto object can often be cheaper than a typical native object since it does less memory allocation. However, there are some limitations -- see my other reply. (Constructing Protobuf objects, meanwhile, will usually be pretty much identical to POCS, since that's essentially what Protobuf objects are.) There is a common myth that Cap'n Proto "just moves the serialization work to object-build time", but ultimately does the same amount of work. This is not true: Although you could describe Cap'n Proto as "doing the serialization at object build time", the work involved is not significantly different from building a regular in-memory object.