5 ms·

Interesting that Cap'n Proto[1] is not even mentioned given that this library has similar if not the same goals and lineage. [1] http://kentonv.github.io/capnp

Interesting that Cap'n Proto[1] is not even mentioned given that this library has similar if not the same goals and lineage. [1] http://kentonv.github.io/capnproto/ http://kentonv.github.io/capnproto/

srean 12y ago
I dont know full details of Cap'n Proto, so please correct me if I am wrong. My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. Would that be a reasonable summary of Cap'n Proto ? Guessing from the name I expected FlatBuffers to be a unified file format for on-disk and on-RAM presence, so that you can just mmap the disk file and you have the data-structure in memory. No need to translate between on disk and on disk representations. I am not claiming that it is so, this is what I expected it to be, given the name. But after spending some time looking around I am still confused if that's what it is. Would appreciate confirmation either way. Have to look at the code when I get around to it. I have an use case in mind for a tree of appendable/insertable matrices. HDf5 comes close, but isnt quite what I am looking for. EDIT @vertex-four thanks for the clarification much appreciated.
- vertex-four 12y ago
  > My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. No. It's a serialisation format that's specifically designed so that data structures in this format can be used as-is by programs, without unnecessarily copying data into or out of the format to use it. The RPC layer is then built on top of that, but the format could also be used for mmapped files, etc.
  - bastih 12y ago
    I always felt that the RPC should live in an extra project to not distract from the format part.
    - kentonv 12y ago
      They are compiled as separate libraries, so you can use the serialization layer without linking in the RPC layer. But if anything it was the serialization that distracted me from the RPC. Cap'n Proto was originally (briefly) a capability-based RPC system on top of Protobufs. :)
cakoose 12y ago
Deeper in the site they have a benchmarks page[1] that has a short "why not Cap'n Proto": "Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support)." [1] http://google.github.io/flatbuffers/md__benchmarks.html http://google.github.io/flatbuffers/md__benchmarks.html
- kentonv 12y ago
  Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets. "Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire size really matters, Cap'n Proto offers a very fast compression pass which deflates zeros, and unset fields are always zero, largely mitigating this issue. The lack of VS support in Cap'n Proto is of course a very fair criticism, and a huge sore point in Cap'n Proto adoption. MSVC is finally catching up in its C++11 support so I hope this can be solved soon. (Disclosure: I'm the author of Cap'n Proto, and also the former maintainer of Protocol Buffers.)
  - mortarion 12y ago
    I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto. You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use protocol buffers instead since it allows them to more easily extend their net code without breaking backwards compatibility (up to a point of course). In the context of games, optional fields are very important as are default values. Also in this context; the author of flatbuffers is Wouter van Oortmerssen, author of the Cube and Cube2 game engines: http://sauerbraten.org http://sauerbraten.org From the white paper: > In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.
deleted 12y ago
[deleted]

srean 12y ago

I dont know full details of Cap'n Proto, so please correct me if I am wrong. My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. Would that be a reasonable summary of Cap'n Proto ? Guessing from the name I expected FlatBuffers to be a unified file format for on-disk and on-RAM presence, so that you can just mmap the disk file and you have the data-structure in memory. No need to translate between on disk and on disk representations. I am not claiming that it is so, this is what I expected it to be, given the name. But after spending some time looking around I am still confused if that's what it is. Would appreciate confirmation either way. Have to look at the code when I get around to it. I have an use case in mind for a tree of appendable/insertable matrices. HDf5 comes close, but isnt quite what I am looking for. EDIT @vertex-four thanks for the clarification much appreciated.

vertex-four 12y ago
> My impression is that Cap'n Proto is a re-implementation of protocol buffers with additional functionality for RPC wrapped in a futures/promises abstraction. No. It's a serialisation format that's specifically designed so that data structures in this format can be used as-is by programs, without unnecessarily copying data into or out of the format to use it. The RPC layer is then built on top of that, but the format could also be used for mmapped files, etc.
- bastih 12y ago
  I always felt that the RPC should live in an extra project to not distract from the format part.
  - kentonv 12y ago
    They are compiled as separate libraries, so you can use the serialization layer without linking in the RPC layer. But if anything it was the serialization that distracted me from the RPC. Cap'n Proto was originally (briefly) a capability-based RPC system on top of Protobufs. :)

cakoose 12y ago

Deeper in the site they have a benchmarks page[1] that has a short "why not Cap'n Proto": "Cap'n'Proto promises to reduce Protocol Buffers much like FlatBuffers does, though with a more complicated binary encoding and less flexibility (no optional fields to allow deprecating fields or serializing with missing fields for which defaults exist). It currently also isn't fully cross-platform portable (lack of VS support)." [1] http://google.github.io/flatbuffers/md__benchmarks.html http://google.github.io/flatbuffers/md__benchmarks.html

kentonv 12y ago
Hmm, my personal (biased) opinion is that using variable offsets and vtables (as FlatBuffer does) is a lot more complicated than Cap'n Proto's fixed offsets. "Optional" fields (ones that don't take space on the wire at all) are not clearly much of an advantage. The usual use of optional fields in Protobufs was to emulate unions, which Protobufs itself never supported directly, but Cap'n Proto does. Also, when wire size really matters, Cap'n Proto offers a very fast compression pass which deflates zeros, and unset fields are always zero, largely mitigating this issue. The lack of VS support in Cap'n Proto is of course a very fair criticism, and a huge sore point in Cap'n Proto adoption. MSVC is finally catching up in its C++11 support so I hope this can be solved soon. (Disclosure: I'm the author of Cap'n Proto, and also the former maintainer of Protocol Buffers.)
- mortarion 12y ago
  I think you have maybe misunderstood the purpose of why flatbuffers exists; message packing for real time high performance games. It's not a replacement for Protocol Buffers or Cap'n Proto. You've heard of Valve Software right? Their game engine, being a distant relative of Quake, used the old quake message packing format up until 1 or 2 years ago. Since then they have retrofitted all their popular games to use protocol buffers instead since it allows them to more easily extend their net code without breaking backwards compatibility (up to a point of course). In the context of games, optional fields are very important as are default values. Also in this context; the author of flatbuffers is Wouter van Oortmerssen, author of the Cube and Cube2 game engines: http://sauerbraten.org http://sauerbraten.org From the white paper: > In particular, FlatBuffers focus is on mobile hardware (where memory size and memory bandwidth is even more constrained than on desktop hardware), and applications that have the highest performance needs: games.

deleted 12y ago

[deleted]