3 ms·

This sounds very similar in concept to Cap'n Proto, and given how much hard-earned experience Kenton has and how much time he's put into it, I'd put my bets on

This sounds very similar in concept to Cap'n Proto, and given how much hard-earned experience Kenton has and how much time he's put into it, I'd put my bets on Cap'n Proto if you're looking for something in this design space. I haven't looked at SBE in detail, but given the high-level description I expect it would have a similar set of trade-offs vs. Protocol Buffers as Cap'n Proto does: it will have unparalleled serialize/deserialize performance (since they are basically no-ops), but sparse messages (where only a few of many defined fields are set) will be arbitrarily larger than protobufs, so a separate compression step will be necessary to be size-competitive, at which point some of the speed advantage is lost.

kasey_junk 13y ago
Kenton Varda deserves the respect you are giving him when it comes to serialization libraries. Cap'n Proto is a great project that I'd love to work with anytime. That said, Martin Thompson whose blog this is a link to is also very well respected in his particular domain (low latency messaging systems especially on the JVM and JVM/C++ hybrid space). All that's to say, the Cap'n Proto and SBE libraries have exceedingly similar design philosophies and pedigrees. I expect them to have near performance parity in the shared language of C++. But if you are looking for a JVM/C# solution I think you can pretty reasonably choose to start working with SBE today as it provides those language bindings out of the gate and there is unlikely to be such a performance mismatch between SBE and Cap'n Proto in the future for it to matter long term.
- haberman 13y ago
  I can totally believe that Martin Thompson is highly capable given what I read in the article. I guess my bias is against a proliferation of similar-but-incompatible messaging formats. But I suppose that if Cap'n Proto doesn't support JVM/C#, then that doesn't leave someone on those platforms a lot of options.
  - kentonv 13y ago
    There's been a lot of false starts on getting Java support implemented (people saying they'd do it, and then not), and one or two for C# as well. I wish I had time to just sit down and crank out a Java implementation myself -- I think it would take a week or two -- but right now I really need to be putting everything into Sandstorm.io. :/ It will happen eventually, though.
    - kasey_junk 13y ago
      I think one problem with this is that a JNI solution would be "trivial" but is wholly unsatisfying (JNI being somewhat incompatible with the performance goals). The Unsafe and/or ByteBuffer solution on the other hand would be a lot of work to create and maintain (so much so that I wouldn't sign up for the job). SBE on the other hand is targeting a space dominated by C++/Java/C#. They can basically ignore every other language binding without losing any of their target audience. It also doesn't hurt that they have a patron with a financial motivation to back them...

kasey_junk 13y ago

Kenton Varda deserves the respect you are giving him when it comes to serialization libraries. Cap'n Proto is a great project that I'd love to work with anytime. That said, Martin Thompson whose blog this is a link to is also very well respected in his particular domain (low latency messaging systems especially on the JVM and JVM/C++ hybrid space). All that's to say, the Cap'n Proto and SBE libraries have exceedingly similar design philosophies and pedigrees. I expect them to have near performance parity in the shared language of C++. But if you are looking for a JVM/C# solution I think you can pretty reasonably choose to start working with SBE today as it provides those language bindings out of the gate and there is unlikely to be such a performance mismatch between SBE and Cap'n Proto in the future for it to matter long term.

haberman 13y ago
I can totally believe that Martin Thompson is highly capable given what I read in the article. I guess my bias is against a proliferation of similar-but-incompatible messaging formats. But I suppose that if Cap'n Proto doesn't support JVM/C#, then that doesn't leave someone on those platforms a lot of options.
- kentonv 13y ago
  There's been a lot of false starts on getting Java support implemented (people saying they'd do it, and then not), and one or two for C# as well. I wish I had time to just sit down and crank out a Java implementation myself -- I think it would take a week or two -- but right now I really need to be putting everything into Sandstorm.io. :/ It will happen eventually, though.
  - kasey_junk 13y ago
    I think one problem with this is that a JNI solution would be "trivial" but is wholly unsatisfying (JNI being somewhat incompatible with the performance goals). The Unsafe and/or ByteBuffer solution on the other hand would be a lot of work to create and maintain (so much so that I wouldn't sign up for the job). SBE on the other hand is targeting a space dominated by C++/Java/C#. They can basically ignore every other language binding without losing any of their target audience. It also doesn't hurt that they have a patron with a financial motivation to back them...