6 ms·

Bebop v3: a fast, modern replacement for Protocol Buffers

46 points by andrew_rfc 3y ago

Kelteseth 3y ago
> In Bebop v3 you'll find that runtimes for TypeScript, Rust, and C# are much faster. Sorry, but I would like some benchmark data to these claims....
- andrew_rfc 3y ago
  The benchmarks are in the repo + provided as screenshots in the README, but to clarify this is speaking to performance improvements relative to v2 of Bebop.
  - Kelteseth 3y ago
    Ah, thanks! Would be interesting to see benchmarks against cCap'n Proto
    - andrew_rfc 3y ago
      We don’t benchmark against FlatBuffers or Cap'n Proto for a few reasons: - They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet. - We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare. Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.)
    - neonsunset 3y ago
      cap'n'proto is ancient, unmaintained, and its C# library is one of the most poorly written binary serialization implementations I have seen in my life. You will be better served by gRPC and, if you need absolute maximum performance, MemoryPack (it is C#-first but can generate TypeScript clients). If you need MQ, NATS is a good choice (special kudos to them adopting Yoshifumi Kawai's AlterNATS into NATS v2 C# client implementation).
      - andrew_rfc 3y ago
        If we’re talking specifically about C# binary serialization I’d recommend checking out this independent benchmark another developer did comparing Bebop to everything else https://youtu.be/XMoNYQPi2k8?feature=shared https://youtu.be/XMoNYQPi2k8?feature=shared
        neonsunset 3y ago
        The comparison uses old and/or deprecated/EOL serializers which no one in their sane mind should use today. Consider referring to my previous comment, thanks! (and by deprecated I mean deprecated 5+ years ago like in the case of Utf8Json) Sadly a lot of videos discussing performance do only limited research and do not approach the matter comprehensively (likely not worth it for YT algorithm favoring quick and easy content). Nothing wrong with offering your own first-party use cases that attempt to simulate realistic workloads on top of suggested patterns for each respective contestant, but it is important to include more performance-oriented frameworks/libraries (which is why I mentioned MemoryPack) that are used e.g. in gamedev sphere which cares a lot about realtime communication. With all that said, it's actually awesome to see a framework that offers first-class C# support, so I probably need to shut up and appreciate the fact :)
      - kentonv 3y ago
        > cap'n'proto is ancient, I mean... not compared to protobuf. > unmaintained, The C++ implementation is well-maintained and heavily used by its own maintainers (e.g. me). Other implementations are maintained to varying degrees, since each implementation has a different maintainer. Agreed the C# implementation isn't maintained. > You will be better served by gRPC Well, this really depends, considering that Cap'n Proto's RPC system is quite different and far more expressive than gRPC.
zettabomb 3y ago
Does Bebop support use as a file format by dumping directly to disk, as some serialization libraries do? Or perhaps a better way to phrase it, since ultimately you can send whatever you want to a binary file, is it intended to be used in such a way?
- andrew_rfc 3y ago
  You can. I use it to persist settings in my app Borderless Gaming https://github.com/Codeusa/Borderless-Gaming https://github.com/Codeusa/Borderless-Gaming I’m also working on a textual configuration language that allowa you to import a schema and define data like JSON.
  - zettabomb 3y ago
    Very cool! I'd definitely be interested in that, especially if it could spit out class definitions for supported languages. Bebop seems to have native support for pretty much every language I care about too.
martinky24 3y ago
Are protocol buffers not considered "modern" or "fast" anymore? How time flies...
- andrew_rfc 3y ago
  They were never particularly all that fast, especially not in the browser where support is far from acceptable. Most implementations rely on runtime magic, reflection, and other hacks. So in that sense they aren’t all that modern as they’ve failed to evolve and keep up with platform evolutions. Bebop just aims to generate code that modern and current with its target language, works everywhere, and avoids pitfalls protobuf suffers from.
  - martinky24 3y ago
    I see. So far protobufs have met my needs and I've been happy with them, but I've used them more in an RPC context and less as a browser based data unit, so that makes sense.
  - TexasMick 3y ago
    I benchmarked protobuf encoding and nanopb was very good. Even though our schema was huge, it still did serialisation in a time that was surprisingly small for our underpowered chip.
  - throwaway143829 3y ago
    Have you seen protobuf.js? I've dug quite deep into its generated code and have not seen any runtime magic. It's just directly working with the protobuf wire format and transferring the data to/from fields with very explicit code. I can't speak for Google's official implementation though.
- flohofwoe 3y ago
  Not sure if it has improved in the last few years or if other target languages are better, but the protobuf code generation for C++ was pretty much the worst C++ code in terms of bloat I have ever seen. And I've seen a lot of bad C++ code ;)
bfrog 3y ago
I mean there's cap'n proto and flatbuffers already? Where's the benchmarks again capnproto in particular?
- andrew_rfc 3y ago
  Cap’n Proto and Flatbuffers are not alternatives to Protocol Buffers. They solve entirely different issues and have distinct limitations. They aren’t relevant to what Bebop is doing and thus no benchmark or comparison can be made between them.
  - kentonv 3y ago
    Sure they are. Cap'n Proto addresses a superset of Protobuf's use cases. Anywhere where you might use Protobuf, Cap'n Proto will work fine. In certain use cases (like mmaping large files or exporting complex APIs over RPC) Cap'n Proto is much better than Protobuf, but it's not bad at the other use cases. The main limitation of Cap'n Proto compared to Protobuf is the ecosystem -- missing or poor-quality implementations in many languages, limited tooling, etc. Admittedly this is probably a showstopper for most users. It's also the hardest thing for any new contender to solve. With all that said I would tend to agree that benchmarks are probably pointless. I've spent a lot of time benchmarking serialization and one thing I know is that benchmark results will vary wildly depending on the use case. A benchmark of an example/toy use case isn't really indicative of performance in a real use case. (I'm the author of Cap'n Proto. I don't know much about Flatbuffers so can't comment there.)
    - andrew_rfc 3y ago
      Apologies if my initial comment came off the wrong way; when I say “not alternatives” I was injecting bias into that statement since I normally use Cap’n Proto on embedded and memory constrained systems (amazing library btw.) When I created Bebop years ago I tried to benchmark it against Cap’n Proto, but the lack of a semi-decent web or C# implementation mentally made me recategorize. So I suppose I’m just comparing what’s closest in terms of ecosystem support rather than a purely functional level.
sour-taste 3y ago
What kind of compatibility guarantees does this make? That's one of the key benefits of protos: I can update clients to add new fields and they can still talk to servers with previous versions of the schema.
- andrew_rfc 3y ago
  you get forward and backwards compatibility out of the box so long as you adhere to the schema guidelines. The docs go into more detail on how different types are handled. We also have some special features like “binary schema” which let you ship your schema with data so clients are always up to date.
zigzag312 3y ago
Interesting. How does it compare to Slice? https://docs.icerpc.dev/slice2 https://docs.icerpc.dev/slice2
- andrew_rfc 3y ago
  Looks neat. Bebop is licensed under Apache 2.0, not GPL and doesn’t require a commercial license to use for profit. And I can tell the C# code it generates is going to be plagued by vtable lookups. Some decent ideas though.
  - neonsunset 3y ago
    (it may be less of an issue today since the DPGO will devirtualize those, when profiling average rpc framework, more time is spent on 1. deep async call stacks, 2. inefficient serde that copies data frequently and heap allocates transient objects and 3. working against the language by not using its features made for performance (Memory/Span slices, using manually rolled copy and search routines over span.CopyTo, .IndexOf)) And overall observation is the less code and the simpler it is, the better, notably gRPC suffers from its pipeline/interceptors-support style implementations and insisting on usings its own types replacing T[] and List<T>, putting irreducible mandatory cost in places where it may matter most (unless you partially bypass those with reflection/unsafe accessor like I had to do once))
  - zigzag312 3y ago
    Slice & IceRPC are also licensed under Apache 2.0. Were you looking at Ice Framework licence? It's nice to finally see some quality alternatives to Protobuf. One idea I like from Slice are custom types. Is there anything similar in Bebop?
vyrotek 3y ago
Looks great! Is there any chance that Tempo RPC supports TS web clients talking to C# backends? That's our dream. https://docs.bebop.sh/tempo/ https://docs.bebop.sh/tempo/ Update: Coming soon! https://twitter.com/andrewmd5/status/1767612314730336353 https://twitter.com/andrewmd5/status/1767612314730336353
0x073 3y ago
Never understand the hype for protocol buffers, it may works well for languages like java and kotlin, but is a mess on many other languages. And then there are faster and better replacements. And all the time I use JSON, there don't exists multiple versions and any JSON file that is somewhere can get read by almost every language. (And I can read the file even with every editor)
- cdelsolar 3y ago
  Typing. And it works fantastic with Go
syrusakbary 3y ago
This is awesome. I'd love to have upstream support in Wasmer ( https://wasmer.io https://wasmer.io )
nullc 3y ago
Will it send unzeroed application memory across like some other "fast" protobuf alternatives have done at the risk of data leaks?
mdhb 3y ago
I’ve been hoping to see someone eventually build a protobuf style serialisation solution on some of the emerging IETF standards like CBOR and it’s corresponding data definition language. I’m not really in the market for another proprietary binary data format that doesn’t integrate with anything outside its immediate ecosystem.
elux101 3y ago
also check out https://github.com/webrpc/webrpc https://github.com/webrpc/webrpc

46 points by andrew_rfc 3y ago

> In Bebop v3 you'll find that runtimes for TypeScript, Rust, and C# are much faster. Sorry, but I would like some benchmark data to these claims....

andrew_rfc 3y ago
The benchmarks are in the repo + provided as screenshots in the README, but to clarify this is speaking to performance improvements relative to v2 of Bebop.
- Kelteseth 3y ago
  Ah, thanks! Would be interesting to see benchmarks against cCap'n Proto
  - andrew_rfc 3y ago
    We don’t benchmark against FlatBuffers or Cap'n Proto for a few reasons: - They are impossible to benchmark against each other without making an assumption about how often you want to access the data, and which parts of it you want to access. But this means Bebop, Cap'n Proto, FlatBuffers can exist side-by-side / solve different problems: what FlatBuffers and Cap'n Proto do makes sense if you want to access only parts of your objects in limited specific ways, what we do is better if you're always interested in the whole packet. - We don’t benchmark against Capt’n Proto because it does not have a stable web-based implementation, at least not one that has the features that make it so fast natively, so there is nothing to compare. Basically, Bebop is great for message oriented applications or where you need the entirety of your packet deserialized in a single step. So we can only benchmark against similar formats (JSON, Protobufs, MsgPack, etc.)
  - neonsunset 3y ago
    cap'n'proto is ancient, unmaintained, and its C# library is one of the most poorly written binary serialization implementations I have seen in my life. You will be better served by gRPC and, if you need absolute maximum performance, MemoryPack (it is C#-first but can generate TypeScript clients). If you need MQ, NATS is a good choice (special kudos to them adopting Yoshifumi Kawai's AlterNATS into NATS v2 C# client implementation).
    - andrew_rfc 3y ago
      If we’re talking specifically about C# binary serialization I’d recommend checking out this independent benchmark another developer did comparing Bebop to everything else https://youtu.be/XMoNYQPi2k8?feature=shared https://youtu.be/XMoNYQPi2k8?feature=shared
      - neonsunset 3y ago
        The comparison uses old and/or deprecated/EOL serializers which no one in their sane mind should use today. Consider referring to my previous comment, thanks! (and by deprecated I mean deprecated 5+ years ago like in the case of Utf8Json) Sadly a lot of videos discussing performance do only limited research and do not approach the matter comprehensively (likely not worth it for YT algorithm favoring quick and easy content). Nothing wrong with offering your own first-party use cases that attempt to simulate realistic workloads on top of suggested patterns for each respective contestant, but it is important to include more performance-oriented frameworks/libraries (which is why I mentioned MemoryPack) that are used e.g. in gamedev sphere which cares a lot about realtime communication. With all that said, it's actually awesome to see a framework that offers first-class C# support, so I probably need to shut up and appreciate the fact :)
    - kentonv 3y ago
      > cap'n'proto is ancient, I mean... not compared to protobuf. > unmaintained, The C++ implementation is well-maintained and heavily used by its own maintainers (e.g. me). Other implementations are maintained to varying degrees, since each implementation has a different maintainer. Agreed the C# implementation isn't maintained. > You will be better served by gRPC Well, this really depends, considering that Cap'n Proto's RPC system is quite different and far more expressive than gRPC.

zettabomb 3y ago

Does Bebop support use as a file format by dumping directly to disk, as some serialization libraries do? Or perhaps a better way to phrase it, since ultimately you can send whatever you want to a binary file, is it intended to be used in such a way?

andrew_rfc 3y ago
You can. I use it to persist settings in my app Borderless Gaming https://github.com/Codeusa/Borderless-Gaming https://github.com/Codeusa/Borderless-Gaming I’m also working on a textual configuration language that allowa you to import a schema and define data like JSON.
- zettabomb 3y ago
  Very cool! I'd definitely be interested in that, especially if it could spit out class definitions for supported languages. Bebop seems to have native support for pretty much every language I care about too.

martinky24 3y ago

Are protocol buffers not considered "modern" or "fast" anymore? How time flies...

andrew_rfc 3y ago
They were never particularly all that fast, especially not in the browser where support is far from acceptable. Most implementations rely on runtime magic, reflection, and other hacks. So in that sense they aren’t all that modern as they’ve failed to evolve and keep up with platform evolutions. Bebop just aims to generate code that modern and current with its target language, works everywhere, and avoids pitfalls protobuf suffers from.
- martinky24 3y ago
  I see. So far protobufs have met my needs and I've been happy with them, but I've used them more in an RPC context and less as a browser based data unit, so that makes sense.
- TexasMick 3y ago
  I benchmarked protobuf encoding and nanopb was very good. Even though our schema was huge, it still did serialisation in a time that was surprisingly small for our underpowered chip.
- throwaway143829 3y ago
  Have you seen protobuf.js? I've dug quite deep into its generated code and have not seen any runtime magic. It's just directly working with the protobuf wire format and transferring the data to/from fields with very explicit code. I can't speak for Google's official implementation though.
flohofwoe 3y ago
Not sure if it has improved in the last few years or if other target languages are better, but the protobuf code generation for C++ was pretty much the worst C++ code in terms of bloat I have ever seen. And I've seen a lot of bad C++ code ;)

bfrog 3y ago

I mean there's cap'n proto and flatbuffers already? Where's the benchmarks again capnproto in particular?

andrew_rfc 3y ago
Cap’n Proto and Flatbuffers are not alternatives to Protocol Buffers. They solve entirely different issues and have distinct limitations. They aren’t relevant to what Bebop is doing and thus no benchmark or comparison can be made between them.
- kentonv 3y ago
  Sure they are. Cap'n Proto addresses a superset of Protobuf's use cases. Anywhere where you might use Protobuf, Cap'n Proto will work fine. In certain use cases (like mmaping large files or exporting complex APIs over RPC) Cap'n Proto is much better than Protobuf, but it's not bad at the other use cases. The main limitation of Cap'n Proto compared to Protobuf is the ecosystem -- missing or poor-quality implementations in many languages, limited tooling, etc. Admittedly this is probably a showstopper for most users. It's also the hardest thing for any new contender to solve. With all that said I would tend to agree that benchmarks are probably pointless. I've spent a lot of time benchmarking serialization and one thing I know is that benchmark results will vary wildly depending on the use case. A benchmark of an example/toy use case isn't really indicative of performance in a real use case. (I'm the author of Cap'n Proto. I don't know much about Flatbuffers so can't comment there.)
  - andrew_rfc 3y ago
    Apologies if my initial comment came off the wrong way; when I say “not alternatives” I was injecting bias into that statement since I normally use Cap’n Proto on embedded and memory constrained systems (amazing library btw.) When I created Bebop years ago I tried to benchmark it against Cap’n Proto, but the lack of a semi-decent web or C# implementation mentally made me recategorize. So I suppose I’m just comparing what’s closest in terms of ecosystem support rather than a purely functional level.

sour-taste 3y ago

What kind of compatibility guarantees does this make? That's one of the key benefits of protos: I can update clients to add new fields and they can still talk to servers with previous versions of the schema.

andrew_rfc 3y ago
you get forward and backwards compatibility out of the box so long as you adhere to the schema guidelines. The docs go into more detail on how different types are handled. We also have some special features like “binary schema” which let you ship your schema with data so clients are always up to date.

zigzag312 3y ago

Interesting. How does it compare to Slice? https://docs.icerpc.dev/slice2 https://docs.icerpc.dev/slice2

andrew_rfc 3y ago
Looks neat. Bebop is licensed under Apache 2.0, not GPL and doesn’t require a commercial license to use for profit. And I can tell the C# code it generates is going to be plagued by vtable lookups. Some decent ideas though.
- neonsunset 3y ago
  (it may be less of an issue today since the DPGO will devirtualize those, when profiling average rpc framework, more time is spent on 1. deep async call stacks, 2. inefficient serde that copies data frequently and heap allocates transient objects and 3. working against the language by not using its features made for performance (Memory/Span slices, using manually rolled copy and search routines over span.CopyTo, .IndexOf)) And overall observation is the less code and the simpler it is, the better, notably gRPC suffers from its pipeline/interceptors-support style implementations and insisting on usings its own types replacing T[] and List<T>, putting irreducible mandatory cost in places where it may matter most (unless you partially bypass those with reflection/unsafe accessor like I had to do once))
- zigzag312 3y ago
  Slice & IceRPC are also licensed under Apache 2.0. Were you looking at Ice Framework licence? It's nice to finally see some quality alternatives to Protobuf. One idea I like from Slice are custom types. Is there anything similar in Bebop?

vyrotek 3y ago

Looks great! Is there any chance that Tempo RPC supports TS web clients talking to C# backends? That's our dream. https://docs.bebop.sh/tempo/ https://docs.bebop.sh/tempo/ Update: Coming soon! https://twitter.com/andrewmd5/status/1767612314730336353 https://twitter.com/andrewmd5/status/1767612314730336353

0x073 3y ago

Never understand the hype for protocol buffers, it may works well for languages like java and kotlin, but is a mess on many other languages. And then there are faster and better replacements. And all the time I use JSON, there don't exists multiple versions and any JSON file that is somewhere can get read by almost every language. (And I can read the file even with every editor)

cdelsolar 3y ago
Typing. And it works fantastic with Go

syrusakbary 3y ago

This is awesome. I'd love to have upstream support in Wasmer ( https://wasmer.io https://wasmer.io )

nullc 3y ago

Will it send unzeroed application memory across like some other "fast" protobuf alternatives have done at the risk of data leaks?

mdhb 3y ago

I’ve been hoping to see someone eventually build a protobuf style serialisation solution on some of the emerging IETF standards like CBOR and it’s corresponding data definition language. I’m not really in the market for another proprietary binary data format that doesn’t integrate with anything outside its immediate ecosystem.

elux101 3y ago

also check out https://github.com/webrpc/webrpc https://github.com/webrpc/webrpc