4 ms·

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing engineering work for the sake of the work, rather than actually ask "what problem are we trying to solve here?" If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.

rkagerer 6y ago
If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.
- ikiris 6y ago
  Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?
  - FpUser 6y ago
    Maybe because "20X slower"
    - dweis 6y ago
      These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.
  - naniwaduni 6y ago
    Well, transmitting a megabyte also takes a full second at 8 Mbps.
    - RhodesianHunter 6y ago
      Nobody's transmitting this, it's the size of the library doing the proto serialization/deser
- ma2rten 6y ago
  I did a search on this page and I didn't find the comment you are referencing. It seems very unlikely to me that google's implementation is 20x slower.
  - rkagerer 6y ago
    See tijsvd's comment about 1 microsecond vs. 50 nanoseconds. (Perhaps I should have caveated that with "in some scenarios")
- asdfasgasdgasdg 6y ago
  Which implementation of protobuf is 20x faster than Google's?
- dguaraglia 6y ago
  I think the general point is 'if it works for Google, you should feel relatively safe adopting it for your own use'. It doesn't mean it'll fit every single usecase. If what you need is something that is super efficient over the wire - or any other such requirement not filled by Protobuf - then maybe look for a different protocol altogether. Or design your own! But for the rest of us - the '99.999%' - the trade-offs are well understood and we'd rather go for the tool we know than reinvent the wheel or use a less maintained tool.
deleted 6y ago
[deleted]
tijsvd 6y ago
No I don't work for Blizzard. When I did this, we were using a very fast custom wire format, but entirely hand-coded. Say flatbuffers without the code generation part. And having massive trouble with schema evolution. Another group had already evaluated pb and found it way too slow. They had designed something similar but faster. I wrote my implementation of pb to prevent this, and showed pb could be fast enough. It was definitely the right choice at the time, as it gave us C++ speed close to the old format, plus easy interop with other languages. > If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. Uh.. no. Google is a massive company, but if you browse the comments in this thread, you'll find multiple remarks like "this was built for Google's servers". Google have specific use cases, and they build software for that. The software may well be lacking for other use cases. I can totally imagine Blizzard wanting to write their own implementation, think of the benefits of reducing parse time in a multiplayer server.
stavrus 6y ago
I remember watching a GDC talk of JAM (https://www.gdcvault.com/play/1018184/Network-Serialization-and-Routing-in https://www.gdcvault.com/play/1018184/Network-Serialization-...), which was Blizzard's solution to network serialization for WoW back in 2004. The talk even had performance comparisons against protobuf and it seemed like a decent alternative that worked well for them. Was this new alternative an extension of JAM, or a full rewrite? I know they have millions of players monthly, but I can't imagine saving a bit on bandwidth could have warranted a full rewrite, especially when the majority of their traffic is from their own datacenters.

rkagerer 6y ago

If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.

ikiris 6y ago
Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?
- FpUser 6y ago
  Maybe because "20X slower"
  - dweis 6y ago
    These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.
- naniwaduni 6y ago
  Well, transmitting a megabyte also takes a full second at 8 Mbps.
  - RhodesianHunter 6y ago
    Nobody's transmitting this, it's the size of the library doing the proto serialization/deser
ma2rten 6y ago
I did a search on this page and I didn't find the comment you are referencing. It seems very unlikely to me that google's implementation is 20x slower.
- rkagerer 6y ago
  See tijsvd's comment about 1 microsecond vs. 50 nanoseconds. (Perhaps I should have caveated that with "in some scenarios")
asdfasgasdgasdg 6y ago
Which implementation of protobuf is 20x faster than Google's?
dguaraglia 6y ago
I think the general point is 'if it works for Google, you should feel relatively safe adopting it for your own use'. It doesn't mean it'll fit every single usecase. If what you need is something that is super efficient over the wire - or any other such requirement not filled by Protobuf - then maybe look for a different protocol altogether. Or design your own! But for the rest of us - the '99.999%' - the trade-offs are well understood and we'd rather go for the tool we know than reinvent the wheel or use a less maintained tool.

deleted 6y ago

[deleted]

tijsvd 6y ago

No I don't work for Blizzard. When I did this, we were using a very fast custom wire format, but entirely hand-coded. Say flatbuffers without the code generation part. And having massive trouble with schema evolution. Another group had already evaluated pb and found it way too slow. They had designed something similar but faster. I wrote my implementation of pb to prevent this, and showed pb could be fast enough. It was definitely the right choice at the time, as it gave us C++ speed close to the old format, plus easy interop with other languages. > If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. Uh.. no. Google is a massive company, but if you browse the comments in this thread, you'll find multiple remarks like "this was built for Google's servers". Google have specific use cases, and they build software for that. The software may well be lacking for other use cases. I can totally imagine Blizzard wanting to write their own implementation, think of the benefits of reducing parse time in a multiplayer server.

stavrus 6y ago

I remember watching a GDC talk of JAM (https://www.gdcvault.com/play/1018184/Network-Serialization-and-Routing-in https://www.gdcvault.com/play/1018184/Network-Serialization-...), which was Blizzard's solution to network serialization for WoW back in 2004. The talk even had performance comparisons against protobuf and it seemed like a decent alternative that worked well for them. Was this new alternative an extension of JAM, or a full rewrite? I know they have millions of players monthly, but I can't imagine saving a bit on bandwidth could have warranted a full rewrite, especially when the majority of their traffic is from their own datacenters.