3 ms·

The current golang protobuf api / implementation is unsuitable for large scale system. See https://groups.google.com/forum/#!topic/Golang-nuts/vxJNXdiMhrc http

The current golang protobuf api / implementation is unsuitable for large scale system. See https://groups.google.com/forum/#!topic/Golang-nuts/vxJNXdiMhrc https://groups.google.com/forum/#!topic/Golang-nuts/vxJNXdiM... for discussion (you have to scroll down the thread a bit).

crawshaw 12y ago
That thread points out a lot of problems with both the current implementation of protos in Go and the inherent difficulties mapping of a code generator that was designed for C++ onto Go. I have had to deal with those issues and more using protos in Go. However, it does not make them unsuitable. We have many large scale systems at Google written in Go using this proto library. Some are high QPS, some use large protos, some have complex definitions. Each requires care and thought, but they work.
- minaandrawos 12y ago
  Whenever you get a chance, would you care to share some of the main pain points to look out for from your experience as well as the workarounds? I haven't encountered major issues (small issues yes, major issues not yet) so far implementing protos , however my projects scales are typically not as massive as Google's. So for the sake of my quest to expand my horizons, any feedback will be greatly appreciated
  - crawshaw 12y ago
    The most important piece of advice is not proto specific: use pprof. It may sound obvious, but even though I have been using it for years, I find myself still not running it enough. Proto decoding specific: if you're parsing a message with lots of fields, you can define a new message with just the fields you care about. I.e. if you're parsing message M { optional int32 F1 = 1; // ... optional int32 F100 = 100; } but you only care about F1 and F2 in this message, define message SmallM { optional int32 F1 = 1; optional int32 F2 = 2; } If you use SmallM to decode your messages, it will skip a lot of work filling out fields F3-F100, saving you a lot of CPU.

crawshaw 12y ago

That thread points out a lot of problems with both the current implementation of protos in Go and the inherent difficulties mapping of a code generator that was designed for C++ onto Go. I have had to deal with those issues and more using protos in Go. However, it does not make them unsuitable. We have many large scale systems at Google written in Go using this proto library. Some are high QPS, some use large protos, some have complex definitions. Each requires care and thought, but they work.

minaandrawos 12y ago
Whenever you get a chance, would you care to share some of the main pain points to look out for from your experience as well as the workarounds? I haven't encountered major issues (small issues yes, major issues not yet) so far implementing protos , however my projects scales are typically not as massive as Google's. So for the sake of my quest to expand my horizons, any feedback will be greatly appreciated
- crawshaw 12y ago
  The most important piece of advice is not proto specific: use pprof. It may sound obvious, but even though I have been using it for years, I find myself still not running it enough. Proto decoding specific: if you're parsing a message with lots of fields, you can define a new message with just the fields you care about. I.e. if you're parsing message M { optional int32 F1 = 1; // ... optional int32 F100 = 100; } but you only care about F1 and F2 in this message, define message SmallM { optional int32 F1 = 1; optional int32 F2 = 2; } If you use SmallM to decode your messages, it will skip a lot of work filling out fields F3-F100, saving you a lot of CPU.