5 ms·

Protobuf is incredibly annoying to use with rust (my usecase was specifically rust to-rust) because of how limited protobuf's type system is. For instance there

by azdle 3y ago

Protobuf is incredibly annoying to use with rust (my usecase was specifically rust to-rust) because of how limited protobuf's type system is. For instance there is no way to represent an `Option` type because protobuf has decided that if something is no present, then it is really some default value instead. The only workaround I found was to have both `foo` and `is_fo_set` fields, which required having duplicate copies of every datatype one for (de)serialization and one for actually using in my codebase, manually implementing changes between them. Don't get me wrong, there's nothing show-stopping, but it was definitely a death by a thousand cuts situation for me. Though, I know that another team at the same company loved it, but they were all in on java (and groovy and kotlin). As I understand it protobuf meshes with Java's type system much better (and probably Go's too?). --- I did some benchmarking on various formats/rust libraries when we were deciding on what the protocol should be used for ^ and, IIRC, JSON fared better than you'd have expected. My assumption is that JSON just has had way more eyes/hands on the implementation than anything else. The things that I can remember beating it were (1) bincode, (2) cbor, and (3) protobuf, then JSON was #4. In retrospect I wish we'd have gone with bincode, but we weren't sure if we were going to need some java code to interact with this service at the time and, at least at the time, bincode was rust-only (and possibly not even stable between compiler releases?). It would have been much faster to develop with and there was a whole bunch of overhead from protobuf that didn't really do anything for us since we were talking over a unix domain socket within a single system.

square_usual 3y ago
Have you tried FlatBuffers? And if you did, did it perform worse than JSON? The idea of deserialize-on-read is tempting to me, but I'd like to see real-world cases of it working out.
- ddorian43 3y ago
  FlatBuffers should be faster than protobuff.
bminor13 3y ago
proto3 messsage fields allow for detecting set vs. not set; other field types (repeated, map, int32, string, bool, enum, etc.) have this "default value if not set" issue. The canonical way of handling this is to use wrapper messages (because one can detect if the wrapper is not set), and there are "well-known" canned messages/protos one can import and use without writing their own: https://protobuf.dev/reference/protobuf/google.protobuf/ https://protobuf.dev/reference/protobuf/google.protobuf/ Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.
rvcdbn 3y ago
recent versions of proto3 have added back the “optional” keyword that can be used on any field. see: https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md#presence-in-proto3-apis https://github.com/protocolbuffers/protobuf/blob/main/docs/f...
foobiekr 3y ago
On optional.. this was a regression in proto that is somewhat helped by https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md#presence-in-proto3-apis https://github.com/protocolbuffers/protobuf/blob/main/docs/f... ; I have no idea whether protobuf for rust has started taking advantage of this. JSON is awful in every way.
athrowaway3z 3y ago
iirc https://github.com/tokio-rs/prost https://github.com/tokio-rs/prost has the proper proto3 'optional' -> Option<T> support
ithkuil 3y ago
Yes it is unsurprising that an IDL designed to be used as an interchange format between different languages (and other requirements such as backward compatibility etc) doesn't perfectly match your specific language type system. An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.
- rattray 3y ago
  To be fair, protobuf is a pretty poor/annoying match to many languages that aren't Go or Java, including TS. Optional types, for example, are very much not unique to Rust.
  - ithkuil 3y ago
    Don't get me wrong, it is a PITA. But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing. There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative. EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/implementing_proto3_presence.md https://github.com/protocolbuffers/protobuf/blob/main/docs/i...

by azdle 3y ago

square_usual 3y ago

Have you tried FlatBuffers? And if you did, did it perform worse than JSON? The idea of deserialize-on-read is tempting to me, but I'd like to see real-world cases of it working out.

ddorian43 3y ago
FlatBuffers should be faster than protobuff.

bminor13 3y ago

proto3 messsage fields allow for detecting set vs. not set; other field types (repeated, map, int32, string, bool, enum, etc.) have this "default value if not set" issue. The canonical way of handling this is to use wrapper messages (because one can detect if the wrapper is not set), and there are "well-known" canned messages/protos one can import and use without writing their own: https://protobuf.dev/reference/protobuf/google.protobuf/ https://protobuf.dev/reference/protobuf/google.protobuf/ Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.

rvcdbn 3y ago

recent versions of proto3 have added back the “optional” keyword that can be used on any field. see: https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md#presence-in-proto3-apis https://github.com/protocolbuffers/protobuf/blob/main/docs/f...

foobiekr 3y ago

On optional.. this was a regression in proto that is somewhat helped by https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md#presence-in-proto3-apis https://github.com/protocolbuffers/protobuf/blob/main/docs/f... ; I have no idea whether protobuf for rust has started taking advantage of this. JSON is awful in every way.

athrowaway3z 3y ago

iirc https://github.com/tokio-rs/prost https://github.com/tokio-rs/prost has the proper proto3 'optional' -> Option<T> support

ithkuil 3y ago

Yes it is unsurprising that an IDL designed to be used as an interchange format between different languages (and other requirements such as backward compatibility etc) doesn't perfectly match your specific language type system. An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.

rattray 3y ago
To be fair, protobuf is a pretty poor/annoying match to many languages that aren't Go or Java, including TS. Optional types, for example, are very much not unique to Rust.
- ithkuil 3y ago
  Don't get me wrong, it is a PITA. But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing. There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative. EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/implementing_proto3_presence.md https://github.com/protocolbuffers/protobuf/blob/main/docs/i...