3 ms·
Protobuf requires schemas, which is good practice anyways but maybe you don't have or want to do for some reason. FlatBuffers doesn't have as many client libra
by neuland 7y ago
Protobuf requires schemas, which is good practice anyways but maybe you don't have or want to do for some reason.
FlatBuffers doesn't have as many client libraries. There's a MessagePack library for a ton of languages.
One thing I really like about MessagePack is that the Python client (and others too) supports reading from a stream. So you can write a bunch of msgpack messages to a file or TCP socket and it just works.
Protobuf can't do this out of the box because it doesn't include how long the message is. You can write a wrapper that specifies the message length, which isn't that hard and I've done before, but it is another thing to maintain. And other formats do that out of the box (ex. Cap'n Proto)
And as someone else mentioned, Protobuf doesn't have NULL which is useful in some cases. (I understand that Go has strong opinions about there being a useful default value, but that doesn't map well to a lot of languages)
One other thing with Protobuf is that the Python client is not very pythonic. I've been keeping my eye on this project [0] which makes Protobuf messages work just like dataclasses. They don't support OneOf types currently, which I happen to need for some of my use cases. But they're working on it [1]
[0] https://github.com/eigenein/protobuf https://github.com/eigenein/protobuf
[1] https://github.com/eigenein/protobuf/issues/85 https://github.com/eigenein/protobuf/issues/85
- blaird 7y agoIf you're streaming messages I guess the good comparison would be protobuf + gRPC?
- oh_sigh 7y agoYou can use e g. in the Java api writeDelimitedTo to do streaming protobufs
- haberman 7y agoIt exists in C++ too: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/util/delimited_message_util.h https://github.com/protocolbuffers/protobuf/blob/master/src/...
- neuland 7y agoMy languages were C, Python, and Lua. Lua supports this, but C and Python don't (at least not in the library generated by protoc at the time I did this project, ie. like 2 years ago).
- thedance 7y agoThere's nothing fundamental about protobuf itself that prevents streaming output. You just have to make a pass over the proto structure first to compute the sizes, then stream it out using the precomputed sizes. At no time do you necessarily need the entire representation of the output in memory. The C++ library, for example, offers this.
- neuland 7y agoYes, and that's what I did. In my case, I was writing to a file. And my languages were C, Python, and Lua. So, I wrote a small package (plus bindings for Python and Lua) that would write out the message length as a uint64, then a uint8 for which schema the message is (I was working with multiple different message types), and then the actual message bytes. So, it's do-able. But, I'm just saying that other formats/libraries support this natively out of the box.
- thedance 7y agoIt sounds a bit like you invented RecordIO[1]. I didn't realize you meant decoding from a stream containing several protobufs. In that case you do indeed need some kind of framing, because there is no end-of-message delimiter. The positive tradeoff is you can easily concatenate several protobufs into another, valid protobuf. That is impossible if there is a terminating symbol. 1: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfrecords_format_details https://www.tensorflow.org/tutorials/load_data/tfrecord#tfre...
- neuland 7y agoOh neat. I'm not familiar with the ML/AI ecosystem. So, I didn't think to look there for formats; a bit of a shame I reinvented the wheel here, but it worked great for my specific use case. Yeah, it's all about the tradeoffs with formats. Protobuf is a good choice a lot of the time and I always think about it as one of the first/preferred options. gRPC is great too.