5 ms·

Is there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Msgpack, unlike json, is not h

Is there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Msgpack, unlike json, is not human readable on the wire, not a purely text based format, and I doubt it is smaller or faster than protobufs or flatbuffers.

mgamache 7y ago
it's about as fast as protobuf using .net. It is not human readible.
throw0101a 7y ago
> Is there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Or CBOR, that has an RFC (which is important for some folks): * https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
eshyong 7y ago
Plus the fact that protobuf is more flexible about interface changes. My previous company had countless incidents that happened when new code deserialized msgpack records with unexpected fields from redis...
alexdeloy 7y ago
I was asking myself the same thing and apparently protobuf has no concept of "null", thus I can see how MessagePack might have an advantage depending on your use case here. https://github.com/neuecc/MessagePack-CSharp#comparison-with-protobuf-json-zeroformatter https://github.com/neuecc/MessagePack-CSharp#comparison-with...
- fizx 7y ago
  protobuf can do nullable values, but you have to ask for them as primitives are the default. e.g. StringValue (nullable) vs string (not)
neuland 7y ago
Protobuf requires schemas, which is good practice anyways but maybe you don't have or want to do for some reason. FlatBuffers doesn't have as many client libraries. There's a MessagePack library for a ton of languages. One thing I really like about MessagePack is that the Python client (and others too) supports reading from a stream. So you can write a bunch of msgpack messages to a file or TCP socket and it just works. Protobuf can't do this out of the box because it doesn't include how long the message is. You can write a wrapper that specifies the message length, which isn't that hard and I've done before, but it is another thing to maintain. And other formats do that out of the box (ex. Cap'n Proto) And as someone else mentioned, Protobuf doesn't have NULL which is useful in some cases. (I understand that Go has strong opinions about there being a useful default value, but that doesn't map well to a lot of languages) One other thing with Protobuf is that the Python client is not very pythonic. I've been keeping my eye on this project [0] which makes Protobuf messages work just like dataclasses. They don't support OneOf types currently, which I happen to need for some of my use cases. But they're working on it [1] [0] https://github.com/eigenein/protobuf https://github.com/eigenein/protobuf [1] https://github.com/eigenein/protobuf/issues/85 https://github.com/eigenein/protobuf/issues/85
- blaird 7y ago
  If you're streaming messages I guess the good comparison would be protobuf + gRPC?
- oh_sigh 7y ago
  You can use e g. in the Java api writeDelimitedTo to do streaming protobufs
  - haberman 7y ago
    It exists in C++ too: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/util/delimited_message_util.h https://github.com/protocolbuffers/protobuf/blob/master/src/...
  - neuland 7y ago
    My languages were C, Python, and Lua. Lua supports this, but C and Python don't (at least not in the library generated by protoc at the time I did this project, ie. like 2 years ago).
ChuckMcM 7y ago
One advantage is that you can pack and unpack msgpack on an Arduino but you can't run gzip on that hardware.
kragen 7y ago
Well, msgpack does support blobs, unlike JSON, and is schemaless, unlike protobufs and FlatBuffers. I think the second one is kind of a hard constraint, while the first one is something you can work around with base64 in many cases.

mgamache 7y ago

it's about as fast as protobuf using .net. It is not human readible.

throw0101a 7y ago

> Is there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Or CBOR, that has an RFC (which is important for some folks): * https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR

eshyong 7y ago

Plus the fact that protobuf is more flexible about interface changes. My previous company had countless incidents that happened when new code deserialized msgpack records with unexpected fields from redis...

alexdeloy 7y ago

I was asking myself the same thing and apparently protobuf has no concept of "null", thus I can see how MessagePack might have an advantage depending on your use case here. https://github.com/neuecc/MessagePack-CSharp#comparison-with-protobuf-json-zeroformatter https://github.com/neuecc/MessagePack-CSharp#comparison-with...

fizx 7y ago
protobuf can do nullable values, but you have to ask for them as primitives are the default. e.g. StringValue (nullable) vs string (not)

neuland 7y ago

Protobuf requires schemas, which is good practice anyways but maybe you don't have or want to do for some reason. FlatBuffers doesn't have as many client libraries. There's a MessagePack library for a ton of languages. One thing I really like about MessagePack is that the Python client (and others too) supports reading from a stream. So you can write a bunch of msgpack messages to a file or TCP socket and it just works. Protobuf can't do this out of the box because it doesn't include how long the message is. You can write a wrapper that specifies the message length, which isn't that hard and I've done before, but it is another thing to maintain. And other formats do that out of the box (ex. Cap'n Proto) And as someone else mentioned, Protobuf doesn't have NULL which is useful in some cases. (I understand that Go has strong opinions about there being a useful default value, but that doesn't map well to a lot of languages) One other thing with Protobuf is that the Python client is not very pythonic. I've been keeping my eye on this project [0] which makes Protobuf messages work just like dataclasses. They don't support OneOf types currently, which I happen to need for some of my use cases. But they're working on it [1] [0] https://github.com/eigenein/protobuf https://github.com/eigenein/protobuf [1] https://github.com/eigenein/protobuf/issues/85 https://github.com/eigenein/protobuf/issues/85

blaird 7y ago
If you're streaming messages I guess the good comparison would be protobuf + gRPC?
oh_sigh 7y ago
You can use e g. in the Java api writeDelimitedTo to do streaming protobufs
- haberman 7y ago
  It exists in C++ too: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/util/delimited_message_util.h https://github.com/protocolbuffers/protobuf/blob/master/src/...
- neuland 7y ago
  My languages were C, Python, and Lua. Lua supports this, but C and Python don't (at least not in the library generated by protoc at the time I did this project, ie. like 2 years ago).

ChuckMcM 7y ago

One advantage is that you can pack and unpack msgpack on an Arduino but you can't run gzip on that hardware.

kragen 7y ago

Well, msgpack does support blobs, unlike JSON, and is schemaless, unlike protobufs and FlatBuffers. I think the second one is kind of a hard constraint, while the first one is something you can work around with base64 in many cases.