7 ms·

Sure, but if performance is the actual issue, then you'd likely pick an even faster and more compact wire format (proto, avro, thrift). I guess the ability to

by meritt 7y ago

Sure, but if performance is the actual issue, then you'd likely pick an even faster and more compact wire format (proto, avro, thrift). I guess the ability to have a schemalass and no type checking format (just like JSON), while enjoying some performance benefits? It just feels like a weird niche to me.

pas 7y ago
It's just a nice default to pick. (But JSON is "simple" so msgpack did not really catch on. And probably won't for the reasons others mentioned.)
- choward 7y ago
  > It's just a nice default to pick. Why would it be your default?
  - jnwatson 7y ago
    It is relatively efficient, schemaless, and cross-language and -platform. It supports floats and binary values as well.
  - ahoka 7y ago
    It's my go to serialization format for redis payloads. Fast, compact, can represent a lot of types without too much magic.
  - pas 7y ago
    It's really very-very fast, and no need for external schema registry. And it's really as simple as JSON. Has great language support. https://github.com/thekvs/cpp-serializers#results https://github.com/thekvs/cpp-serializers#results That said, currently I don't work with it in any projects. Apparently it hasn't really caught on.
dickeytk 7y ago
don't forget https://capnproto.org/ https://capnproto.org/
mumblemumble 7y ago
I guess. . . for my part, it's always seemed to me like messagepack falls into a sort of uncanny valley of serialization formats. Schemaless is very undesirable for internal APIs, IMO. Plaintext is desirable for public-facing APIs that don't need to be particularly performant, but only if you're willing to go full HATEOAS. If you're going binary for performance or don't want to bother with HATEOAS, I'm back to preferring a published schema over something that's (semi-)self-describing plus some documentation that's invariably incomplete or out of date.
- ClumsyPilot 7y ago
  I have never seen any benefits of HATEOAS materialise - it is too free-form to be machine parceable, is it far more difficult to navigate than Swagger or GraphQ. Maybe I dont get something?
  - mumblemumble 7y ago
    I don't ever machine parse it, but it can be useful for manual discovery if you get an API that isn't super well documented. I think I might generally prefer Swagger, too, but not everything is implemented in a language that has a library like that. At the end of the day, though, I'd seriously much rather just have a halfway well commented *.proto file. Work smarter, not harder. It's just that I won't recommend that for public-facing APIs because GRPC isn't universally well supported, while JSON over HTTP is.
ken 7y ago
Protobuf/Thrift require predefined schemas, which is a different model. Avro is pretty similar to MsgPack in concept. It seems less popular (at least in my circles), and has fewer implementations, which may or may not matter. As for performance and efficiency, the first benchmark I found [1] shows MsgPack is more space efficient, and faster to serialize, while Avro is faster to de-serialize. The second benchmark I found [2] found exactly the opposite. So it's not clear to me that either is better, and I'm sure it depends on your data, which library you're using, and how you're using it. Being nearly as performant as the predefined-schema serializers, while being nearly a drop-in replacement for JSON, seems like a major and valuable use case to me. [1]: https://medium.com/@nitinpaliwal87/compression-and-serialization-techniques-benchmarking-fd1f34c1098b https://medium.com/@nitinpaliwal87/compression-and-serializa... [2]: https://github.com/saint1991/serialization-benchmark https://github.com/saint1991/serialization-benchmark
- meritt 7y ago
  > shows MsgPack is more space efficient While I see the same numbers as you do, there's no way a format that includes plaintext keys alongside the values is going to be smaller than a format that only includes a packed field number+type and the corresponding values. There's something fishy going on here, and I can't seem to find a link to the source behind his benchmarks?
  - deleted 7y ago
    [deleted]
  - ludocode 7y ago
    I'm not sure if the benchmarks above reflect this, but MessagePack allows you to use integers (or any data type) for map keys. This makes it a lot smaller than using plaintext keys, and makes it comparable to formats like Protobuf despite the lack of formal schema. (You also lose a lot of the context-free readability of messages, so it's a tradeoff.)
2J0 7y ago
Whatever was wrong with ASN.1 for the wire? it's not a direct replacement but a sanitising schema that's truly amazing in application breadth, support, tooling, and implementation efficiency. In force documentation: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.680-201508-I!!PDF-E&type=items https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.68... Blog on ASN.1 schema for JSON: https://www.obj-sys.com/blog/?p=508 https://www.obj-sys.com/blog/?p=508
- jnwatson 7y ago
  I really like ASN.1. The big difference is that messagepack is schemaless.
- tptacek 7y ago
  It's incredibly annoying to marshal and unmarshal arbitrary ASN.1/BER. Variable-length odd-bit-length length fields annoying. BER is like the worst parts of every other format, collected in a single standard.
- imtringued 7y ago
  I don't know why but I constantly hear about things that never caught on and will never catch on in the future. Ok lets say I am hyped for ASN.1 as a schema for JSON. How do I integrate it into my bog standard Java application? Suddenly we are running into a huge problem. >ASN.1 is a mature standard. As I already mentioned, it has been around since the 1980’s. Though stable, it is not stagnant; the most recent revision occurred in 2008. It might be old but it is not widely used. To me it doesn't matter if something was created yesterday or 20 years ago as long as I can use it and using ASN.1 is significantly harder than necessary. >ASN.1 has tool support. There are both commercial and open source tools that will generate code from your ASN.1 specification. Ok, so where is it and why is it so difficult to find? Even if we assume that we should just build it today then all the above claims suddenly become worthless. If someone builds ASN.1 tooling in 2020 then it is just as immature as e.g. GraphQL tooling that was built 2020. If there is renewed interest in ASN.1 then it might gain new features that will cause it to become less mature/stable again. Having "mature" software is useless if it doesn't meet user demands.
Scarbutt 7y ago
Because all your existing json will just work, you don't have to change your data representation to support some other format(proto, avro, thrift).

by meritt 7y ago

pas 7y ago

It's just a nice default to pick. (But JSON is "simple" so msgpack did not really catch on. And probably won't for the reasons others mentioned.)

choward 7y ago
> It's just a nice default to pick. Why would it be your default?
- jnwatson 7y ago
  It is relatively efficient, schemaless, and cross-language and -platform. It supports floats and binary values as well.
- ahoka 7y ago
  It's my go to serialization format for redis payloads. Fast, compact, can represent a lot of types without too much magic.
- pas 7y ago
  It's really very-very fast, and no need for external schema registry. And it's really as simple as JSON. Has great language support. https://github.com/thekvs/cpp-serializers#results https://github.com/thekvs/cpp-serializers#results That said, currently I don't work with it in any projects. Apparently it hasn't really caught on.

dickeytk 7y ago

don't forget https://capnproto.org/ https://capnproto.org/

mumblemumble 7y ago

I guess. . . for my part, it's always seemed to me like messagepack falls into a sort of uncanny valley of serialization formats. Schemaless is very undesirable for internal APIs, IMO. Plaintext is desirable for public-facing APIs that don't need to be particularly performant, but only if you're willing to go full HATEOAS. If you're going binary for performance or don't want to bother with HATEOAS, I'm back to preferring a published schema over something that's (semi-)self-describing plus some documentation that's invariably incomplete or out of date.

ClumsyPilot 7y ago
I have never seen any benefits of HATEOAS materialise - it is too free-form to be machine parceable, is it far more difficult to navigate than Swagger or GraphQ. Maybe I dont get something?
- mumblemumble 7y ago
  I don't ever machine parse it, but it can be useful for manual discovery if you get an API that isn't super well documented. I think I might generally prefer Swagger, too, but not everything is implemented in a language that has a library like that. At the end of the day, though, I'd seriously much rather just have a halfway well commented *.proto file. Work smarter, not harder. It's just that I won't recommend that for public-facing APIs because GRPC isn't universally well supported, while JSON over HTTP is.

ken 7y ago

Protobuf/Thrift require predefined schemas, which is a different model. Avro is pretty similar to MsgPack in concept. It seems less popular (at least in my circles), and has fewer implementations, which may or may not matter. As for performance and efficiency, the first benchmark I found [1] shows MsgPack is more space efficient, and faster to serialize, while Avro is faster to de-serialize. The second benchmark I found [2] found exactly the opposite. So it's not clear to me that either is better, and I'm sure it depends on your data, which library you're using, and how you're using it. Being nearly as performant as the predefined-schema serializers, while being nearly a drop-in replacement for JSON, seems like a major and valuable use case to me. [1]: https://medium.com/@nitinpaliwal87/compression-and-serialization-techniques-benchmarking-fd1f34c1098b https://medium.com/@nitinpaliwal87/compression-and-serializa... [2]: https://github.com/saint1991/serialization-benchmark https://github.com/saint1991/serialization-benchmark

meritt 7y ago
> shows MsgPack is more space efficient While I see the same numbers as you do, there's no way a format that includes plaintext keys alongside the values is going to be smaller than a format that only includes a packed field number+type and the corresponding values. There's something fishy going on here, and I can't seem to find a link to the source behind his benchmarks?
- deleted 7y ago
  [deleted]
- ludocode 7y ago
  I'm not sure if the benchmarks above reflect this, but MessagePack allows you to use integers (or any data type) for map keys. This makes it a lot smaller than using plaintext keys, and makes it comparable to formats like Protobuf despite the lack of formal schema. (You also lose a lot of the context-free readability of messages, so it's a tradeoff.)

2J0 7y ago

Whatever was wrong with ASN.1 for the wire? it's not a direct replacement but a sanitising schema that's truly amazing in application breadth, support, tooling, and implementation efficiency. In force documentation: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.680-201508-I!!PDF-E&type=items https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-X.68... Blog on ASN.1 schema for JSON: https://www.obj-sys.com/blog/?p=508 https://www.obj-sys.com/blog/?p=508

jnwatson 7y ago
I really like ASN.1. The big difference is that messagepack is schemaless.
tptacek 7y ago
It's incredibly annoying to marshal and unmarshal arbitrary ASN.1/BER. Variable-length odd-bit-length length fields annoying. BER is like the worst parts of every other format, collected in a single standard.
imtringued 7y ago
I don't know why but I constantly hear about things that never caught on and will never catch on in the future. Ok lets say I am hyped for ASN.1 as a schema for JSON. How do I integrate it into my bog standard Java application? Suddenly we are running into a huge problem. >ASN.1 is a mature standard. As I already mentioned, it has been around since the 1980’s. Though stable, it is not stagnant; the most recent revision occurred in 2008. It might be old but it is not widely used. To me it doesn't matter if something was created yesterday or 20 years ago as long as I can use it and using ASN.1 is significantly harder than necessary. >ASN.1 has tool support. There are both commercial and open source tools that will generate code from your ASN.1 specification. Ok, so where is it and why is it so difficult to find? Even if we assume that we should just build it today then all the above claims suddenly become worthless. If someone builds ASN.1 tooling in 2020 then it is just as immature as e.g. GraphQL tooling that was built 2020. If there is renewed interest in ASN.1 then it might gain new features that will cause it to become less mature/stable again. Having "mature" software is useless if it doesn't meet user demands.

Scarbutt 7y ago

Because all your existing json will just work, you don't have to change your data representation to support some other format(proto, avro, thrift).