3 ms·

In your opinion, what's the value of MessagePack? I worked with it when writing some Fluentd tools, and it was neat, but I didn't love it enough to switch from

In your opinion, what's the value of MessagePack? I worked with it when writing some Fluentd tools, and it was neat, but I didn't love it enough to switch from JSON on other projects. Maybe there's a killer feature I didn't know I need?

alexchamberlain 7y ago
In recent benchmarks I ran for a project, the performance of MessagePack blew JSON out the water. Obviously, it may differ in your case, but generally, MessagePack is designed to be machine readable and easy to parse, so performance with generally be better. Message sizes are generally better too.
Mordak 7y ago
At my work we recently went through a large exercise to decide on a common data storage format. The contenders were JSON, MessagePack, and Avro. MessagePack won because: - Msgpack serialization and deserialization is very fast in many languages - often 100x faster then JSON - Msgpack natively supports encoding binary data - Msgpack has type extensions, making it trivial to represent common types in an efficient way (eg. IPv4 address, timestamps) - Msgpack has good libraries available in many languages If you do not care about those things (no binary data, no need for extended types, not performance critical) then JSON is just fine.
- pdimitar 7y ago
  I'm curious why didn't you consider FlatBuffers as well.
  - Mordak 7y ago
    FlatBuffers are not self-describing. FlatBuffers, Protobuf, Cap'n Proto, etc., all require an external schema configuration that you compile into a code chunk that you include into your program. Without this it is impossible to make sense of the data. In our case, the data is semi-structured and changes frequently. The prospect of maintaining a schema registry for all the data users and keeping everyone up to date and backwards compatible is enough of a burden that it was excluded. Avro also uses schemas, but since the schema is embedded in the file it is self-describing so the reader does not need to do anything special to interpret the data. But Avro's C library is buggy and the python deserialization performance was terrible, so Avro was not selected.
camgunz 7y ago
(TL;DR: If you're looking for a low-overhead binary serialization format, MessagePack is great. It sits somewhere between "send JSON to a JS client" and "I have a fleet of backend services all sending data back and forth".) MessagePack is marketed as "It's like JSON, but fast and small", but I don't view it as a JSON replacement at all. I think JSON is great for most projects, especially anything talking to a JS front end, and as soon as you move into something requiring more performance, you probably need schema/versioning guarantees that something like FlatBuffers or Cap'n Proto would give you. I think MessagePack is a great on-disk binary format, and a pretty good network binary format if you don't have complicated application architecture (i.e., a video game client/server, a chat client, etc.) It's way, way faster than JSON which can be important for mobile, IoT, and embedded work). It's more compact than JSON if you want to avoid the overhead of gzipping/gunzipping everything. It compresses similarly as JSON+gzip if that's important to you. It's not confounded by things like canonicalization or attempts to add a schema. It's also amenable to streaming, something you... mostly can't do with JSON without going to a lot of trouble. It's also easy to implement -- which is important in some cases. Try implementing a JSON encoder/decoder!
- PopeDotNinja 7y ago
  I did write a json encoder/decoder! I was just goofing around and decided to see how hard it was. Apparently I didn't get around to flushing out the docs, so here's the tests: https://github.com/amorphid/json_momoa/blob/master/test/json_momoa/parser_test.exs https://github.com/amorphid/json_momoa/blob/master/test/json...
  - camgunz 7y ago
    Incredible name..... Edit: to be a little more constructive, I read this a while ago and it scared me off implementing anything JSON related forever: http://seriot.ch/parsing_json.php#4 http://seriot.ch/parsing_json.php#4.
    - PopeDotNinja 7y ago
      I found it much more pleasant to parse JSON than YAML. At least there's a fairly soecific RFC for JSON! YAML parsing felt like a whole bunch of "meh, we came up with this, but just sorta do what you want". And on the topic of cute names, back when I was learning Elixir, I wrote a wrapper for an Erlang YAML parser. I called it Mark Yamill :) https://hex.pm/packages/mark_yamill https://hex.pm/packages/mark_yamill

alexchamberlain 7y ago

In recent benchmarks I ran for a project, the performance of MessagePack blew JSON out the water. Obviously, it may differ in your case, but generally, MessagePack is designed to be machine readable and easy to parse, so performance with generally be better. Message sizes are generally better too.

Mordak 7y ago

At my work we recently went through a large exercise to decide on a common data storage format. The contenders were JSON, MessagePack, and Avro. MessagePack won because: - Msgpack serialization and deserialization is very fast in many languages - often 100x faster then JSON - Msgpack natively supports encoding binary data - Msgpack has type extensions, making it trivial to represent common types in an efficient way (eg. IPv4 address, timestamps) - Msgpack has good libraries available in many languages If you do not care about those things (no binary data, no need for extended types, not performance critical) then JSON is just fine.

pdimitar 7y ago
I'm curious why didn't you consider FlatBuffers as well.
- Mordak 7y ago
  FlatBuffers are not self-describing. FlatBuffers, Protobuf, Cap'n Proto, etc., all require an external schema configuration that you compile into a code chunk that you include into your program. Without this it is impossible to make sense of the data. In our case, the data is semi-structured and changes frequently. The prospect of maintaining a schema registry for all the data users and keeping everyone up to date and backwards compatible is enough of a burden that it was excluded. Avro also uses schemas, but since the schema is embedded in the file it is self-describing so the reader does not need to do anything special to interpret the data. But Avro's C library is buggy and the python deserialization performance was terrible, so Avro was not selected.

camgunz 7y ago

(TL;DR: If you're looking for a low-overhead binary serialization format, MessagePack is great. It sits somewhere between "send JSON to a JS client" and "I have a fleet of backend services all sending data back and forth".) MessagePack is marketed as "It's like JSON, but fast and small", but I don't view it as a JSON replacement at all. I think JSON is great for most projects, especially anything talking to a JS front end, and as soon as you move into something requiring more performance, you probably need schema/versioning guarantees that something like FlatBuffers or Cap'n Proto would give you. I think MessagePack is a great on-disk binary format, and a pretty good network binary format if you don't have complicated application architecture (i.e., a video game client/server, a chat client, etc.) It's way, way faster than JSON which can be important for mobile, IoT, and embedded work). It's more compact than JSON if you want to avoid the overhead of gzipping/gunzipping everything. It compresses similarly as JSON+gzip if that's important to you. It's not confounded by things like canonicalization or attempts to add a schema. It's also amenable to streaming, something you... mostly can't do with JSON without going to a lot of trouble. It's also easy to implement -- which is important in some cases. Try implementing a JSON encoder/decoder!

PopeDotNinja 7y ago
I did write a json encoder/decoder! I was just goofing around and decided to see how hard it was. Apparently I didn't get around to flushing out the docs, so here's the tests: https://github.com/amorphid/json_momoa/blob/master/test/json_momoa/parser_test.exs https://github.com/amorphid/json_momoa/blob/master/test/json...
- camgunz 7y ago
  Incredible name..... Edit: to be a little more constructive, I read this a while ago and it scared me off implementing anything JSON related forever: http://seriot.ch/parsing_json.php#4 http://seriot.ch/parsing_json.php#4.
  - PopeDotNinja 7y ago
    I found it much more pleasant to parse JSON than YAML. At least there's a fairly soecific RFC for JSON! YAML parsing felt like a whole bunch of "meh, we came up with this, but just sorta do what you want". And on the topic of cute names, back when I was learning Elixir, I wrote a wrapper for an Erlang YAML parser. I called it Mark Yamill :) https://hex.pm/packages/mark_yamill https://hex.pm/packages/mark_yamill