5 ms·

MessagePack: Fast and Compact Serialization

50 points by tilt 15y ago

xal 15y ago
We use this as data exchange format for our RPC services at Shopify. Incredibly fast, stable, easy to use.
mdonahoe 15y ago
Is this actually better than gzipped JSON?
- roel_v 15y ago
  Depends on how you define 'better', I guess - at the very least it has to be several magnitudes faster, with no overhead for compression, parsing, etc.
- frsyuki 15y ago
  You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes. Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.
- rabidsnail 15y ago
  This appears to be mmap-friendly. Since arrays have length values it's possible to jump around in a large messagepack-serialized structure without having to scan it. In JSON if you want to get to element 2 of an array you have to scan to the end of element 1. If there's a length value at the beginning of element 1 (as it looks like there is with msgpack) then you can seek straight past it.
Ralith 15y ago
This sounds an awful lot like yet another ProtocolBuffers. What, if anything, new does it bring to the table?
- ericmoritz 15y ago
  This is schemaless. Protobufs require a .proto file that describes the data being serialized. A .proto file serves a similar purpose as the `struct` keyword in C. MessagePack uses type prefixes to describe the data. For instance an unsigned 8-bit int starts with a `0xcc` followed by a byte.
- frsyuki 15y ago
  Schemaless (dynamic typing) is the most obvious difference. I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242 http://news.ycombinator.com/item?id=2836242
- frsyuki 15y ago
  Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON. Since Protocol Buffers brings their original semantics, you must change lots of codes.
zdw 15y ago
Does it have it suitably low memory/CPU requirements to run on embedded devices? How well does it work on ARM/MIPS?
- frsyuki 15y ago
  It works very well at least on ARM. I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2. I've never tried to run it on MIPS. But it probably works if you use gcc. (I'm author of MessagePack)
dimva 15y ago
In node.js, JSON is actually faster than msgpack: https://github.com/aikar/wormhole/issues/3 https://github.com/aikar/wormhole/issues/3 Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.
- mraleph 15y ago
  > Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers. JavaScript _does_not_ have a number type wide enough to hold 64-bit integer. V8 API has nothing to do with it.
erluser 15y ago
Would be nice if suuport for RPC (and generally) for Erlang was better. It's "currently" in alpha.
jstephens 15y ago
For perl it's fast as long as you have a small payload. Here's my benchmarks using progressively bigger data structures. perl: 5.008008 Storable: 2.21 JSON::XS: 2.25 Data::MessagePack: 0.34 ==== Size ==== .-----------------+-----------+---------+---------. | src | storable | json | msgpack | +-----------------+-----------+---------+---------+ | 1 | 4 | fail | 1 | | 3.14 | 8 | fail | 5 | | {} | 7 | 2 | 1 | | [] | 7 | 2 | 1 | | [('a')x10] | 37 | 41 | 21 | | {('a')x10} | 14 | fail | 11 | | +{1,+{1,+{}}} | 29 | 14 | 7 | | +[+[+[]]] | 19 | 6 | 3 | '-----------------+-----------+---------+---------' ========================== Testing: theirs:HASH (4995 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.14 usr + 0.00 sys = 1.14 CPU) @ 26947.37/s (n=30720) mp: 1 wallclock secs ( 1.11 usr + 0.00 sys = 1.11 CPU) @ 70446.85/s (n=78196) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 18442.45/s (n=19549) Rate storable json mp storable 18442/s -- -32% -74% json 26947/s 46% -- -62% mp 70447/s 282% 161% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @ 23424.51/s (n=23893) mp: 1 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @ 35136.27/s (n=35839) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 25357.55/s (n=26879) Rate json storable mp json 23425/s -- -8% -33% storable 25358/s 8% -- -28% mp 35136/s 50% 39% -- ========================== Testing: xdbi:HASH (24138 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 2414.15/s (n=2559) mp: 1 wallclock secs ( 1.07 usr + 0.00 sys = 1.07 CPU) @ 8373.83/s (n=8960) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 7244.34/s (n=7679) Rate json storable mp json 2414/s -- -67% -71% storable 7244/s 200% -- -13% mp 8374/s 247% 16% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.11 usr + 0.00 sys = 1.11 CPU) @ 4842.34/s (n=5375) mp: 2 wallclock secs ( 1.13 usr + 0.00 sys = 1.13 CPU) @ 4324.78/s (n=4887) storable: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 4654.29/s (n=4887) Rate mp storable json mp 4325/s -- -7% -11% storable 4654/s 8% -- -4% json 4842/s 12% 4% -- ========================== Testing: pdf:HASH (35239 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.04 usr + 0.00 sys = 1.04 CPU) @ 9845.19/s (n=10239) mp: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 5119.05/s (n=5375) storable: 1 wallclock secs ( 1.10 usr + 0.00 sys = 1.10 CPU) @ 7518.18/s (n=8270) Rate mp storable json mp 5119/s -- -32% -48% storable 7518/s 47% -- -24% json 9845/s 92% 31% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.08 usr + 0.00 sys = 1.08 CPU) @ 3110.19/s (n=3359) mp: 1 wallclock secs ( 1.07 usr + 0.00 sys = 1.07 CPU) @ 2790.65/s (n=2986) storable: 1 wallclock secs ( 1.08 usr + 0.00 sys = 1.08 CPU) @ 3554.63/s (n=3839) Rate mp json storable mp 2791/s -- -10% -21% json 3110/s 11% -- -13% storable 3555/s 27% 14% -- ========================== Testing: globallib:HASH (197319 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @ 1760.55/s (n=1919) mp: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 791.51/s (n=839) storable: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 1279.05/s (n=1343) Rate mp storable json mp 792/s -- -38% -55% storable 1279/s 62% -- -27% json 1761/s 122% 38% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.71 usr + 0.00 sys = 1.71 CPU) @ 301.75/s (n=516) mp: 1 wallclock secs ( 1.03 usr + 0.00 sys = 1.03 CPU) @ 404.85/s (n=417) storable: 1 wallclock secs ( 1.12 usr + 0.00 sys = 1.12 CPU) @ 544.64/s (n=610) Rate json mp storable json 302/s -- -25% -45% mp 405/s 34% -- -26% storable 545/s 80% 35% --

50 points by tilt 15y ago

xal 15y ago

We use this as data exchange format for our RPC services at Shopify. Incredibly fast, stable, easy to use.

mdonahoe 15y ago

Is this actually better than gzipped JSON?

roel_v 15y ago
Depends on how you define 'better', I guess - at the very least it has to be several magnitudes faster, with no overhead for compression, parsing, etc.
frsyuki 15y ago
You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes. Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.
rabidsnail 15y ago
This appears to be mmap-friendly. Since arrays have length values it's possible to jump around in a large messagepack-serialized structure without having to scan it. In JSON if you want to get to element 2 of an array you have to scan to the end of element 1. If there's a length value at the beginning of element 1 (as it looks like there is with msgpack) then you can seek straight past it.

Ralith 15y ago

This sounds an awful lot like yet another ProtocolBuffers. What, if anything, new does it bring to the table?

ericmoritz 15y ago
This is schemaless. Protobufs require a .proto file that describes the data being serialized. A .proto file serves a similar purpose as the `struct` keyword in C. MessagePack uses type prefixes to describe the data. For instance an unsigned 8-bit int starts with a `0xcc` followed by a byte.
frsyuki 15y ago
Schemaless (dynamic typing) is the most obvious difference. I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242 http://news.ycombinator.com/item?id=2836242
frsyuki 15y ago
Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON. Since Protocol Buffers brings their original semantics, you must change lots of codes.

zdw 15y ago

Does it have it suitably low memory/CPU requirements to run on embedded devices? How well does it work on ARM/MIPS?

frsyuki 15y ago
It works very well at least on ARM. I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2. I've never tried to run it on MIPS. But it probably works if you use gcc. (I'm author of MessagePack)

dimva 15y ago

In node.js, JSON is actually faster than msgpack: https://github.com/aikar/wormhole/issues/3 https://github.com/aikar/wormhole/issues/3 Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

mraleph 15y ago
> Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers. JavaScript _does_not_ have a number type wide enough to hold 64-bit integer. V8 API has nothing to do with it.

erluser 15y ago

Would be nice if suuport for RPC (and generally) for Erlang was better. It's "currently" in alpha.

jstephens 15y ago

For perl it's fast as long as you have a small payload. Here's my benchmarks using progressively bigger data structures. perl: 5.008008 Storable: 2.21 JSON::XS: 2.25 Data::MessagePack: 0.34 ==== Size ==== .-----------------+-----------+---------+---------. | src | storable | json | msgpack | +-----------------+-----------+---------+---------+ | 1 | 4 | fail | 1 | | 3.14 | 8 | fail | 5 | | {} | 7 | 2 | 1 | | [] | 7 | 2 | 1 | | [('a')x10] | 37 | 41 | 21 | | {('a')x10} | 14 | fail | 11 | | +{1,+{1,+{}}} | 29 | 14 | 7 | | +[+[+[]]] | 19 | 6 | 3 | '-----------------+-----------+---------+---------' ========================== Testing: theirs:HASH (4995 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.14 usr + 0.00 sys = 1.14 CPU) @ 26947.37/s (n=30720) mp: 1 wallclock secs ( 1.11 usr + 0.00 sys = 1.11 CPU) @ 70446.85/s (n=78196) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 18442.45/s (n=19549) Rate storable json mp storable 18442/s -- -32% -74% json 26947/s 46% -- -62% mp 70447/s 282% 161% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @ 23424.51/s (n=23893) mp: 1 wallclock secs ( 1.02 usr + 0.00 sys = 1.02 CPU) @ 35136.27/s (n=35839) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 25357.55/s (n=26879) Rate json storable mp json 23425/s -- -8% -33% storable 25358/s 8% -- -28% mp 35136/s 50% 39% -- ========================== Testing: xdbi:HASH (24138 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 2414.15/s (n=2559) mp: 1 wallclock secs ( 1.07 usr + 0.00 sys = 1.07 CPU) @ 8373.83/s (n=8960) storable: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 7244.34/s (n=7679) Rate json storable mp json 2414/s -- -67% -71% storable 7244/s 200% -- -13% mp 8374/s 247% 16% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.11 usr + 0.00 sys = 1.11 CPU) @ 4842.34/s (n=5375) mp: 2 wallclock secs ( 1.13 usr + 0.00 sys = 1.13 CPU) @ 4324.78/s (n=4887) storable: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 4654.29/s (n=4887) Rate mp storable json mp 4325/s -- -7% -11% storable 4654/s 8% -- -4% json 4842/s 12% 4% -- ========================== Testing: pdf:HASH (35239 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.04 usr + 0.00 sys = 1.04 CPU) @ 9845.19/s (n=10239) mp: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 5119.05/s (n=5375) storable: 1 wallclock secs ( 1.10 usr + 0.00 sys = 1.10 CPU) @ 7518.18/s (n=8270) Rate mp storable json mp 5119/s -- -32% -48% storable 7518/s 47% -- -24% json 9845/s 92% 31% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.08 usr + 0.00 sys = 1.08 CPU) @ 3110.19/s (n=3359) mp: 1 wallclock secs ( 1.07 usr + 0.00 sys = 1.07 CPU) @ 2790.65/s (n=2986) storable: 1 wallclock secs ( 1.08 usr + 0.00 sys = 1.08 CPU) @ 3554.63/s (n=3839) Rate mp json storable mp 2791/s -- -10% -21% json 3110/s 11% -- -13% storable 3555/s 27% 14% -- ========================== Testing: globallib:HASH (197319 bytes) ==== Serialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.09 usr + 0.00 sys = 1.09 CPU) @ 1760.55/s (n=1919) mp: 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @ 791.51/s (n=839) storable: 1 wallclock secs ( 1.05 usr + 0.00 sys = 1.05 CPU) @ 1279.05/s (n=1343) Rate mp storable json mp 792/s -- -38% -55% storable 1279/s 62% -- -27% json 1761/s 122% 38% -- ==== Deserialize ==== Benchmark: running json, mp, storable for at least 1 CPU seconds... json: 1 wallclock secs ( 1.71 usr + 0.00 sys = 1.71 CPU) @ 301.75/s (n=516) mp: 1 wallclock secs ( 1.03 usr + 0.00 sys = 1.03 CPU) @ 404.85/s (n=417) storable: 1 wallclock secs ( 1.12 usr + 0.00 sys = 1.12 CPU) @ 544.64/s (n=610) Rate json mp storable json 302/s -- -25% -45% mp 405/s 34% -- -26% storable 545/s 80% 35% --