5 ms·

MessagePack - Extremely efficient object serialization library

28 points by kzk_mover 17y ago

MessagePack is a binary-based efficient object serialization library. It enables to exchange structured objects between many languages like JSON. But unlike JSON, it is very fast and small.<p>Typical small integer (like flags or error code) is saved only in 1 byte, and typical short string only needs 1 byte except the length of the string itself

jrockway 17y ago
I bet gzipped JSON compares favorably for larger objects, though, and has the advantage of not being a new protocol. The fact that it supports Haskell and Perl with proper hackagedb / cpan modules is nice, however. Very rare for web 2.0 stuff.
- houseabsolute 17y ago
  I don't understand why JSON would get faster for larger types. Anyway a 524 byte type is already fairly large.
  - jrockway 17y ago
    No, the compression algorithms begin to behave as well as hand-optimized binary protocols as the message size increases, because the constant overhead for mapping away the redundancy becomes a small percentage of the overall message.
    - houseabsolute 17y ago
      But if just building the JSON representation already takes more time than the binary protocol, I can't see how adding gzip to it is going to make it faster.
- KirkWylie 17y ago
  gzipping JSON or XML compares relatively favorably against a pure-binary protocol for suitably large messages, but not for pretty small messages. In addition, the speed of the compression stream really becomes an issue if you're sending messages within a cluster. Over the internet your bandwidth and latency will start to become more important, but within a data centre all pure-binary protocols are much faster.
slpsys 17y ago
Very interesting, though I'll take more of a look later on. The speed test does seem a little contrived, as it's for one case, and it looks [briefly] like some of the gains are due to greater specificity of primitive types.
prodigal_erik 17y ago
I recommend ASN.1 PER over any of the myriad byte-aligned no-schema formats that waste bits on ranges of values and entire types you aren't sending. But this is better than some other formats I've seen.
- andrewcooke 17y ago
  isn't ASN.1 proprietary? i wanted to implement it once and found i had to pay someone for specs. is that wrong? if not, i think many people would prefer something more open (more implementations; less likely to die from lack of support) [edit: sorry, i am talking rubbish here. they are all linked from here, for example: http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One i have no idea why i thought they were closed. has the ITU policy changed recently?]
  - alavrik 17y ago
    The specs were definitely closed last time when I checked (about 6 months ago). Still don't think this old telecom technology is superior compared to modern serialization tools like Protocol Buffers.
    - viraptor 17y ago
      old and modern do not mean anything in programming world. C is old - doesn't mean it's better or worse than modern languages. Could you rephrase it in terms of features we can actually compare? I prefer to rely on "old telecom technology" many times, since it's meant to be as close as possible to 100% reliability (telecom being the keyword, not old).
      - alavrik 17y ago
        I agree that old and modern are not very appropriate terms when comparing technologies. However I didn't mean to express an objective statement in my comment.
    - 17y ago
houseabsolute 17y ago
Why does their graph show the JSON bar as twice as tall as protocol buffers when it's not even 50% slower.
- viraptor 17y ago
  PB takes 0.47s, JSON takes 1.2s. JSON is 2.55 times slower, so it's over 2 times higher... Not sure where the problem is.
  - houseabsolute 17y ago
    The label for the JSON column on http://msgpack.sourceforge.net/ http://msgpack.sourceforge.net/ is "0.68 sec" and the label on the PB column is "0.47sec". So, yes, if the times were what you claimed, the graph would be fine, but according to the graph they were not.
    - viraptor 17y ago
      I think we're hitting some browser cache issue. Could you try force-reloading everything? My graph shows the numbers I mentioned above. Maybe the old version showed something different (if I wget the image without any caching, I still get MP: 0.11s, PB: 0.47s, JSON: 1.2s)
      - houseabsolute 17y ago
        Browser cache it is. After a reload the number shows as 1.2.
deleted 17y ago
[deleted]
KirkWylie 17y ago
Why not just use Avro if you want a packed encoding system that's schema-bound, or Fudge if you want one that's schema free and self-describing? Link to Avro: http://hadoop.apache.org/avro/ http://hadoop.apache.org/avro/ Link to Fudge: http://www.fudgemsg.org/ http://www.fudgemsg.org/ Couldn't do hyperlinks for some reason. Perhaps because I'm a moron.

28 points by kzk_mover 17y ago

jrockway 17y ago

I bet gzipped JSON compares favorably for larger objects, though, and has the advantage of not being a new protocol. The fact that it supports Haskell and Perl with proper hackagedb / cpan modules is nice, however. Very rare for web 2.0 stuff.

houseabsolute 17y ago
I don't understand why JSON would get faster for larger types. Anyway a 524 byte type is already fairly large.
- jrockway 17y ago
  No, the compression algorithms begin to behave as well as hand-optimized binary protocols as the message size increases, because the constant overhead for mapping away the redundancy becomes a small percentage of the overall message.
  - houseabsolute 17y ago
    But if just building the JSON representation already takes more time than the binary protocol, I can't see how adding gzip to it is going to make it faster.
KirkWylie 17y ago
gzipping JSON or XML compares relatively favorably against a pure-binary protocol for suitably large messages, but not for pretty small messages. In addition, the speed of the compression stream really becomes an issue if you're sending messages within a cluster. Over the internet your bandwidth and latency will start to become more important, but within a data centre all pure-binary protocols are much faster.

slpsys 17y ago

Very interesting, though I'll take more of a look later on. The speed test does seem a little contrived, as it's for one case, and it looks [briefly] like some of the gains are due to greater specificity of primitive types.

prodigal_erik 17y ago

I recommend ASN.1 PER over any of the myriad byte-aligned no-schema formats that waste bits on ranges of values and entire types you aren't sending. But this is better than some other formats I've seen.

andrewcooke 17y ago
isn't ASN.1 proprietary? i wanted to implement it once and found i had to pay someone for specs. is that wrong? if not, i think many people would prefer something more open (more implementations; less likely to die from lack of support) [edit: sorry, i am talking rubbish here. they are all linked from here, for example: http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One i have no idea why i thought they were closed. has the ITU policy changed recently?]
- alavrik 17y ago
  The specs were definitely closed last time when I checked (about 6 months ago). Still don't think this old telecom technology is superior compared to modern serialization tools like Protocol Buffers.
  - viraptor 17y ago
    old and modern do not mean anything in programming world. C is old - doesn't mean it's better or worse than modern languages. Could you rephrase it in terms of features we can actually compare? I prefer to rely on "old telecom technology" many times, since it's meant to be as close as possible to 100% reliability (telecom being the keyword, not old).
    - alavrik 17y ago
      I agree that old and modern are not very appropriate terms when comparing technologies. However I didn't mean to express an objective statement in my comment.
  - 17y ago

houseabsolute 17y ago

Why does their graph show the JSON bar as twice as tall as protocol buffers when it's not even 50% slower.

viraptor 17y ago
PB takes 0.47s, JSON takes 1.2s. JSON is 2.55 times slower, so it's over 2 times higher... Not sure where the problem is.
- houseabsolute 17y ago
  The label for the JSON column on http://msgpack.sourceforge.net/ http://msgpack.sourceforge.net/ is "0.68 sec" and the label on the PB column is "0.47sec". So, yes, if the times were what you claimed, the graph would be fine, but according to the graph they were not.
  - viraptor 17y ago
    I think we're hitting some browser cache issue. Could you try force-reloading everything? My graph shows the numbers I mentioned above. Maybe the old version showed something different (if I wget the image without any caching, I still get MP: 0.11s, PB: 0.47s, JSON: 1.2s)
    - houseabsolute 17y ago
      Browser cache it is. After a reload the number shows as 1.2.

deleted 17y ago

[deleted]

KirkWylie 17y ago

Why not just use Avro if you want a packed encoding system that's schema-bound, or Fudge if you want one that's schema free and self-describing? Link to Avro: http://hadoop.apache.org/avro/ http://hadoop.apache.org/avro/ Link to Fudge: http://www.fudgemsg.org/ http://www.fudgemsg.org/ Couldn't do hyperlinks for some reason. Perhaps because I'm a moron.