6 ms·

msgpack always seemed an odd thing to me. Compressed json (gzip, lz*) is small, and fast (see: http://news.ycombinator.com/item?id=4091051 http://news.ycombinat

msgpack always seemed an odd thing to me. Compressed json (gzip, lz*) is small, and fast (see: http://news.ycombinator.com/item?id=4091051 http://news.ycombinator.com/item?id=4091051). If you need structure, use protocol buffers or thrift. I actually like tnetstrings for backend messaging, but I don't see it used very often. json is pretty damn ubiquitous these days.

kiyoto 14y ago
I think your view, while warranted, is very web programming centric. MessagePack was initially developed for RPC. That might explain some of the "oddities" when viewed as a drop-in, space-efficient replacement for JSON.
- prodigal_erik 14y ago
  msgpack.org probably shouldn't be promoting it as a "perfect replacement", when valid messages which have no JSON equivalent have already been seen in practice. At best it's a superset, and if none of the implementations can confine you to a JSON-compatible subset (no byte strings, no int64, no non-string keys) which assures interop, that's a problem. Mostly I wish people would agree on a schema and use ASN.1 PER, rather than choosing from the ever-growing list of binary type-length-value formats which put a redundant copy of the schema on the wire in every message (making them neither small nor readable). I've never had occasion to do anything useful with a message whose format wasn't already known when I wrote the code.
- stock_toaster 14y ago
  > I think your view, while warranted, is very web programming centric FWIW, I do very little "web work" (by which I assume you mean front-end?). Instead it is mostly api endpoints for mobile and server-to-server backend messaging. Lots of http transport stuff. If I were doing rpc, I would either not (and use a restful or type-2 api), or use something like protobuf or thrift. Msgpack just doesn't seem, to me, as a really great fit for anything in particular.
  - kiyoto 14y ago
    I don't mean frontend but anything related to web services, really. "Lots of http transport stuff" is still "web programming" (for me, at least). >Msgpack just doesn't seem, to me, as a really great fit for anything in particular. Pinterest disagrees: http://engineering.pinterest.com/posts/2012/memcache-games/ http://engineering.pinterest.com/posts/2012/memcache-games/ To give you some context, frsyuki developed MessagePack originally for RPC (he was building a distributed key-value store in college) and made a conscious decision not to add a string type for maximal cross-language compatibility. Like a lot of other technology, MessagePack has outlived its original purpose, and there is a lot of demand now for it to support the (utf-8) string type, not just byte array.
    - stock_toaster 14y ago
      > "Lots of http transport stuff" is still "web programming" (for me, at least). Web programming to me sounds like javascript and css. A the very least something to do with html output at some point. Oh well. > Pinterest disagrees: http://engineering.pinterest.com/posts/2012/memcache-games/ I didn't see any reference in that post to them trying compression. It would have been an interesting comparison. Good data point on usefulness though. Thanks for that.
makmanalp 14y ago
I think the problem with that analysis is that it ignores the gzip compression / decompression time. Also, that file is so uniform that it probably presents a best-case for gzip compression.
- mistercow 14y ago
  It seems unlikely to me that the compression/decompression time would outweigh the time to transmit 382 extra KB of data, but I suppose it depends on the application and how valuable computation time is on the server side.
latchkey 14y ago
Agreed. I tried to use msgpack and ran into all sorts of weird recursion issues (basically, it threw up on even the simplest objects). I ended up just using gzip and it works great.
- stock_toaster 14y ago
  I think if you control both ends of the interaction, lz4 or snappy may be interesting to use. But for most things, I do think gzip is better supported (especially http).
  - latchkey 14y ago
    I'm storing data in redis via node.js. It was taking forever to xfer it over the wire. I compressed it with gzip using node's built in support. Problem solved. I tried snappy, but found it was a) slower and b) bigger. Not having to have yet another dependency was also a plus for gzip.
- kiyoto 14y ago
  Which language's binding did you use? MessagePack is a protocol, not an implementation. Anyway, if gzipping worked great for you, that sounds like the way to go =)
nitrogen 14y ago
It's for speed. I write web software in Ruby that runs on embedded devices. This software needs to communicate with a C backend that is consuming much of the available CPU power, so there's not much time left to waste on parsing the messaging protocol. I did some tests and found that JSON is significantly slower than simple key-value pair parsing (implemented as a state machine in C), which itself is half as fast as msgpack. This doesn't even consider the overhead of gzip compression. Edit: here are my results, which lack the numbers for msgpack except a mention at the end (I hate linking to Posterous, but I haven't moved my blog yet): http://nitrogen.posterous.com/164964342 http://nitrogen.posterous.com/164964342
- badgar 14y ago
  Have you tried using thrift or protocol buffers, which aren't self-describing (introducing enormous overhead for <1MB messages.
  - nitrogen 14y ago
    I haven't. I'd expect them to be faster, but my biggest issue is CPU time rather than throughput. At any rate, my current solution is no longer a bottleneck on performance, but if it becomes one, I'll definitely be looking into other binary serialization formats.
- duaneb 14y ago
  > I write web software in Ruby that runs on embedded devices. Ok I gotta ask.... Why on earth would you do that? It seems like an exercise in masochism. I wasn't even aware that ruby would compile on embedded arches.
  - nitrogen 14y ago
    High-end embedded Linux (1.2GHz ARM CPUs). It saved development time, but I am considering rewriting everything in C. This is an example of what it's for (self-promoting link in 3, 2, 1...): http://www.nitrogenlogic.com/ http://www.nitrogenlogic.com/
azov 14y ago
Msgpack library is almost an order of magnitude smaller then protobuf lite, and almost 100 times smaller then full protobuf. This is not counting the size of generated protocol classes. There are situations when size matters :-)
drewcrawford 14y ago
For me the advantage of msgpack is simplicity / efficiency in parsing the response. If you are decoding a JSON string, you run over the string twice: first to search for the terminating quote character so that you know how much memory to allocate, and second to copy the string. Whereas with msgpack, it's more like pascal-model strings where you know the length of the string up front; you can do it one pass. You also get some structural hints that make more of the parsing a parallelizable problem, whereas with JSON it is very difficult to get any benefit out of more cores. This may not make any difference if you are serving up an API request or three to x86 computers with broadband, but they make a lot of difference if the data you're working with is large relative to the bandwidth or latency of the pipe or the CPU power available on the other side of it.
- stock_toaster 14y ago
  > For me the advantage of msgpack is simplicity / efficiency in parsing the response If you like that, you might want to take a look at tnetstrings as a format.
  - drewcrawford 14y ago
    Simple, yes. Efficient, no. For example, msgpack can encode a 2-byte binary blob in 3 bytes. tnetstrings takes eleven bytes, for no appreciable benefit in decoding time. Also there's this gem: > You are not allowed to implement any of the following features: UTF-8 Strings glances at title of article

kiyoto 14y ago

I think your view, while warranted, is very web programming centric. MessagePack was initially developed for RPC. That might explain some of the "oddities" when viewed as a drop-in, space-efficient replacement for JSON.

prodigal_erik 14y ago
msgpack.org probably shouldn't be promoting it as a "perfect replacement", when valid messages which have no JSON equivalent have already been seen in practice. At best it's a superset, and if none of the implementations can confine you to a JSON-compatible subset (no byte strings, no int64, no non-string keys) which assures interop, that's a problem. Mostly I wish people would agree on a schema and use ASN.1 PER, rather than choosing from the ever-growing list of binary type-length-value formats which put a redundant copy of the schema on the wire in every message (making them neither small nor readable). I've never had occasion to do anything useful with a message whose format wasn't already known when I wrote the code.
stock_toaster 14y ago
> I think your view, while warranted, is very web programming centric FWIW, I do very little "web work" (by which I assume you mean front-end?). Instead it is mostly api endpoints for mobile and server-to-server backend messaging. Lots of http transport stuff. If I were doing rpc, I would either not (and use a restful or type-2 api), or use something like protobuf or thrift. Msgpack just doesn't seem, to me, as a really great fit for anything in particular.
- kiyoto 14y ago
  I don't mean frontend but anything related to web services, really. "Lots of http transport stuff" is still "web programming" (for me, at least). >Msgpack just doesn't seem, to me, as a really great fit for anything in particular. Pinterest disagrees: http://engineering.pinterest.com/posts/2012/memcache-games/ http://engineering.pinterest.com/posts/2012/memcache-games/ To give you some context, frsyuki developed MessagePack originally for RPC (he was building a distributed key-value store in college) and made a conscious decision not to add a string type for maximal cross-language compatibility. Like a lot of other technology, MessagePack has outlived its original purpose, and there is a lot of demand now for it to support the (utf-8) string type, not just byte array.
  - stock_toaster 14y ago
    > "Lots of http transport stuff" is still "web programming" (for me, at least). Web programming to me sounds like javascript and css. A the very least something to do with html output at some point. Oh well. > Pinterest disagrees: http://engineering.pinterest.com/posts/2012/memcache-games/ I didn't see any reference in that post to them trying compression. It would have been an interesting comparison. Good data point on usefulness though. Thanks for that.

makmanalp 14y ago

I think the problem with that analysis is that it ignores the gzip compression / decompression time. Also, that file is so uniform that it probably presents a best-case for gzip compression.

mistercow 14y ago
It seems unlikely to me that the compression/decompression time would outweigh the time to transmit 382 extra KB of data, but I suppose it depends on the application and how valuable computation time is on the server side.

latchkey 14y ago

Agreed. I tried to use msgpack and ran into all sorts of weird recursion issues (basically, it threw up on even the simplest objects). I ended up just using gzip and it works great.

stock_toaster 14y ago
I think if you control both ends of the interaction, lz4 or snappy may be interesting to use. But for most things, I do think gzip is better supported (especially http).
- latchkey 14y ago
  I'm storing data in redis via node.js. It was taking forever to xfer it over the wire. I compressed it with gzip using node's built in support. Problem solved. I tried snappy, but found it was a) slower and b) bigger. Not having to have yet another dependency was also a plus for gzip.
kiyoto 14y ago
Which language's binding did you use? MessagePack is a protocol, not an implementation. Anyway, if gzipping worked great for you, that sounds like the way to go =)

nitrogen 14y ago

It's for speed. I write web software in Ruby that runs on embedded devices. This software needs to communicate with a C backend that is consuming much of the available CPU power, so there's not much time left to waste on parsing the messaging protocol. I did some tests and found that JSON is significantly slower than simple key-value pair parsing (implemented as a state machine in C), which itself is half as fast as msgpack. This doesn't even consider the overhead of gzip compression. Edit: here are my results, which lack the numbers for msgpack except a mention at the end (I hate linking to Posterous, but I haven't moved my blog yet): http://nitrogen.posterous.com/164964342 http://nitrogen.posterous.com/164964342

badgar 14y ago
Have you tried using thrift or protocol buffers, which aren't self-describing (introducing enormous overhead for <1MB messages.
- nitrogen 14y ago
  I haven't. I'd expect them to be faster, but my biggest issue is CPU time rather than throughput. At any rate, my current solution is no longer a bottleneck on performance, but if it becomes one, I'll definitely be looking into other binary serialization formats.
duaneb 14y ago
> I write web software in Ruby that runs on embedded devices. Ok I gotta ask.... Why on earth would you do that? It seems like an exercise in masochism. I wasn't even aware that ruby would compile on embedded arches.
- nitrogen 14y ago
  High-end embedded Linux (1.2GHz ARM CPUs). It saved development time, but I am considering rewriting everything in C. This is an example of what it's for (self-promoting link in 3, 2, 1...): http://www.nitrogenlogic.com/ http://www.nitrogenlogic.com/

azov 14y ago

Msgpack library is almost an order of magnitude smaller then protobuf lite, and almost 100 times smaller then full protobuf. This is not counting the size of generated protocol classes. There are situations when size matters :-)

drewcrawford 14y ago

For me the advantage of msgpack is simplicity / efficiency in parsing the response. If you are decoding a JSON string, you run over the string twice: first to search for the terminating quote character so that you know how much memory to allocate, and second to copy the string. Whereas with msgpack, it's more like pascal-model strings where you know the length of the string up front; you can do it one pass. You also get some structural hints that make more of the parsing a parallelizable problem, whereas with JSON it is very difficult to get any benefit out of more cores. This may not make any difference if you are serving up an API request or three to x86 computers with broadband, but they make a lot of difference if the data you're working with is large relative to the bandwidth or latency of the pipe or the CPU power available on the other side of it.

stock_toaster 14y ago
> For me the advantage of msgpack is simplicity / efficiency in parsing the response If you like that, you might want to take a look at tnetstrings as a format.
- drewcrawford 14y ago
  Simple, yes. Efficient, no. For example, msgpack can encode a 2-byte binary blob in 3 bytes. tnetstrings takes eleven bytes, for no appreciable benefit in decoding time. Also there's this gem: > You are not allowed to implement any of the following features: UTF-8 Strings glances at title of article