5 ms·

sigh. Why do we always want to declare a winner? Anyone who has tried to store binary data in JSON knows that MessagePack does indeed have a use case. And anyon

by judofyr 14y ago

sigh. Why do we always want to declare a winner? Anyone who has tried to store binary data in JSON knows that MessagePack does indeed have a use case. And anyone who is working with browsers knows that JSON/JS is the king there. > MessagePack is not smaller than gzip'd JSON But gzip'd MessagePack is smaller than gzip'd JSON, so? > MessagePack is not faster than JSON when a browser is involved Using it in the browser was never the use case for MessagePack. JSON these days are used for far more than browser-server-communication. > MessagePack is not human readable Fair enough. Although I don't consider JSON without newlines to be "human readable" either. I'll have to pipe it through a beautifier to read it. > MessagePack has issues with UTF-8 MessagePack only stores bytes. It's up to you to decide how to interpret those bytes. > Small packets also have significant TCP/IP overhead. And?

Luyt 14y ago
Or, a JSON AddOn for your browser, like http://jsonview.com/ http://jsonview.com/
benatkin 14y ago
> Using it in the browser was never the use case for MessagePack. Hopefully it will be soon, with degradation for old browsers. I'd like to be able to load an avatar as binary png data with the rest of the data.
- millerm 14y ago
  Assuming you are using a browser (other clients support this too)... For now, can't you just encode the image in your json using base 64 and use the 'data' uri scheme to decode it? Sure the encoded data can't be more than 32k in most browsers, but that is huge avatar.
  - falien 14y ago
    IE8 has the 32k limitation. I don't think anything else does.
    - millerm 14y ago
      Doh, you are correct. I had that piece of info reversed. Thanks for clarifying.
josephlord 14y ago
>> MessagePack has issues with UTF-8 > MessagePack only stores bytes. It's up to you to decide how to interpret those bytes. Then it's not really "like JSON" is it? I'm not saying that it is a bad thing or that it isn't a better option for many current uses of JSON but the "like JSON but fast and small" is a misleading pitch.
- mikeash 14y ago
  How's that? Just use UTF-8 for your strings and you're on par with JSON, no?
  - jerf 14y ago
    JSON can't carry binary data because it must be UTF-8. Glancing over the spec, msgpack commits the complementary sin, it can't carry UTF-8 data because it can only carry binary data. Yes, it is possible to stuff some UTF-8 bytes into a binary, but you still semantically have a binary, which could be anything. You must bring additional external information to the party to know what that binary hunk actually is. In theory this may sound like no big deal; in practice I've observed in similar cases it's a disaster. Average developers routinely muck this up (and that's just me being conservative, above-average ones can choke on this problem too). msgpack really ought to have a dedicated string type, and either declare that this string is always a particular encoding, or give a way to declare what encoding the string is in. (The second is more flexible and arguably more correct, but in something like this where there's going to be dozens of libraries trying to implement it, it is virtually guaranteed that a number of them will muck the variable encoding support up badly, so in practice I'd go with mandatory UTF-8 too.)
  - josephlord 14y ago
    For me JSON's strength is that you can throw typed data in simple structures straight at it and be confident of getting the same out the other end without worrying about the exact details of the encoding. If you exceed the supported types you have to work harder but in great many cases that is not necessary. If MsgPack doesn't let me throw strings, numbers plus arrays and dictionaries containing arbitrary further structures without me having to worry about defining the encoding it isn't on par. Obviously any sane data format can contain any data but not equally easily or efficiently.
adventured 14y ago
I think developers like to declare winners for very, very good reasons typically. Winners are often very useful. For example in settling on a standard once a good or great solution is found so a community can thrive around it, lots of open tools and open code can be built for it, developers can build upon it with a roadmap in mind, sites can publish with it knowing they can reach a maximum audience, etc etc etc etc. Who really wants to have even a dozen different 'standards' for data interchange? The gradual move from XML to JSON demonstrated perfectly well how messy it can all get, just trying to support two quasi standards.
- oscargrouch 14y ago
  Choice is a good thing.. do you use a fork, to open doors, eat,scratch your foot, kill flies, start a fire, just because you have to use the one-and-ultimate tool ? json are not the best for every scenario, as also msgpack does not have the best fitness for everything.. Nature is pretty wise on that matter, it "launch" several possibilities, cause the environment is always changing.. so in a particular giver scenario, one of the possibilities (biological agents) can succed.. but that one that succeded in a particular scenario, will fail in another, where now, another subject who has failed is the better option, adaptation.. so.. choise is a good thing.. why people are so lazy, and dont like to think? everything must be ready.. its our duty to choose the best tool to solve a given problem.. its cool that i can even use something "obsolet" from the 1800´s to solve a 2012 problem.. So if we keep thinking.. "oh that old unwanted thing from the past", we will never see it.. Winners are only winners in a given scenario.. its good to have choice, even against all boards and standards i prefer to decide from myself, than to have others to do that decision for me.. This is technology, not fashion.
- klodolph 14y ago
  Thank you for taking the time to write that, instead of linking to that one comic that everyone links to in these threads about competing standards. Although I'm not sure why you say "quasi" standard. XML and JSON seem like pretty robust standards to me, and they should both be around for a long time.
marcosdumay 14y ago
> Why do we always want to declare a winner? You are right. But I came to that page after reading about a faster and smaller JSON at HN, thus I was expecting to find a faster and smaller JSON. It's just another case of bad marketing hurting a brand.
- pjscott 14y ago
  Oh for crying out loud. In a lot of use cases, msgpack can be treated as a smaller, faster JSON. Let's do a quick experiment. Here's a small, very ordinary JSON document: http://graph.facebook.com/btaylor http://graph.facebook.com/btaylor Using Python's json and simplejson modules, which are quite zippy, the encoding time was about 11.4 us, and the decoding time was about 8 us. With msgpack, the encoding and decoding times were 2.7 us and 1.7 us, respectively. The encoded size was 187 bytes with JSON, and 140 bytes with msgpack. Zlib compression brought those down to 127 and 125 bytes, while bringing the total encode-and-compress time up to 25 us for JSON and 16 us for msgpack.
- Tloewald 14y ago
  My first thought was "maybe it would be a useful JSON replacement for Unity" (which still lacks native or even aolid third party JSON support). I'm not sure how big a deal the lack of seamless UTF8 supportwill be in practice (in most cases no big deal i think) but it does look promising owing to its C# implementation. I do agree the tagline is bound to annoy.
NelsonMinar 14y ago
I did a quick test on some JSON data I use in my webapps; ~100k-1M blobs, mostly numbers. gzipped MessagePack data was just about the same size as gzipped JSON data. If you care about size on the wire over HTTP, then MessagePack may not be an improvement. It's amazing how well gzip compresses the stupid ASCII encoding of numbers in JSON.
- Too 14y ago
  Doesn't have to be stupid. A normal integer in binary representation ALWAYS consumes 4 bytes. With string representation you have 1000 numbers that will beat that as they only need <=3 characters. Numbers in that range are actually used very frequently.
  - pjscott 14y ago
    Funny you should mention that. Msgpack uses a variable-length binary encoding for integers. It can, for example, fit integers in the interval [-32, 127] into a single byte. And unlike that fixed four-byte encoding you mention, it can also handle 64-bit integers. (I'm not disagreeing with you, or anything. I just like talking about binary encodings.)
    - jspthrowaway2 14y ago
      There is also zigzag encoding, something Protocol Buffers (I believe) introduced for general usage. Thrift later copied it for the compact encoding. It's pretty interesting.
  - rorrr 14y ago
    > A normal integer in binary representation ALWAYS consumes 4 bytes That's not true. Most languages have 1-byte integers, 2-byte integers and 4-byte integers.
    - reeses 14y ago
      Truly awesome languages use binary-coded decimal. Ok, I'll show myself out.
smosher 14y ago
Why do we always want to declare a winner? To be fair, MessagePack seems to be throwing down the gauntlet with their tagline. The parent comment also makes some important points that anyone considering the two should be aware of.
mcgwiz 14y ago
sigh Why do we always have to misconstrue messages that conflict with our personal beliefs? :) The GP was not declaring a winner. He was responding to overreaching title of the link. Yes there are similarities to JSON, and yes it may be faster and smaller, but only in certain cases. It is very relevant for the GP to point out that in the most common JSON cases (browser communication), those claims are basically false or only marginal. This is reasoned argumentation, not arbitrary judgement to declare The One.
masklinn 14y ago
> sigh. Why do we always want to declare a winner? You may want to ask whoever copywrote messagepack's website, they kind-of are the one declaring: > It's like JSON. but fast and small.

by judofyr 14y ago

Luyt 14y ago

Or, a JSON AddOn for your browser, like http://jsonview.com/ http://jsonview.com/

benatkin 14y ago

> Using it in the browser was never the use case for MessagePack. Hopefully it will be soon, with degradation for old browsers. I'd like to be able to load an avatar as binary png data with the rest of the data.

millerm 14y ago
Assuming you are using a browser (other clients support this too)... For now, can't you just encode the image in your json using base 64 and use the 'data' uri scheme to decode it? Sure the encoded data can't be more than 32k in most browsers, but that is huge avatar.
- falien 14y ago
  IE8 has the 32k limitation. I don't think anything else does.
  - millerm 14y ago
    Doh, you are correct. I had that piece of info reversed. Thanks for clarifying.

josephlord 14y ago

>> MessagePack has issues with UTF-8 > MessagePack only stores bytes. It's up to you to decide how to interpret those bytes. Then it's not really "like JSON" is it? I'm not saying that it is a bad thing or that it isn't a better option for many current uses of JSON but the "like JSON but fast and small" is a misleading pitch.

mikeash 14y ago
How's that? Just use UTF-8 for your strings and you're on par with JSON, no?
- jerf 14y ago
  JSON can't carry binary data because it must be UTF-8. Glancing over the spec, msgpack commits the complementary sin, it can't carry UTF-8 data because it can only carry binary data. Yes, it is possible to stuff some UTF-8 bytes into a binary, but you still semantically have a binary, which could be anything. You must bring additional external information to the party to know what that binary hunk actually is. In theory this may sound like no big deal; in practice I've observed in similar cases it's a disaster. Average developers routinely muck this up (and that's just me being conservative, above-average ones can choke on this problem too). msgpack really ought to have a dedicated string type, and either declare that this string is always a particular encoding, or give a way to declare what encoding the string is in. (The second is more flexible and arguably more correct, but in something like this where there's going to be dozens of libraries trying to implement it, it is virtually guaranteed that a number of them will muck the variable encoding support up badly, so in practice I'd go with mandatory UTF-8 too.)
- josephlord 14y ago
  For me JSON's strength is that you can throw typed data in simple structures straight at it and be confident of getting the same out the other end without worrying about the exact details of the encoding. If you exceed the supported types you have to work harder but in great many cases that is not necessary. If MsgPack doesn't let me throw strings, numbers plus arrays and dictionaries containing arbitrary further structures without me having to worry about defining the encoding it isn't on par. Obviously any sane data format can contain any data but not equally easily or efficiently.

adventured 14y ago

I think developers like to declare winners for very, very good reasons typically. Winners are often very useful. For example in settling on a standard once a good or great solution is found so a community can thrive around it, lots of open tools and open code can be built for it, developers can build upon it with a roadmap in mind, sites can publish with it knowing they can reach a maximum audience, etc etc etc etc. Who really wants to have even a dozen different 'standards' for data interchange? The gradual move from XML to JSON demonstrated perfectly well how messy it can all get, just trying to support two quasi standards.

oscargrouch 14y ago
Choice is a good thing.. do you use a fork, to open doors, eat,scratch your foot, kill flies, start a fire, just because you have to use the one-and-ultimate tool ? json are not the best for every scenario, as also msgpack does not have the best fitness for everything.. Nature is pretty wise on that matter, it "launch" several possibilities, cause the environment is always changing.. so in a particular giver scenario, one of the possibilities (biological agents) can succed.. but that one that succeded in a particular scenario, will fail in another, where now, another subject who has failed is the better option, adaptation.. so.. choise is a good thing.. why people are so lazy, and dont like to think? everything must be ready.. its our duty to choose the best tool to solve a given problem.. its cool that i can even use something "obsolet" from the 1800´s to solve a 2012 problem.. So if we keep thinking.. "oh that old unwanted thing from the past", we will never see it.. Winners are only winners in a given scenario.. its good to have choice, even against all boards and standards i prefer to decide from myself, than to have others to do that decision for me.. This is technology, not fashion.
klodolph 14y ago
Thank you for taking the time to write that, instead of linking to that one comic that everyone links to in these threads about competing standards. Although I'm not sure why you say "quasi" standard. XML and JSON seem like pretty robust standards to me, and they should both be around for a long time.

marcosdumay 14y ago

> Why do we always want to declare a winner? You are right. But I came to that page after reading about a faster and smaller JSON at HN, thus I was expecting to find a faster and smaller JSON. It's just another case of bad marketing hurting a brand.

pjscott 14y ago
Oh for crying out loud. In a lot of use cases, msgpack can be treated as a smaller, faster JSON. Let's do a quick experiment. Here's a small, very ordinary JSON document: http://graph.facebook.com/btaylor http://graph.facebook.com/btaylor Using Python's json and simplejson modules, which are quite zippy, the encoding time was about 11.4 us, and the decoding time was about 8 us. With msgpack, the encoding and decoding times were 2.7 us and 1.7 us, respectively. The encoded size was 187 bytes with JSON, and 140 bytes with msgpack. Zlib compression brought those down to 127 and 125 bytes, while bringing the total encode-and-compress time up to 25 us for JSON and 16 us for msgpack.
Tloewald 14y ago
My first thought was "maybe it would be a useful JSON replacement for Unity" (which still lacks native or even aolid third party JSON support). I'm not sure how big a deal the lack of seamless UTF8 supportwill be in practice (in most cases no big deal i think) but it does look promising owing to its C# implementation. I do agree the tagline is bound to annoy.

NelsonMinar 14y ago

I did a quick test on some JSON data I use in my webapps; ~100k-1M blobs, mostly numbers. gzipped MessagePack data was just about the same size as gzipped JSON data. If you care about size on the wire over HTTP, then MessagePack may not be an improvement. It's amazing how well gzip compresses the stupid ASCII encoding of numbers in JSON.

Too 14y ago
Doesn't have to be stupid. A normal integer in binary representation ALWAYS consumes 4 bytes. With string representation you have 1000 numbers that will beat that as they only need <=3 characters. Numbers in that range are actually used very frequently.
- pjscott 14y ago
  Funny you should mention that. Msgpack uses a variable-length binary encoding for integers. It can, for example, fit integers in the interval [-32, 127] into a single byte. And unlike that fixed four-byte encoding you mention, it can also handle 64-bit integers. (I'm not disagreeing with you, or anything. I just like talking about binary encodings.)
  - jspthrowaway2 14y ago
    There is also zigzag encoding, something Protocol Buffers (I believe) introduced for general usage. Thrift later copied it for the compact encoding. It's pretty interesting.
- rorrr 14y ago
  > A normal integer in binary representation ALWAYS consumes 4 bytes That's not true. Most languages have 1-byte integers, 2-byte integers and 4-byte integers.
  - reeses 14y ago
    Truly awesome languages use binary-coded decimal. Ok, I'll show myself out.

smosher 14y ago

Why do we always want to declare a winner? To be fair, MessagePack seems to be throwing down the gauntlet with their tagline. The parent comment also makes some important points that anyone considering the two should be aware of.

mcgwiz 14y ago

sigh Why do we always have to misconstrue messages that conflict with our personal beliefs? :) The GP was not declaring a winner. He was responding to overreaching title of the link. Yes there are similarities to JSON, and yes it may be faster and smaller, but only in certain cases. It is very relevant for the GP to point out that in the most common JSON cases (browser communication), those claims are basically false or only marginal. This is reasoned argumentation, not arbitrary judgement to declare The One.

masklinn 14y ago

> sigh. Why do we always want to declare a winner? You may want to ask whoever copywrote messagepack's website, they kind-of are the one declaring: > It's like JSON. but fast and small.