10 ms·
I took the first example from http://www.json.org/example.html http://www.json.org/example.html and msgpack makes it 304 bytes. a simple gzip on the json is 289
by supermatt 7y ago
I took the first example from http://www.json.org/example.html http://www.json.org/example.html and msgpack makes it 304 bytes. a simple gzip on the json is 289 bytes. The larger examples are even more in gzips "favour"
I am not 100% sure why I would choose to use this - maybe for super tiny documents?
- pletnes 7y agoMsgpack is a different encoding, not quite the same as compression. What happens if you do both?
- blattimwind 7y agogzip is very slow. I don't know why anyone would still use it today.
- jgalt212 7y agoreally? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than the others we tested.
- thedance 7y agoWell yah, those are all ratio-tuned codecs that are very slow. LZMA beats bzip2 on both ratio and speed so you might as well forget about bzip2 forever. Zstd, snappy, LZ4, or even brotli are probably better choices than zlib for most people. Brotli has LZMA-like ratios at dramatically higher speeds.
- blattimwind 7y agoThe algorithms you name are all rather outdated. Typical gzip decompression speed is somewhere in the 200-250 MB/s region, compression is much slower. LZ4 for example tends to compress at ~600-700 MB/s, and decompress at several GB/s. zstd is tweakable over a very wide range of ratio-speed trade-offs. LZMA(2) (xz) is a rather troubled format and should not be used any more. bzip2 has always been slower than gzip with usually marginally better compression. It has been irrelevant for a long time.
- oefrha 7y ago> LZMA(2) (xv) is a rather troubled format and should not be used any more. You mean xz. Not sure where you got the idea that it’s a troubled format, but if you’re talking about the infamous “Xz format inadequate for long-term archiving”, IMO that’s just bzip2 authors taking a dump on xz for no good reason, and fortunately for us it’s bzip2 that’s basically irrelevant today, not xz.
- gnulinux 7y ago> It has been irrelevant for a long time. Funny you say that, my company uses bz2 for compressing pretty much everything.
- blattimwind 7y agoAnd many companies use fixed-width record formats for data exchange... what's your point exactly?
- gnulinux 7y agoI guess that it's not irrelevant. Deprecated, old, etc sure, but irrelevant?
- deleted 7y ago[deleted]
- ants_a 7y agohttps://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ Brotli is better, zstd is faster, lz4 is often good enough and a lot faster.
- kbutler 7y agoWhen you say, "very slow" what do you mean? gzip is fast relative to other compression algorithms and relative to internet speeds. gzip is slower than an ideal gigabit network, but faster than 100mb. https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/ https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...
- blattimwind 7y ago"gzip vs bzip2 vs xz performace comparison" is like comparing which centenarian can limp faster, it might be entertaining for some people, but is generally not relevant. > gzip is fast relative to other compression algorithms gzip looks fast perhaps compared to "xz -9", but not to anything modern.
- abjKT26nO8 7y agoYour comment could benefit from pointing out the alternatives that outperform gzip.
- vlovich123 7y agoCheck out zstd
- willvarfar 7y agoGzip is old but still good. It’s not as good as Zstd nor cases where lz4 and ilk shine, but it’s not fair to claim its slow. Gzip is quite close to the Pareto frontier, meaning it is a good trade off of time and space. See the charts at http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html (And read up on the Hutter Prize!)
- meritt 7y agoI fail to see the use case too. If you want a compact wire format, there are numerous options (protobufs, avro, thrift, etc) and their size is achieved by storing the data in a compact binary format and the schema separately. With MessagePack the schema is embedded alongside the data, exactly like JSON. So it's just binary instead of text encoding, which saves some space, but as you pointed out, standard text compression algorithms are going to likely perform similar or superior.
- blattimwind 7y agoOne particular advantage of msgpack is that it allows embedding binary data without binary-to-text overhead (~30 %). I don't really see msgpack as something that makes much sense to use inside a browser, though.
- hinkley 7y agoFirst piece of software I ever wrote was a base92 encoder. When I looked at b64 is was even worse than the 3:4 I had been told. B64 as originally used (with an extra >2.6% for line splitting) is handily over 35%. As we use it now it’s still a bit over 33%, because of the == endings to detect truncation... 66% of the time.
- nick0garvey 7y agoPerformance. Overhead of JSON encoding followed by gzip will be slower and more CPU expensive than msgpack.
- meritt 7y agoSure, but if performance is the actual issue, then you'd likely pick an even faster and more compact wire format (proto, avro, thrift). I guess the ability to have a schemalass and no type checking format (just like JSON), while enjoying some performance benefits? It just feels like a weird niche to me.
- pas 7y agoIt's just a nice default to pick. (But JSON is "simple" so msgpack did not really catch on. And probably won't for the reasons others mentioned.)
- choward 7y ago> It's just a nice default to pick. Why would it be your default?
- jnwatson 7y agoIt is relatively efficient, schemaless, and cross-language and -platform. It supports floats and binary values as well.
- ahoka 7y agoIt's my go to serialization format for redis payloads. Fast, compact, can represent a lot of types without too much magic.
- pas 7y agoIt's really very-very fast, and no need for external schema registry. And it's really as simple as JSON. Has great language support. https://github.com/thekvs/cpp-serializers#results https://github.com/thekvs/cpp-serializers#results That said, currently I don't work with it in any projects. Apparently it hasn't really caught on.
- jszymborski 7y agoI'm not sure this is a fair apples to apples comparison. Some thoughts: 1. msgpack + gz would be a more fair comparison to json + gz when comparing file size 2. have you run any time comparison? It would seem to me that msgpack gets pretty close to json + gz with far fewer resources, time
- supermatt 7y agoYeah, that is a very fair point. I suppose if i can shave a few bytes per request, why not, and if the parsing is indeed faster, than all the better. I havent had need to deal with such constraints personally, but maybe its something I should consider more. It doesnt need to be a 10x win to still be useful, and if its a simple import to use then i would likely be crazy not to consider it.
- dddbbb 7y agoWhy would you make a comparison by gzipping one and not the other? Running gzip on the msgpack example reduced it from 304 to 248 bytes.
- dwild 7y agoSure, but I think what he meant is that what the website is showing isn't the real gains. Without GZIP: JSON 583 bytes MessagePack 304 bytes 52 % With GZIP: JSON (GZIP) 289 bytes MessagePack (GZIP) 248 bytes 85%
- giantrobot 7y agoSo MessagePack lets you skip the CPU and memory usage needed to compress the JSON yet get similarly sized messages to compression JSON. The compression of the JSON Is a non-trivial extra load on a busy server.
- imtringued 7y agoOr you just use JSON with gzip and never think about this problem ever again. Choosing msgpack vs JSON is a micro optimization at this point. There are significantly better alternatives if you want to save CPU cycles.
- dwild 7y agoMost likely you'll have GZIP enabled on all your HTTP request. Your bandwidth will be a bottleneck way before GZIP CPU load become one. I just took a quick look and they do compress their queries, using Brotli, which is even more efficient than GZIP. They are behind Cloudflare, so it's probably from them. I just tried with https://jsonplaceholder.typicode.com/todos https://jsonplaceholder.typicode.com/todos and I get this: Raw: JSON: 24,311 MessagePack: 14,704 60 % With GZIP: JSON: 3,965 MessagePack: 4,063 102% With Brotli: JSON: 3,495 MessagePack: 3,704 106% So essentially, MessagePack compressed is WORSE than JSON, and not compressed, is at least 3 times worse than compressed.
- 7y ago
- 2StepsOutOfLine 7y agoGzip'ing the msgpack encoded example results in 238 bytes.
- lalaithion 7y agoIt's also likely faster to parse; string and array sizes are statically known, and all parsing happens in order, so there's no backtracking. Want to parse a string in JSON? okay, this is a starting quote, so let's iterate down the rest of the document, looking for unescaped end quotes, and then copy that into a buffer. In msgpack, it's a single memcpy. Additionally, with number heavy messages, then it shines. {"red": 127,"green": 28,"blue": 27,"x": -18,"y": 27,"z": 2008,"time": 1583862185,"dollars": 1298,"cents":12,"name":"tom",timeseries":[1,2,3,4,55,66,77,218,239,340,9009,80008,400004]} gives 106 bytes with msgpack and 157 with gzip. Also, if you gzip the msgpack, for your message example, it's 248 bytes.
- cptwunderlich 7y agoAnd you trust a message from an unknown source? You can't simply memcpy according to some length indicator, that's just not safe. You still have to parse and validate.
- kingofpandora 7y agoGenuine question - is it dangerous to memcpy X bytes that we know must be interpreted as, say, an integer?
- deleted 7y ago[deleted]
- tangent128 7y agoPotentially. Most network protocols are big-endian while x86 is little-endian.
- byte1918 7y agoNo. Everything is 0s and 1s after all. Take for eg. a byte. It has 8 bits and by permutating all the 0s and 1s you end up with all the possible values of a signed byte; all the numbers -128 to 127. So now, if you were to copy a byte from a random memory location that byte will just contain a permutation of 0s and 1s which when interpreted as a signed int, will simply contain a number between -128 and 127.
- quantguy11959 7y agoDo a speed test comparison as well. Serialization to msgpack is likely much quicker than json + gzip. This is for latency sensitive applications.
- deleted 7y ago[deleted]
- ccktlmazeltov 7y agoNobody gzip their json over https, this is due to compression attacks.
- harunurhan 7y agoNot every use case is affected by BREACH attack.
- ccktlmazeltov 7y agoJSON is mostly used on the web, and with HTTPS, where compression is disabled.
- willvarfar 7y agoA decade ago I benchmarked and used msgpack in a Python system instead of json because it was so much faster that it halved my hardware requirements. I think thrift and protobuf and stuff were around then but I don’t recall how they compared. Edit: found my post https://stackoverflow.com/questions/9884080/fastest-packing-of-data-in-python-and-java https://stackoverflow.com/questions/9884080/fastest-packing-...
- nitrogen 7y agoI made a similar choice for a Ruby project, with msgpack over RabbitMQ. I no longer have any connection to this repo, but the benchmarks (and a schema validator gem I wrote) are here: https://github.com/deseretbook/classy_hash/blob/master/README.md#why-classy-hash https://github.com/deseretbook/classy_hash/blob/master/READM...