16 ms·
MessagePack: like JSON, but fast and small
- wittedhaddock 7y agoI implemented this in Swift w/o using the Foundation library a few years back if anyone wants it: https://github.com/wittedhaddock/bytepress https://github.com/wittedhaddock/bytepress
- kortilla 7y agoBut is is readable as plaintext? That’s one of the main appeals of json.
- mrlala 7y agoSure doesn't look like it.. I agree- that's why json is honestly great. I avoided it for the longest time, but now I totally see the appeal.
- yohannparis 7y agoMaybe where you want to have a human debug or see what is happening (like in localStorage, or cookies). But if you're going to transfer a significant amount of data between front and back-end, for example, that can save money on a cloud service. You can easily make a function wrapper that uses JSON in a dev environment and MessagePack in Production, for example.
- enitihas 7y agoIf you want to transfer a significant amount of data and don' t need readability, using protobufs makes even more sense, since the key names do not go as string on the wire, so the data should be much smaller.
- ludocode 7y agoMessagePack lets you use integers for keys, so you can use enums or integer constants in the code instead of strings. There are good reasons to use MessagePack over Protobuf even if you don't need readability, such as easier integration into buildsystems and better support for embedded platforms.
- Cymen 7y agoI used to care about that but once the JSON gets at all complex, it's nice to switch to pretty formatting of it. At that point, a dev tool in the browser could do the same for this format so I don't think that is as pressing (for me and I suspect most people).
- mlavin 7y agoIt's taking a different angle from JSON, encoding it in JavaScript is done from standard objects into a UInt8Array.
- dathinab 7y agoActually , I think thats no longer the case. (Except if you want to use JSON for configs, which a strongly recommend against for a bunch of reasons including missing support for comments). I mean by now nearly all times you send thinks over the wire they are ecrypted or at least compressed. So inspecting on-wire messages without "proper" more complex tooling doesn't really work. But if you have already more complex tooling involved there is no reason why you need to be able to read the raw message, it could be just converted on-the-fly into a readable format. Same applies for application development when e.g. logging, you always do some formatting/conversion when logging some data, so not a problem to convert msg-pack to a human readable format. But then today logging of e.g. servers should always go to some form of log server which does add features like search ability by indexing the log and similar. So no problem to have a msg-pack viewer there, too. Honestly I believe the only reason human readable formats made/(still do make sometimes) sense was due to limited tooling sometimes caused by server limits in computation power on the developers system. And the fact that many binary formats are totally over engineered making handling of them painful. Especially debugging of slightly corrupted data. Which isn't the case for MsgPack.
- gnulinux 7y agoCompression is not encryption and security by obscurity is bad practice. So it doesn't matter if the data is json or compressed or protobuf or messagePack or custom struct. As long as it's not properly encrypted, it's just plaintext.
- ludocode 7y agoI don't think the parent poster was confusing encryption for compression or suggesting security by obscurity. He just meant that JSON is never sent in pretty-printed plain text over the wire, so you always need tooling to view it. This has been my experience with JSON as well. Almost all JSON in the wild especially in web services and RPC is at least minified, so you need to pass it through something like `python -m json.tool` to reformat it for viewing. So you might as well use MessagePack and pass it through `msgpack2json -d` to view it instead. It makes no difference whether the underlying format is human readable.
- enitihas 7y agoIs there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Msgpack, unlike json, is not human readable on the wire, not a purely text based format, and I doubt it is smaller or faster than protobufs or flatbuffers.
- mgamache 7y agoit's about as fast as protobuf using .net. It is not human readible.
- throw0101a 7y ago> Is there any advantage of msgpack over json or gzipped json on one side and soemthing like protobuf or flatbuffers on the other? Or CBOR, that has an RFC (which is important for some folks): * https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
- eshyong 7y agoPlus the fact that protobuf is more flexible about interface changes. My previous company had countless incidents that happened when new code deserialized msgpack records with unexpected fields from redis...
- alexdeloy 7y agoI was asking myself the same thing and apparently protobuf has no concept of "null", thus I can see how MessagePack might have an advantage depending on your use case here. https://github.com/neuecc/MessagePack-CSharp#comparison-with-protobuf-json-zeroformatter https://github.com/neuecc/MessagePack-CSharp#comparison-with...
- fizx 7y agoprotobuf can do nullable values, but you have to ask for them as primitives are the default. e.g. StringValue (nullable) vs string (not)
- neuland 7y agoProtobuf requires schemas, which is good practice anyways but maybe you don't have or want to do for some reason. FlatBuffers doesn't have as many client libraries. There's a MessagePack library for a ton of languages. One thing I really like about MessagePack is that the Python client (and others too) supports reading from a stream. So you can write a bunch of msgpack messages to a file or TCP socket and it just works. Protobuf can't do this out of the box because it doesn't include how long the message is. You can write a wrapper that specifies the message length, which isn't that hard and I've done before, but it is another thing to maintain. And other formats do that out of the box (ex. Cap'n Proto) And as someone else mentioned, Protobuf doesn't have NULL which is useful in some cases. (I understand that Go has strong opinions about there being a useful default value, but that doesn't map well to a lot of languages) One other thing with Protobuf is that the Python client is not very pythonic. I've been keeping my eye on this project [0] which makes Protobuf messages work just like dataclasses. They don't support OneOf types currently, which I happen to need for some of my use cases. But they're working on it [1] [0] https://github.com/eigenein/protobuf https://github.com/eigenein/protobuf [1] https://github.com/eigenein/protobuf/issues/85 https://github.com/eigenein/protobuf/issues/85
- mgamache 7y agoI used MessagePack in a real-time streaming app. It was smaller than Protobuf (and about as fast in .net). In fact, if you are a .net dev and use the popular SignalR it can use MessagePack for real-time browser binary messaging. I highly recommend it. But, it is not a direct replacement for JSON as it's not a human readable or accessible without libraries.
- GordonS 7y agoI've also used MessagePack on .NET, but as a serialisation format for use with a message queue - I don't recall the numbers off hand, but it's something like an order of magnitude to serialise and deserialise than JSON, and resulted in a lot less allocations too. It supports compression too. Even if the resulting serialised size isn't always smaller than compressed JSON, from a performance standpoint it's a lot better.
- pkolaczk 7y agoIt might be better than text formats, but it still encodes most things as variable-size and this is not particularly machine-friendly. A format with separate metadata and fixed length encodings for most of the things (eg numbers) would be much more efficient to serialize / deserialize.
- qes 7y ago> resulted in a lot less allocations too Especially if you use the MessagePack-CSharp library. That dude knows his high-perf .Net.
- supermatt 7y agoI took the first example from http://www.json.org/example.html http://www.json.org/example.html and msgpack makes it 304 bytes. a simple gzip on the json is 289 bytes. The larger examples are even more in gzips "favour" I am not 100% sure why I would choose to use this - maybe for super tiny documents?
- pletnes 7y agoMsgpack is a different encoding, not quite the same as compression. What happens if you do both?
- blattimwind 7y agogzip is very slow. I don't know why anyone would still use it today.
- jgalt212 7y agoreally? what do you suggest that is faster? Our shop has tested a number of compression formats (xv, bz2, gzip, etc) and gzip is good enough and faster than the others we tested.
- thedance 7y agoWell yah, those are all ratio-tuned codecs that are very slow. LZMA beats bzip2 on both ratio and speed so you might as well forget about bzip2 forever. Zstd, snappy, LZ4, or even brotli are probably better choices than zlib for most people. Brotli has LZMA-like ratios at dramatically higher speeds.
- blattimwind 7y agoThe algorithms you name are all rather outdated. Typical gzip decompression speed is somewhere in the 200-250 MB/s region, compression is much slower. LZ4 for example tends to compress at ~600-700 MB/s, and decompress at several GB/s. zstd is tweakable over a very wide range of ratio-speed trade-offs. LZMA(2) (xz) is a rather troubled format and should not be used any more. bzip2 has always been slower than gzip with usually marginally better compression. It has been irrelevant for a long time.
- gravypod 7y agoIs there any comparison of the tooling/performance/etc of MessagePack, BSON, Protobuffers, Flatbuffers, and plain JSON?
- tracker1 7y agoLots.. however, best bet is to search for your language, platform and use-case. The performance can vary widely by platform and language. JSON is actually faster in some platforms or libraries, binary options faster in others. Connection interface and overhead are also an issue.
- GordonS 7y agoIt's for a specific library of course, but there are some numbers here: https://github.com/neuecc/MessagePack-CSharp https://github.com/neuecc/MessagePack-CSharp
- brianpgordon 7y agoAlso Avro and Thrift, please!
- ludocode 7y agoI did some pretty extensive benchmarking of various schemaless serialization libraries a few years back. All of these libraries have advanced quite a bit since then so it's a bit out of date, but the relative speeds of MessagePack vs JSON and BSON are probably still relevant: https://github.com/ludocode/schemaless-benchmarks https://github.com/ludocode/schemaless-benchmarks I haven't compared them to schema formats like Protobuf or FlatBuffers yet because the use cases are pretty different. I like MessagePack for small projects or rapid prototyping because you don't need to integrate any big libraries or set up code generation as part of your buildsystem. (Mostly I got sick of integrating the C++ Protobuf library into embedded projects.) The MessagePack format is a lot simpler than Protobuf and the best implementations are nowhere near as allocation-prone as the reference implementation so I expect they would beat it flat out on performance, though the messages may be slightly larger. They would probably beat FlatBuffers for encoding speed as well, but I don't expect any schemaless format could beat FlatBuffers for decoding speed.
- 7y ago
- etaioinshrdlu 7y agoIt also supports binary data in strings, unlike JSON. This means no more slow base64 encoding/decoding.
- dguaraglia 7y agoNot sure why you are being down voted, sending binary data is a well known issue with JSON, and base64 can be pretty costly if using a naive implementation. In a previous life I inherited a service that shipped tons of data (billions of requests a week) as base64 encoded protobuf strings over HTTP. It was a bad solution in so many ways, but there were historical reasons why it had gotten there. The system required a number of servers and I decided to do some profiling to see if there were some quick gains that could be made. As it turns out, about 60% of CPU time was spent decoding base64 using Python's standard library. I was shocked.
- Cymen 7y agoThe "Try!" demo fails with large JSON submissions with an exception in jQuery: Uncaught RangeError: Maximum call stack size exceeded I'm curious to see the savings difference and hoped to with "Try!" but it'll have to wait.
- otoburb 7y agoObligatory Cap'n Proto[1] reference whenever serialization format discussions crop up. [1] https://capnproto.org/index.html https://capnproto.org/index.html
- mikepurvis 7y agoCapnproto is schema based rather than self-describing. So it's related, and it does compete with formats like json and msgpack, but there isn't total overlap in the applications.
- twic 7y agoI would be nice to have a taxonomy, or at least a bestiary, of serialisation formats. Tentatively: Schema-driven no-compromise fast compact binary formats with no cross-version compatibility: Cap'n Proto, FlatBuffers, SBE, ASN.1 PER, XDR, OMG CDR. Schema-driven binary formats which allow some cross-version compatibility: Protocol Buffers, Thrift. Self-describing binary formats: MessagePack, CBOR, BJSON, Bencode, ASN.1 BER, Avro (?), Fast Infoset, AMF3. Self-describing textual formats: JSON, XML, YAML, TOML. I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it.
- kentonv 7y ago> no cross-version compatibility: Cap'n Proto, FlatBuffers This is incorrect: Cap'n Proto absolutely allows cross-version compatibility, using roughly the same semantics as Protobuf. I believe FlatBuffers does too. (I'm unsure about the rest, haven't studied them in a while.) > I'm using "self-describing" here to mean simply that you can recover the structure of the encoded data without a separate schema, rather than that you can attach any semantic meaning to it. Protobuf, Cap'n Proto, and probably several of the other binary formats can parse data into a message tree without the help of a schema, but all the fields will be labeled numerically. MessagePack is only considered "self-describing" in comparison because in encodes human-readable field names on the wire.
- Felk 7y agoI found myself using msgpack as a drop-in alternative for acceptable mimetypes for HTTP responses in a flask app. Browsers would get the response data with the pretty-printed json embedded (or even a custom template with the data fitted in, if there was one for that specific endpoint), api clients asking for nothing in particular would get pure json, and clients asking for msgpack would get that. Seemed like a free way to offer slightly better performance on the API (both serialize/parse times and bandwidth), since it can just serialize any data without specifying protocols or schemas, like json serializers can too. I didn't know about any alternatives that would also require no further configuration or infrastructure, so it seems like msgpack fills this 'free performance boost' niche quite nicely
- CiPHPerCoder 7y agoUnder PHP... > Msgpack is an PECL extension, thus you can simply install it by: Having something available in PECL is a good first step, but nobody will use it unless you either: 1. Get it into the standard library (which requires an RFC for PHP Internals), OR 2. Write a pure-PHP polyfill installable from Composer, OR 3. Do #2 then #1 (using the polyfill's popularity to argue for the importance of the RFC acceptance to make #1 a reality). Reason: A lot of the places PHP is deployed, you can't compile C code or install binary dependencies (.so, .dll files). You can't access the OS package manager, either. But Composer is a pure-PHP package manager that still operates in these environments. So if anyone on HN ever wants your thing to be used by PHP developers, don't just stop at "PHP extension, written in C, available in PECL".
- snuxoll 7y ago> Reason: A lot of the places PHP is deployed, you can't compile C code or install binary dependencies (.so, .dll files). You can't access the OS package manager, either. If you’re in the target audience for msgpack you probably aren’t relying on shared hosting and can build a pecl extension.
- somehnguy 7y agoThats some strange logic. I have multiple racks of company owned bare metal that I deploy to. Still don't want to build a pecl extension. Much simpler and way less likely to not run into random build issues if I can just install via composer.
- snuxoll 7y ago> Still don't want to build a pecl extension. Docker? Build it as a system (RPM/DEB) package? Man, my life would be difficult if I just flat out refused to use packages for PHP/Python/Ruby requiring native extensions because it required some minimal effort on my part to deploy it.
- PetahNZ 7y agoIf you literally read further down the page there is a pure PHP implementation, the rybakit/msgpack composer module.
- tracker1 7y agoDepends on what you're doing... for real-time data streaming, sure... for request-response, I'm less convinced. JSON being human readable, relatively lightweight, and highly compressible is pretty convincing. I'm not against messagepack or protobuf, however, much like all things, I'd rather start with simple http+gz+json (maybe websockets) and optimize as needed. Not everyone is at the scale of FAANG, and most don't really need this level of optimization.
- aledalgrande 7y agoAgree. Most systems have to be optimized in other areas first and I wouldn't give up the debuggability of JSON for a fraction of percent of increase in performance.
- IshKebab 7y agoProtobuf is entirely different to MessagePack and JSON because it has a schema.
- tracker1 7y agoschema isn't part of the actual encoding though afaik.
- SlowRobotAhead 7y ago>[JSON] relatively lightweight Um... no? Epoch as binary, 4 bytes. As JSON, 10 bytes. 64bit big num in binary, 8 bytes, in JSON 19 bytes. For a small example object I can think of { “x” : “y” } In binary, 5 bytes. In JSON, 9 bytes. Now... light ENOUGH because you’re using a PC and gigs of ram and a 100mbit internet, sure. Light in terms of a microcontroller? No. I flat out could not use JSON in any of my projects and I’m not at FAANG level.
- tracker1 7y agoseems like what you're talking about would fall under the real-time streaming category... HTTP overhead alone would outweigh the JSON differences you're talking about.
- rglover 7y agoI tried to work with MessagePack last year while teaching someone who was building it into their product. Absolute nightmare.
- sonicxxg 7y agoMissing links to benchmarks. Also, Json is not really a good baseline for comparison. If you support binary data natively in the format, you should compare it to Bson.
- ants_a 7y agoBson is just terrible. Just as a taste, arrays are encoded as key-value, where key is the array index converted to a decimal string and then stored as zero terminated string with 4 byte length prefixed to it. It boggles the mind why somebody would design a format like this.
- ChuckMcM 7y agoI like it! Sort of the unholy love child of XDR and JSON :-). Now we need an IDL that will let you define a structure and have it produce <language> marshalling and unmarshalling routines.
- Dansvidania 7y agoI thought the appeal of JSON was that it is, among other things, somewhat human readable. The gain in bytes does not seem to justify the loss of human readability. Am I missing something?
- GordonS 7y agoIt depends on your use case; having human readable serialised messages isn't always a big deal. For example, I recently used it for a high-throughput messaging system, where the vastly improved performance was a huge selling point.
- Dansvidania 7y agoAre you referring to the reduced payloads or does this also provide a faster way to serialize/de-serialize?
- GordonS 7y agoMuch faster serialisation/deserialisation (and a lot less allocations on .NET). It does support compression, but messages sizes are generally comparable to compressed JSON (YMMV, it will depend on your messages).
- dang 7y agoRelated from 2012: https://news.ycombinator.com/item?id=4092969 https://news.ycombinator.com/item?id=4092969 2011: https://news.ycombinator.com/item?id=2886106 https://news.ycombinator.com/item?id=2886106 https://news.ycombinator.com/item?id=2571729 https://news.ycombinator.com/item?id=2571729 2010: https://news.ycombinator.com/item?id=1453332 https://news.ycombinator.com/item?id=1453332 https://news.ycombinator.com/item?id=1203444 https://news.ycombinator.com/item?id=1203444 Others?
- andymoe 7y agoYeah, this same conversation about Message Pack happens once every two years here but that’s ok I guess...
- dang 7y agoHave there been threads in the past 8 years?
- andymoe 7y agoI guess you’re right that these are the top level threads before 2012. What I’m remembering (and found via google search) is this kind of discussion around message pack comes up in most other threads about serialization formats or adjacent topics.
- dang 7y agoThere definitely have been some comments: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%22message%20pack%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- guessmyname 7y agoHa! Yes, I have been using MessagePack for several years. It never occurred to me that people would be interested to have a discussion about it on Hacker News. MessagePack to me is as ubiquitous as JSON and Protobuf, I submitted www.json.org [1] just to prove my point. I figure OP is today’s lucky 10,000 [2] and that’s why they decided to submit the link, thinking it was a new type of data interchange format. [1] https://news.ycombinator.com/item?id=22538794 https://news.ycombinator.com/item?id=22538794 [2] https://xkcd.com/1053/ https://xkcd.com/1053/
- dilann 7y agoOh, it's turd polishing anyway. Numbers in JSON are a pain.
- pjscott 7y agoThe data types that msgpack supports aren't quite the same as JSON -- msgpack has encodings for signed and unsigned 8, 16, 32, and 64-bit integers, as well as single- and double-precision floats. (This runs into trouble if you're dealing with JavaScript, which doesn't support the full 64-bit range of integers, but everybody else should be okay.)
- dathinab 7y ago> MessagePack: It's like JSON. but fast and small and complete. Both are minimal self describing data serialization formats, but JSON is incomplete. It misses a type to represent one of the fundamental type: byte blobs. And also parts of floats. Which means there are a lot inconsistent ways to hack that in e.g. base64 encoded strings. But this means it's loosing partially it's property of being self-describing (same if you allow numbers as strings e.g. "-12"). Just to be clear I'm not saying JSON should have bytes directly in it. But a "native" base64 string additional to string, number, null, list and map would help. E.g. `{ "bytes": b"YWJjZGU=" }` Wrt. floats it's missing (+/-) Infinity, NaN is more like a error variant so it's okish to be missing, but then why not have it. Also for completeness it would be better to differentiate between int and float as float is imprecise due to rounding errors. ---- PS: I don't like `null` but having it in this kind of data serialization format is still required. But the difference between a field not being there and it being `null` can be a mess and not all tools handle it well. PPS: Both JSON and MsgPack can be used for non self describing serialization, e.g. by serializing the a record (fixed number of fields in known order) as a list of values instead of a mapping. But both are focused on enabling self describing serialization. EDIT: PPPS: Yes, it's a very weak form of self-describing is use here, there are systems which are much more self describing e.g. XML+XML Schema linked from the XML document but also tend to be far more complex
- sdegutis 7y agoI've become particularly fond of Ion[1]. The main benefit JSON has over Ion is its ubiquity, even in many standard libraries. [1] https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/
- dathinab 7y agoThis looks quite nice. But then S-Expressions and the soon to be there templates add quite a bit of unnecessary complexity IMHO.
- sdegutis 7y agoI doubt it's any more complicated to add S-Expressions than to change list parsing to take either [] or () and set a type flag based on which one it finds. Semantically, it's not more complex either, just giving one extra piece of user-defined metadata to the actual data that's still basically a list.
- kstenerud 7y agoI'll throw my hat into the ring as well. I've been building a new ad-hoc data format to replace JSON for a couple of years [1], and am nearing completion of the reference implementation in go. It natively supports the following types: * Nil : No data (NULL) * Boolean : True or false * Integer : Positive or negative, arbitrary size * Float : Binary or decimal floating point, arbitrary size * Time : Date, time, or timestamp, arbitrary size * URI : RFC-3986 URI * String : UTF-8 string, arbitrary length * Bytes : Array of octets, arbitrary length * List : List of objects * Map : Mapping keyable objects to other objects * Markup : Presentation data, similar to XML * Reference : Points to previously defined objects or other documents * Metadata : Data about data * Comment : Arbitrary comments about anything, nesting supported But the most important feature is that it is a paired format: a binary format [2] and a text format [3], which are 1:1 compatible. This allows you to transmit in the binary format, and only convert to text when a human is involved. I've put together a quick comparison here: https://github.com/kstenerud/concise-encoding#comparison-to-other-formats https://github.com/kstenerud/concise-encoding#comparison-to-... Currently, I'm finishing off the go implementation [4], which so far I've managed to get running 30% faster than the json codec, using less than half the memory. I'll be pushing the binary codec to master soon, and the text codec shouldn't take much longer since the code is pretty modular. [1] https://github.com/kstenerud/concise-encoding#concise-encoding https://github.com/kstenerud/concise-encoding#concise-encodi... [2] https://github.com/kstenerud/concise-encoding/blob/master/cbe-specification.md#concise-binary-encoding https://github.com/kstenerud/concise-encoding/blob/master/cb... [3] https://github.com/kstenerud/concise-encoding/blob/master/cte-specification.md#concise-text-encoding https://github.com/kstenerud/concise-encoding/blob/master/ct... [4] https://github.com/kstenerud/go-cbe/tree/new-implementation https://github.com/kstenerud/go-cbe/tree/new-implementation
- brokensegue 7y agoWhy add special support for URIs and not any other types that can be easily represented as strings e.g. UUID or ISBN
- kstenerud 7y agoIt's mostly a question of how common they are. Both uuid and ISBN can be represented as URI. However, I'm still on the fence regarding uuids, because they tend to get a lot of use in general... I may add it after all. The idea is to make a format for the 80% case, so things like ISBN are definitely out.
- hesdeadjim 7y agoThese formats are great for private communication, but become a nice attack surface area if publicly exposed. I was using thrift on an old project and had to add a sanity check layer to make sure an attacker couldn’t just specify a valid request with a list set to have int.max elements.
- dathinab 7y agoBut any parser needs valid checks of this kind, including the JSON one. A common way to mess with JSON parsers is e.g. to nest a lot arrays and objects. So you needs max nesting depth. There are a bunch of other ways to mess with JSON parsers, too. (In case of MsgPack lists and maps: A parser should only per-allocate memory for given size if it has done proper sanity checks, e.g. compares the given length with the remaining byte length of the message. Also you can simply not preallocate and instead, like in JSON parsers, grow your list on demand and just use the length to know when the list ends). But yes you have to make sure the parser works for your use case.
- PaulHoule 7y agoYou can't beat JSON by very much with a format that has the same free-form structure. (Particularly when you use gzip, zstd, ...) Parsing code has to be branchy to handle many different possible structures. If you want extreme performance, variable-length strings are a problem -- the old mainframes that had fixed-length "HOLLERITH" strings had a good idea. I like just about everything about Apache Arrow except that it ignores the problem of fast/portable string handling.
- GordonS 7y agoWell, you can't always beat it much from a size perspective, but MessagePack can be an order of magnitude faster to serialise/deserialise than JSON.
- pjscott 7y agoThe serialization format here was designed so you can parse it (or detect a parse error and bail out) in a single O(n) pass through the message data. No backtracking required. (You mentioned strings, so I'll use that to give flavor: a string is represented as a one-byte prefix, followed by a byte length, followed by that many bytes of UTF-8. I'm not sure whether you'd categorize this as fixed or variable length, but that's how it's represented on the wire.)
- nabla9 7y agoSize and speed comparison between JSON.zst and MessagePack please.
- kentonv 7y agoWhy only compress the JSON? MessagePack will compress about as well as JSON does.
- nabla9 7y agoThe sales pitch for MessagePack is: like JSON, but fast and small.
- kentonv 7y agoCorrect: Uncompressed MessagePack is faster and smaller than uncompressed JSON, and compressed MessagePack is faster and smaller than compressed JSON. I still don't see why you'd compare uncompressed MesasgePack to compressed JSON.
- nabla9 7y agoBecause I suspect that the MesasgePack–compressed or not–is not the worth of the effort until I see comparison. In other words, compressed MesasgePack is probably only tiny amount smaller than compressed JSON.
- kentonv 7y agoSure, a comparison between compressed JSON vs. compressed MessagePack is interesting. I interpreted your original message as requesting a comparison between compressed JSON vs. uncompressed MessagePack, which didn't make sense (but which I see people ask a lot, including elsewhere in this thread). Sorry if I misunderstood.
- nabla9 7y ago
- scandum 7y agoBit of a shameless plug, but yet another alternative is VTON, though it is typeless. https://github.com/scandum/vton https://github.com/scandum/vton
- banana_giraffe 7y agoI've skimmed over it and didn't see if there's a compelling reason to use this over CBOR, or vice versa. Anyone have any insight?
- marton78 7y agoThey're almost the same. To see how similar they are, compare their implementation in nlohmann's JSON library for C++. They are both processed by the same class template, only some constants are different.
- brianolson 7y agoCBOR went to the trouble of being an IETF standard, RFC 7049. So, you know, the lovely thing about using standards!
- ludocode 7y agoCBOR also made a lot of changes to MessagePack making it far more complicated, both to use and to implement. I've talked about this on HN before so I'm repeating myself a bit but here's a short list: - CBOR has two ways of encoding maps and arrays: fixed length and variable-length. This complicates decoders, especially those that would pre-allocate arrays and maps to the proper sizes, which significantly reduces decoding performance. The CBOR spec has nothing useful to say about this; it just requires you to allocate indefinitely. - CBOR defines a canonical representation, including a key sorting order based on binary representation which is just awful. It requires multi-pass encoding which is slow, complex, error prone, and completely non-intuitive: [1,2,3] comes before 100000 which comes before [1,2,3,4]. - CBOR has more types in the core spec, ones that are extremely specific to certain applications or programming languages. It has a 16-bit float, and it has both null and undefined as separate types. - CBOR defined a system of "tags" with a huge number of extension types. These are supposed to be optional, but of course they only work if both ends support them. Some features like BigNum are well-supported in some programming languages but not others, so CBOR implementations tend to diverge in supported message types. CBOR as a standard is far worse than the "non-standard" MessagePack it purports to replace. Here's a great HN comment on it from another user (and another MessagePack library implementer) a few years back: https://news.ycombinator.com/item?id=14072598 https://news.ycombinator.com/item?id=14072598
- erling 7y agoApologies if I missed it, but what are the advantages of this over bEncode?
- squaresmile 7y agoThis reminds me of a quite popular mobile game that sends master game data using messagepack which is then gzipped and then encrypted and then base64 encoded in a json response. Fun.
- twic 7y agoI once used an API where to send a command, you would make a POST where the message body was an x-www-form-urlencoded dictionary where one of the values is a binary string which is a zipped XML document. It was very clearly just a page on the vendor's site where you could manually upload a zip of documents, which they had simply declared to be an API.
- dwild 7y agoThat remind me of one of my first contracts, 10 years ago, for an Android mobile app. It was an app made to quickly record something using video, audio or picture, that would then be uploaded through their API. The API worked essentially like you said, it was converting the binary file to base64, adding it to a JSON, which was in a GET variable (thus urlencoded) and sending that over an HTTP connection (I don't remember if it was HTTPS or not though, I hope it was, but at the time I could have ignored that part). Android didn't allowed more than 16 MB of memory for an app, thus I had to build streams to support each things individually, that was an interesting challenge. I was amazed to find out that there was officially supported Base64 streams, but their API strangely didn't accept the default Base64, it was only accepting the one for URL (which replace a few characters with others one), so I had to add another stream on top to do this.
- brianolson 7y agoOr you could use IETF standard CBOR (Concise Binary Object Representation) https://cbor.io/ https://cbor.io/ RFC 7049
- pan69 7y agoReminds me of Adobe AMF (Action Message Format) that was used in their Flash Player: https://www.adobe.com/content/dam/acom/en/devnet/pdf/amf-file-format-spec.pdf https://www.adobe.com/content/dam/acom/en/devnet/pdf/amf-fil...
- PopeDotNinja 7y agoI've worked extensively with MessagePack and JSON. I much prefer JSON because it's human readable. It's just a pain to debug something that looks like raw binary data, and I usually have to debug it at times I'm just not in the mood to deal with shenanigans.
- magicalhippo 7y agoI was looking at MessagePack for communicating to and from my STM32F1-based microcontroller project from the PC controller software I'd be writing. At least the official C library was not optimized for memory usage and code size. I also considered BSON, but it also lacked suitable libraries. So I ended up using JSON. Yes the message sizes are larger in byte size with JSON but using the jsmn[1] parser I could avoid dynamic memory usage and code size was small. The jsmn parser outputs an array of tokens that point to the buffer holding the message (ie start and end of key name etc), so overhead is quite limited. For JSON output I modified json-maker[2]. It already allowed for static memory usage and rather small code size, but I changed it to support a write-callback so I could send output directly over the data link, so I didn't have to buffer the whole message. This is nice when sending larger arrays of data for example. Combined it took about 10kB of program (flash) memory, of which float to string support is about 50%. Memory usage is determined by how larger incoming messages I'd need, for now 1kB is plenty. A nice advantage of using JSON is that it's very easy to debug over UART. Though having compact messages would be nice for wireless stuff and similar, so does anyone know of a MessagePack C/C++ library that is microcontroller friendly? [1]: https://github.com/zserge/jsmn https://github.com/zserge/jsmn [2]: https://github.com/rafagafe/json-maker https://github.com/rafagafe/json-maker
- TickleSteve 7y agoNot messagepack, but if protobuf is ok, then nanopb has given me good results on uC projects.
- magicalhippo 7y agoAh, that looks pretty spiffy, thanks!
- SAI_Peregrinus 7y agoI'll second nanopb as pretty good. Used it on an STM32F4.
- SlowRobotAhead 7y ago
- rsynnott 7y agoSo not at all like JSON, then?
- rudolph9 7y agoJSON is as pervasive these days as XML and despite it's shortcomings is not going away for the foreseeable future.
- tsegratis 7y agoProbably we need a human readable text view for messagepack That would let it fully take on the role of json But what's the text format? -- now there is endless happy bikeshedding Maybe a library with json (or compatible superset, to get all messagepack features). Then the standard just serializes to messagepack, before gzip or whatever
- ludocode 7y agoI wrote a MessagePack to JSON conversion tool which is ideal for viewing MessagePack. It supports a pseudo-JSON debug output so you can view messages even when they use features outside of JSON, like binary blobs or arbitrary key types: https://github.com/ludocode/msgpack-tools https://github.com/ludocode/msgpack-tools It works best if you use MessagePack like JSON where all map (object) keys are strings, so you can easily understand a message without context. If you want to optimize your MessagePack more, you would tend to use integers for map keys, but this makes the JSON-equivalent view not super clear because you just see a bunch of numbers in a tree structure.
- elcritch 7y agoThanks! Your `msgpack-tools` are really handy. When I use MessagePack usually I'll add it as an option to an endpoint (`/api/request/123?type=mpack`) in addition to json but there's still cases where having a mpack-to-json tool comes in handy.
- atombender 7y agoAnyone know of an efficient binary format similar to MessagePack that supports deserializing only whitelisted keys, and not paying the penalty of parsing data that isn't needed? MessagePack is great, but libraries generally only support deserializing the whole thing. I have an application where these structured documents can be very large, and scanning code sometimes only needs a very small subset of keys. I believe Cap'n Proto has this feature, but unlike MessagePack it's not schemaless. For example, given a struct like this: { "id": "123", "name": "Developers", "members": [{ "id": "567", "permissions": [ {"type": "read"}, {"type": "write"} } }] } Let's say I only want the name, the ID of each member, and whether permissions.read is set. I may want to do something like (Go): StreamingUnmarshal(b, func(keypath string, parse func() interface{}) { switch keypath { case "id", "members.id": value := parse() // ... use value ... case "members.permissions.type": if parse().(string) == "read" { // ... } } }) Random access could be also work, as long as it didn't need to sequentially parse from the beginning of the data to get to the right value each time. Something like: id := GetKey(b, "id") memberIDs := GetKey(b, "members.id") permissionTypes := GetKey(b, "roles.permissions.type") The trickiest bit is treating arrays of nested structs (as in "roles.permissions.read") correctly, although efficient scanning of keys becomes an important optimization point, too. Probably the best method would be to store the keys pre-sorted at the beginning of the data, so that they'd better fit in the CPU cache, and have pointers to the offsets of the values: KEY1,KEY2,KEY3,VALUE1,VALUE2,VALUE3 Arrays of structs are tricky here, again, but this is solvable.
- kentonv 7y agoHave you considered using a sqlite file? No, seriously: It doesn't require an external schema, supports every kind of indexing and fast random access you could want, is supported in like every language, OS, and architecture, has copious tooling, documentation, and community support, and is battle-tested across literally billions of installations worldwide.
- atombender 7y agoNo, that would not make any sense. SQLite cannot deal with structured, hierarchical data like the stuff I described in my comment. Also, I am talking about individual documents that already live in a database such as PostgreSQL. I can't store an entire SQLite database in a single column.
- paulrpotts 7y agoI implemented a streaming deserializer for MessagePack data in C for small microcontrollers. It is quite small and not complete. Then I tested it with test data streams generated by the reference C++ implementation and Python implementation. It's actually kind of a pain, because the C++ serializer generates a variety of different data types depending on the values you are packing, not the type of the values you are packing. Let's say I encode a uint32_t field. The stream might get a uint8, a uint16, a uint32, or even an int32 (for reasons that completely elude me). Also, C++ strings come out as the 'ext' type while Python strings come out as the 'string' type, so I have to accommodate both, even though they are both basically byte strings. So, I want to tell the deserializer what kind of output field I'm expecting, for each struct member or array, and then look in the data stream to see if there is a data object there that came from the same type. But this is impossible, so the per-type decode functions have to be quite complicated to handle a variety of types. So - it works but I can't do much on the deserialization side to verify that the data I'm unpacking really matches what was encoded. I can only detect very broken cases, like when I'm unpacking a uint32_t and in the stream there is an int32_t with a negative value. I guess this is mostly done for optimization, but two things would make it a lot better: - if the spec actually specified how data types in different languages were allowed to be encoded - if the encoded data contained _two_ type fields, one indicating the original source data type and another indicating the type it was encoded into. Basically the spec is just way too "loose" to make it usable for the use case I'm trying to use it for, which is to easily generate data that is sent to a micro and stored in EEPROM, then deserialized out of EEPROM later. That's probably not very close to a use case the original designer had in mind. But I haven't found anything that works better (less decode logic).
- ludocode 7y agoThe spec doesn't really make this clear, but reifying the packed encoding types is not, I think, how MessagePack was intended to be used. At least it's not how it's implemented in MessagePack libraries. As far as I know they pretty much all use the most efficient representation for all values, so the original data type is always lost. You generally shouldn't worry about the low-level type that a value was encoded into in the MessagePack stream. It's dynamically typed, so you should just care about values. When encoding you should allow the encoder to use the most efficient representation, and when decoding you should be able to tell your MessagePack parser the integer width you want instead of caring about the original type or how it was encoded. It should then accept any packed integer type as long as the value is in range. This is how my MessagePack implementation works as well. If you expect to receive an integer that fits in, say, `uint16_t`, you can call `mpack_expect_u16()` or `mpack_node_u16()`, and it will allow any integer representation as long as the value is in range. It sounds like this is where you were going with your implementation as well, so this may not be comforting because it's not what you want, but it is at least the correct way to understand the format. I've talked about this pretty extensively and wrote up a protocol clarifications document that explains a bit more about how and why MessagePack libraries discard integer width and signedness: https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/issues/35 https://github.com/ludocode/mpack/blob/develop/docs/protocol.md https://github.com/ludocode/mpack/blob/develop/docs/protocol... If you really want things like original integer width represented in the format, ultimately you're going to want to use a different format, probably one that is non-dynamic and uses schemas. As far as the string vs ext, you may have meant string vs bin; there was a format change a while back that separated string and bin types and not all MessagePack libraries have adapted to that. Many libraries (including mine) support a compatibility mode so they will use only compatible string representations.
- tzs 7y agoOK, I give up...how is that two column list of languages organized? Why is the Perl implementation [1] not listed? [1] https://metacpan.org/pod/Data::MessagePack https://metacpan.org/pod/Data::MessagePack
- gbrits 7y agoSlightly related: recently wrote a binary encoder/decoder that used bitpacking, delta-encoding, and other well-known 'tricks' to efficiently pack batches of 10k-100k events of the same uniform type (essentially encoding column by column and using similarities to my advantage). Nothing too fancy, but it was A) a huge success in terms of compresison-ratio and B) a hassle to write. Do any, more or less, turn-key solutions exist for this? Specifically targeting Node but a commandline util might work as well.
- jiggawatts 7y agoI had a play with something similar and I noticed that -- at least for my use case -- you can get ~90% of the gains through just a couple of simple tricks: 1) Identify medium-scale similarity boundaries in the data structures. E.g.: a sequence of messages in a protocol, such as a C "struct" with a bunch of fields. 2) Compute the binary difference between these structures so that most of the subsequent bytes after the first message are either zeroes or small numbers. Both the sender and receiver have to keep the previous message in a buffer to allow this. 3) Use a high-performance compression algorithm that supports "user provided dictionaries", such as Zstandard. Train it with sample data. This above is surprisingly straightforward because it doesn't require complex changes to the underlying data structures. You don't even necessarily need to be able to parse it at all, as long as it has large-scale repeating structures that you can identify.
- gbrits 7y agoAgreed. It really blows generic compression algos out of the water. Didn't know about Zstandard dictionary encoding. This might just be what I'm after. Thanks
- Waterluvian 7y agoSmall lesson learned with MessagePack: We prematurely utilized it and paid the price of "nothing's human readable without first unpacking it" without actually benefiting much given we weren't shipping much data that often."
- IshKebab 7y agoI was looking into binary JSON formats recently (there are a ton), and there are some important differences to note. Especially for my application - you can't do what Amazon calls a "sparse read". Say I have a 10 GB JSON file and I only want to read one key, no dice. You have to parse the entire file. "But, maybe some of these binary JSON formats are smarter!" you think. Well, almost all of them aren't. Only two are: BSON, and Amazon Ion. Unfortunately BSON limits the message size to 2 GB. Amazon Ion is also the only format that actually deduplicates object keys. Definitely the most capable and well-designed of these formats. Unfortunately it is also the most complicated. Sadly a lot of these formats make questionable choices, like storing numbers in big endian format (why?), using explicit `uint8`/`uint16`/`uint32` sizes rather than something like Protobuf's varint, etc. And there are also a load of them that are nearly identical. You really have to dig deep to find the critical flaws.
- ludocode 7y agoBinn also does binary length prefixing of all values, allowing you to skip through it. It's also a lot simpler than Amazon Ion and doesn't have the awful mistakes of BSON so it might do the trick for you. It's not at all popular though. https://github.com/liteserver/binn https://github.com/liteserver/binn The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. You have to basically encode a message inside-out to calculate the nested sizes of everything. This is going to be hugely slow and memory-intensive if you're encoding a 10 GB file, and it's useless for messages on the scale of kilobytes so there isn't any point. MessagePack on the other hand can be encoded in one pass from start to finish as long as you know the element counts of your maps and arrays beforehand. > storing numbers in big endian format (why?) Embedded processors tended to be big-endian, like older PowerPC and older ARM. These formats are designed for embedded so it (probably?) improved performance on those processors. This is less true now since virtually all modern ARM processors and probably most other embedded processors now run in little-endian mode. Ultimately what it comes down to is that these formats are designed for the opposite of your use case. I don't know what you're using a 10 GB JSON file for but there must be a better storage solution for you than a schemaless serialization format.
- 7y ago
- dilipray 7y agohttps://medium.com/unbabel/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/unbabel/the-need-for-speed-experimenting-... > It’s that is not enough to just know some new cool technology, nod along and go about your day with your assumptions unchallenged. You need to find out more, test it out, have a grasp before committing to it, and, if you’re lucky, learn a thing or two in the process.
- emergie 7y agoUnfortunately messagepack also forces IEEE754 standard for floating point numbers. That means it is useless for things like money or large/arbitrary precision numbers. In JSON a number-type value has to be parsed like in javascript, so IEEE754 double. for example: {"number": 0.1} in reality it is parsed as: {"number": 0.100000000000000005551115123126} Solely deserialization followed by serialization might introduce a change in serialized value. Because of problems with IEEE754 in all of our api we use only floats as strings, like {"number": "0.1"}. For us enforced IEEE754/double format for floating point numbers renders number type near useless.
- echlebek 7y agoI think you're conflating implementation details of your JSON decoder with the JSON spec. There is no requirement that JSON numbers are parsed into IEEE floats or doubles. Many JSON parsers support decoding numbers into other decimal data types. https://www.json.org https://www.json.org
- ludocode 7y agoThis is technically correct, except that almost all JSON libraries parse JSON numbers into either 64-bit integer or 64-bit double, so you get those lossy conversions anyway. I've worked on a production app where we had to swap out the JSON parsing libraries in the server (Rails) and all clients (Android, iOS, Rails again) for ones that preserved numbers as BigDecimals. This was a huge pain, it made everything slower, and even then it wasn't ideal because different BigDecimal libraries aren't even necessarily compatible. Foundation's NSDecimalNumber has different limits than Android's BigDecimal for example, and these are the types used by the parsers so you can't just get the raw data to do it yourself. If I had to do it over again I would never rely on JSON's decimal support. I'd rather stuff my decimals in strings and parse them myself.
- StavrosK 7y agoThis is great, but, to complement this, does anyone know a better alternative to TCP? I've recently needed to make two processes communicate over the network and, while MessagePack handles the serialization, I found myself needing something higher level than TCP but lower than HTTP. An annoyance of TCP was that I never knew whether I read all the data. I either read less and leave data unread, or I read more and end up blocking for a long time (or implement timeouts and get the worst of both worlds). What's a good alternative? Maybe 0mq? All I need is to send some MsgPack bytes to another client over the wire, hopefully without having to guess whether there's more to read in the socket or not.
- naikrovek 7y agoDo you mean just opening a port and listening for connections? And a client which connects to that port and sends/receives data? That's not very low level and it's pretty easy to do, if I'm understanding you correctly.
- StavrosK 7y agoHow do you guarantee that you've read the entire buffer without trying to read more and blocking?
- lttlrck 7y agoMaybe I misunderstood but if you add a length header to every packet reading becomes trivial.
- StavrosK 7y agoIt does, but if I can have a library handle that for me, that'd be better. Looks like 0mq is what I want, it also does pub/sub so it frees me from doing that myself.
- kragen 7y agoØMQ is probably a good fit (I think nanomsg is dead?) but there are also a number of protocols at the TCP level that provide a message-sequence interface rather than a byte-sequence interface: SCTP is perhaps the best-known, and Plan9 IL is another (deprecated) solution. For some applications, though, the simplest solution is to shut down one half of the TCP connection once you've finished sending your data. That's how rsh, finger, and HTTP/0.9 responses work, and it's a supported option in HTTP/1.0 and HTTP/1.1. Failing that, preceding each message with a byte count, a la netstrings, is fairly simple; or you can use SLIP-like or COBS framing.
- naikrovek 7y agoOk so without taking away from this library, what's wrong with using a fully binary format of your own? No string parsing, no nonsense, just binary data. It seems like people are becoming afraid of plain binary, and I don't understand why. It's so easy, plain binary is very small, and is extremely easy to parse when compared to anything with text in it, especially if you have to support multiple encodings. 1.) Decide upon a binary format. 2.) Use it. No 3rd party libraries required; you could write writer or reader code for a Commodore 64, if you had to. Keep it simple. I suspect any simple "type-length-value" type of binary file format would be written or read at least as quickly as this, without third party code.
- bufferoverflow 7y agoIt's unlikely you can write a serializer/unserializer that is faster than what we already have. And bug-free on complex objects.
- naikrovek 7y agoIt's unlikely that I can? That's a strong statement. Binary file formats are not difficult. Writing code that does the right thing is also not difficult. I am opposed to the blind use of libraries like this. Developers need to understand what they're doing and how what they're doing is being done at a reasonably low level if they ever hope to become better developers. Masking it all away behind a third party library is not how you understand the code that's running on your systems.
- bufferoverflow 7y agoHow about you write a simple binary substitute that covers just the standard JSON (number, string, true, false, null, array, object), see how you do compared to the existing JSON serializeres / deserializers and MsgPack.
- enitihas 7y agoCommunication. The new guy joining your team can be immediately productive with JSON, protobuf, etc, but will take some time to understand the homegrown format. Also much more error prone. Most teams don't want to be writing binary encoders and decoders as that's not where their expertise lies, and would much rather focus their time on business features.
- Aardappel 7y agoIf you want a binary JSON, https://google.github.io/flatbuffers/flexbuffers.html https://google.github.io/flatbuffers/flexbuffers.html is worth looking at, since it carriers over the advantages of FlatBuffers (in-place access without unpacking), but without the need of schema like FlatBuffers.
- nojvek 7y agoWhy would one use msgpack over flat buffers? AFAIK offers an amazing zero copy feature meaning you can get to the data of interest without having to parse the whole object in memory. Since every type is length prefixed you’re can jump around indexes quickly to access what’s needed and only read that. It seems msgpack still needs a deserialize step to partially read data right ? Netflix uses flatbuffers and works wonders in low powered devices.
- infinityStudio 7y agoVery informative and nice to see your article. Here is a list of technology blogs ideas check more on http://beneficialreviews.com/ http://beneficialreviews.com/
- DmitryOlshansky 7y agoNeeds honorable mentions to: - Avro - CBOR - SMILE And BSON anyone? I think not much people besides MongoDB using it though. Yes, compressing JSON with gzip-style compressor usually yields 0.5-1% better results then equally compressed binary format (in my limited testing). Still the serialization speed and savings on compression are great to have.