4 ms·
MessagePack: It's like JSON, but fast and small.
- mattbillenstein 2y agoI've built a few systems using msgpack-rpc - serves really well as a transport format in my experience!
- slurpyb 2y agoShouts to msgspec - i havent had a project without it in awhile.
- ubutler 2y ago+1 It’s almost as indispensable as tqdm for a data scientist at least.
- vdqtp3 2y agoIgnorant question - is the relatively small size benefit worth another standard that's fairly opaque to troubleshooting and loses readability? Is there a direct comparison of why someone should choose this over alternatives? 27 bytes down to 18 bytes (for their example) just doesn't seem like enough of a benefit. This clearly isn't targeted to me in either case, but for someone without much knowledge of the space, it seems like a solution in search of a problem.
- 2muchcoffeeman 2y agoI’m not sure why you wouldn’t just develop in json, then flick a switch to use binary.
- cryptonector 2y agoWe already have CBOR and other binary JSONs.
- jakewins 2y agoCBOR is a clone of msgpack: https://en.m.wikipedia.org/wiki/CBOR https://en.m.wikipedia.org/wiki/CBOR
- willhbr 2y agoIf you need a format that can transport byte arrays unmodified (image data, etc), msgpack (or protos or whatever) is much better than JSON since you don't have to base64 encode or escape the data. It also supports non-string keys which can be convenient.
- div3rs3 2y agoI'd argue the value goes up with larger payloads. The tradeoff is ease of use vs efficiency.
- palmfacehn 2y agoIt starts to make sense if you are returning a large array of objects and each object contains several long values. Unfortunately it looks like msgpack doesn't support u128 or arbitrary precision big integers. I suppose you can always cast to byte[].
- diath 2y agoIt's useful when dealing with high traffic networked services, the little saves here and there have compounding effects over time and save you a lot of bandwidth.
- goodpoint 2y agoMessagePack and CBOR allow zero-copy parsing.
- crabbone 2y agoFrom cursory reading of the specification, the format doesn't seem to offer anything groundbreaking, no particular benefits compared to other similar formats. Whatever your messaging format is going to be, the performance will mostly depend on the application developer and their understanding of the specifics of the format. So, the 20% figure seems arbitrary. In practical terms, I'd say: if you feel confident about dealing with binary formats and like fiddling with this side of your application, probably, making your own is the best way to go. If you don't like or don't know how to do that, then, probably, choosing the one that has the most mature and robust tools around it is the best option. ---- NB. It's also useful to remember that data transfer of the network is discrete, with the minimum chunk of information being MTU. So, for example, if most of the messages exchanged by the application were smaller than one MTU before attempting to optimize for size, then making these messages shorter will yield no tangible benefit. It's really only worth to start thinking about optimizations when a significant portion of the messages are measured in at least low double digits of MTUs, if we believe in the 20% figure. It's a similar situation with the storage, which is also discrete, with the minimum chunks being one block. Similar reasoning applies here as well.
- karteum 2y agoI discovered JSON Binpack recently, which works either schemaless (like msgpack) or - supposedly more efficiently - with a schema. I haven't tried the codebase yet but it looks interesting. https://jsonbinpack.sourcemeta.com/ https://jsonbinpack.sourcemeta.com/
- mdhb 2y agoCBOR: It’s like JSON but fast and small but also an official IETF standard. https://cbor.io/ https://cbor.io/
- nmadden 2y agoCBOR is basically a fork of MsgPack. I prefer the original - it’s simpler and there are more high-quality implementations available.
- jpc0 2y agoCBOR is actively used in the WebAuthN spec (passkeys) so browsers ship with en implementation... And if you intend to support it even via a library you will be shipping an implementation as well. https://www.w3.org/TR/webauthn-2/#sctn-conforming-all-classes https://www.w3.org/TR/webauthn-2/#sctn-conforming-all-classe...
- throw5959 2y agoWhich Web API can encode and decode CBOR? I'm not aware of any, and unless I'm mistaken you will need to ship your own implementation in any case.
- camgunz 2y agoDisclaimer: I wrote and maintain a MessagePack implementation. Reading through this, it looks like they toss out indefinite length values, "canonicalization", and tags, making it essentially MP (MP does have extension types, I should say). https://fidoalliance.org/specs/fido-v2.0-ps-20190130/fido-client-to-authenticator-protocol-v2.0-ps-20190130.html#ctap2-canonical-cbor-encoding-form https://fidoalliance.org/specs/fido-v2.0-ps-20190130/fido-cl...
- feverzsj 2y agoIt's too complex, and the implementation is poor[0]. [0]: https://github.com/getml/reflect-cpp/tree/main/benchmarks https://github.com/getml/reflect-cpp/tree/main/benchmarks
- janalsncm 2y agoIn my experience protobuf was smaller than MessagePack. I even tried compressing both with zstd and protobuf was still smaller. On the other hand protobuf is a lot less flexible.
- comonoid 2y agoMessagePack is self-describing (it contains tags like "next bytes are an integer"), but Protobuf uses external scheme.
- jabwd 2y agoYou can decode protobuf in the same way, I've written several decoders in the past that don't rely on an external schema. There are some types you can't always decode with a 100% confidence, but then again JSON or something like it isn't strongly typed either.
- comonoid 2y agoIndeed.
- SigmundA 2y agoprotobuf doesn't have key names without the schema right? Wouldn't it just decode to 1,type,value 2,type,value without the schema no names? Human readable key names is a big part of what makes a self describing format useful but also contributes to bloat a format with an embedded schema in the header would help.
- jabwd 2y agoSorry I never ended up replying. You are correct, in protobuf you have field numbers ( and I have to admit what I commented has been a number of years ago by now my knowledge is crusty ). Usually when doing this type of work, as I was doing, is to reverse some protocol. It didn't make me hate the format though; many protocols I had reversed in the past were vastly more complicated to decode and understand without having field names. If you ever want to melt your brain get into RTMP decoding :3
- toomim 2y agoMessagePack saves a little bit of space and CPU ... but not a lot: https://media.licdn.com/dms/image/v2/D5612AQF-nFt1cYZhKg/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1703965440210?e=2147483647&v=beta&t=XOmhYJP5X6iYJ17RYsk3SjX9FTvfcZbQFP-OTOg6flk https://media.licdn.com/dms/image/v2/D5612AQF-nFt1cYZhKg/art... Source: https://www.linkedin.com/pulse/json-vs-messagepack-battle-data-efficiency-akshay-singh-kanawat-7dx2c https://www.linkedin.com/pulse/json-vs-messagepack-battle-da...
- koito17 2y agoAn approximate 20% reduction in bandwidth looks significant to me. I think the problem here is that the chart uses a linear scale instead of a logarithmic scale. Looking at the data, I'm inclined to agree that not much CPU is saved, but the point of MessagePack is to save bandwidth, and it seems to be doing a good job at that.
- motorest 2y ago> An approximate 20% reduction in bandwidth looks significant to me. Significante with regards to what? Not doing anything? Flipping the toggle to compress the response?
- tasuki 2y ago> An approximate 20% reduction in bandwidth looks significant to me. To me it doesn't. There's compression for much bigger gains. Or just, you know, just send less data? I've worked at a place where our backend regularly sent humongous jsons to all the connected clients. We were all pretty sure this could be reduced by 95%. But, who would try to do that? There wasn't a business case. If someone tried succeeded, no one would notice. If someone tried and broke something, it'd look bad. So, status quo...
- bythreads 2y agoIn a system that requires the absolute speediest throughput compression is actually usually the worst thing in a parsechain - so parsing without first decompression is valuable. I've tried messagepack a few times, but to be honest the hassle of the debugging was never really worth it
- agieocean 2y agoMade an open image format with this for constrained networks and it works great
- eeasss 2y agoSerialziation vulnerabilities anyone
- fshafique 2y agoDoes it solve the problem of repeating set of keys in an object array, eg. when representing a table? I don't think using a dictionary of key values is the way to go here. I think there should be a dedicated "table" type, where the column keys are only defined once, and not repeated for every single row.
- ubutler 2y agoMessagePack can encode rows as well and then you just need to manage linking the keys during deserialization. In fact, it can encode arbitrary binary without needing base64 like JSON.
- feverzsj 2y agoYou can just use array of array like most scientific applications do.
- fshafique 2y agoI can do that with JSON too. I was hoping MessagePack would have built-in functionality to do it.
- ubutler 2y agoAlthough MessagePack is definitely not a drop-in replacement for JSON, it is certainly extremely useful. Unlike JSON, you can’t just open a MessagePack file in Notepad or vim and have it make sense. It’s often not human readable. So using MessagePack to store config files probably isn’t a good idea if you or your users will ever need to read them for debugging purposes. But as a format for something like IPC or high-performance, low-latency communication in general, MessagePack brings serious improvements over JSON. I recently had to build an inference server that needed to be able to communicate with an API server with minimal latency. I started with gRPC and protobuf since it’s what everyone recommends, yet after a lot of benchmarking, I found a way faster method to be serving MessagePack over HTTP with a Litestar Python server (it’s much faster than FastAPI), using msgspec for super fast MessagePack encoding and ormsgpack for super fast decoding. Not sure how this beat protobuf and gRPC but it did. Perhaps the Python implementation is just slow. It was still faster than JSON over HTTP, however.
- deleted 2y ago[deleted]
- junon 2y agoMakes me wish Cap'n Proto was more prevalent and developer friendly. It ticks quite a few of my boxes.
- sd9 2y agoIt drops the most useful aspect of JSON, which is that you can open it in a text editor. It's like JSON in that it's a serialisation format.
- PittleyDunkin 2y agoAnd here I was thinking serialization was the most useful aspect of json (and it's not even great at that)
- majewsky 2y agoSerialization is what it is, not why it is useful. The most useful aspect of JSON is undoubtedly wide support (historically, that every web browser carried a parser implementation from day 1; contemporarily, the availability as a library in nearly every programming language, often in std, plus shell accessibility through jq).
- PittleyDunkin 2y ago> Serialization is what it is, not why it is useful. Is there a non-teleological manner in which to evaluate standards? > The most useful aspect of JSON is undoubtedly wide support This is a fantastic example of how widespread technology doesn't imply quality. Don't get me wrong, I love JSON. It's a useful format with many implementations of varying quality. But it's also a major pain in the ass to deal with: encoding errors, syntax errors, no byte syntax, schemas are horribly implemented. It's used because it's popular, not because it has some particular benefit. In fact, I'd argue JSON's largest benefit as opposed to competitive serializers has been to not give a fuck about the quality of (de)serialization. Who gives a fuck about the semantics of parsing a number when that's your problem?!?
- buserror 2y agoI played quite a bit with MessagePack, used it for various things, and I don't like it. My primary gripes are: + The Object and Array needs to be entirely and deep parsed. You cannot skip them. + Object and Array cannot be streamed when writing. They require a 'count' at the beginning, and since the 'count' size can vary in number of bytes, you can't even "walk back" and update it. It would have been MUCH, MUCH better to have a "begin" and "end" tag --- err pretty much like JSON has, really. You can alleviate the problems by using extensions, store a byte count to skip etc etc but really, if you start there, might as well use another format altogether. Also, from my tests, it is not particularly more compact, unless again you spend some time and add a hash table for keys and embed that -- but then again, at that point where it becomes valuable, might as well gzip the JSON! So in the end it is a lot better in my experience to use some sort of 'extended' JSON format, with the idiocies removed (trailing commas, forcing double-quote for keys etc).
- masklinn 2y ago> Object and Array cannot be streamed when writing. They require a 'count' at the beginning Most languages know exactly how many elements a collection has (to say nothing of the number of members in a struct).
- touisteur 2y agoI think the pattern in question might be (for example) the way some people (like me) sometimes write JSON as a trace of execution, sometimes directly to stdout (so, no going back in the stream). You're not serializing a structure but directly writing it as you go. So you don't know in advance how many objects you'll have in an array.
- deleted 2y ago[deleted]
- buserror 2y agoIf your code is compartimented properly, a lower layers (sub objects) doesn't have to have to do all kind of preparations just because a higher layer has "special needs". For example, pseudo code in a sub-function: if (that) write_field('that'); if (these) write_field('these'); With messagepack you have to go and apply the logic to count, then again to write. And keep a state for each levels etc.
- _vrtq 2y agoI like how it makes binary easy. I wish there was a simple way to do binary in JSON.
- dallbee 2y agoA trick I often use to get the most out of messagepack is using array encoding of structs. Most msgp libraries have support for this. This gives you a key benefit of protobuf, without needing external schema files: you don't have to pay the space of the keys of your data. This is simply not something you can do with JSON, and depending on your data can yield substantial space savings.
- xcircle 2y agoIn our C++ project, we use the nlohmann library for handling JSON data. When sending JSON over MQTT, we leverage a built-in function of nlohmann to serialize our JSON objects into the MessagePack format. You can simply call jsonObj.to_msgpack(). Similarly, the data can be decoded back easily.
- jedisct1 2y agoZigPack is a pretty good MessagePack implementation written in Zig: https://github.com/thislight/zigpak https://github.com/thislight/zigpak
- accelbred 2y agoI really like MessagePack as a serialization format but it does leave a lot underspecified, such as how to handle decoding/encoding integers. While the encoding has signed and unsigned types, the data model does not. Its not straightforward to translate to a data midel with separate signed and unsigned types. The separation between str and bin types doesnt make sense for my usecases, and others with similar usecases either always use bin or always use str. logically bin would make sense, but str has fixstr and theres no fixbin, so more efficient to use str. Both are valid but thats more discrepancies to deal with. For my personal usecases I'm thinking of forking MessagePack, removing str and ext, reallocting tag space a bit, having a clear mapping to low level language types, adding 128 bit integers, and maybe a frame type for streams.