15 ms·
> MessagePack: It's like JSON. but fast and small and complete. Both are minimal self describing data serialization formats, but JSON is incomplete. It misses
by dathinab 7y ago
> MessagePack: It's like JSON. but fast and small
and complete.
Both are minimal self describing data serialization formats, but JSON is incomplete. It misses a type to represent one of the fundamental type: byte blobs. And also parts of floats.
Which means there are a lot inconsistent ways to hack that in e.g. base64 encoded strings. But this means it's loosing partially it's property of being self-describing (same if you allow numbers as strings e.g. "-12").
Just to be clear I'm not saying JSON should have bytes directly in it. But a "native" base64 string additional to string, number, null, list and map would help. E.g. `{ "bytes": b"YWJjZGU=" }`
Wrt. floats it's missing (+/-) Infinity, NaN is more like a error variant so it's okish to be missing, but then why not have it.
Also for completeness it would be better to differentiate between int and float as float is imprecise due to rounding errors.
----
PS: I don't like `null` but having it in this kind of data serialization format is still required. But the difference between a field not being there and it being `null` can be a mess and not all tools handle it well.
PPS: Both JSON and MsgPack can be used for non self describing serialization, e.g. by serializing the a record (fixed number of fields in known order) as a list of values instead of a mapping. But both are focused on enabling self describing serialization.
EDIT: PPPS: Yes, it's a very weak form of self-describing is use here, there are systems which are much more self describing e.g. XML+XML Schema linked from the XML document but also tend to be far more complex
- sdegutis 7y agoI've become particularly fond of Ion[1]. The main benefit JSON has over Ion is its ubiquity, even in many standard libraries. [1] https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/
- dathinab 7y agoThis looks quite nice. But then S-Expressions and the soon to be there templates add quite a bit of unnecessary complexity IMHO.
- sdegutis 7y agoI doubt it's any more complicated to add S-Expressions than to change list parsing to take either [] or () and set a type flag based on which one it finds. Semantically, it's not more complex either, just giving one extra piece of user-defined metadata to the actual data that's still basically a list.
- 2J0 7y agoSee my comment about ASN.1 which is anything but incomplete...
- progval 7y agoJSON also misses dictionaries/"objects" with non-string keys. Msgpack discourages you from using them because languages like JS don't support it, but allows it.
- mr_toad 7y agoJavaScript has Maps which allow anything as a key, but they were added after JSON was thought up, and they don’t have an literal format.
- rgovostes 7y agoWhy are we passing around binary data re-encoded to only use 6 bits per transmitted byte? In some cases it might be practical to shoehorn binary data where a text string has historically gone (e.g., data URLs, emails) but making a new format that can’t handle binary data without costly armoring seems like we’ve basically given up on using the right tool for the job.
- AnIdiotOnTheNet 7y ago> seems like we’ve basically given up on using the right tool for the job That's webtech in a nutshell isn't it?
- emj 7y agoCognitive load. A readable dataformat is easy to understand, binary is not readable. The thing is standards for formats get broken all the time, and a broken CSV files is usually easier to fix than a non-standard binary format.
- londons_explore 7y agoThe same argument could have been made when the ppm image format was replaced with binary formats like JPEG. Nobody says "what a shame I can't open that jpeg image in a text editor". Everyone understands that an image is an image, and there are tools for dealing with images, like photoshop. The same could be said for data interchange formats like json.
- emj 7y agoThis was specifically about having a fileformat encoding binary data, and in that vein there is a place for PPM, it's probably better for many small bitmaps than PNG. My old PPMs from my email archive look horrible as jpegs https://xkcd.com/1683/ https://xkcd.com/1683/ . JSON vs. Bencode, JSON can get you the data good enough a lot faster, while a bencode parser is easier to write. My feeling is that most formats with schemas usually fail somewhere in the details anyways. Honestly I think I want too much from them; e.g. the YANG Data model language is a kitchen sink but lacks a lot.
- nprateem 7y agoAnd comments. Not being able to put comments in JSON files is beyond stupid.
- jrockway 7y agoJSON is a binary computer-to-computer interchange format. No computer-to-computer interchange format has comments. The problem with JSON is that it's just human-readable enough that people think it's a config file format. It's not.
- knodi123 7y agoWhat are the material differences between a config file format, and a machine-to-machine only format?
- inetknght 7y agoWell, for one: most config file formats specifically include a method for the user to add bytes to the data which won't be parsed or interpreted. In JSON... you might try adding a {"comment": "this is a bad way to comment"} object or key/value and hope you don't collide with an object's dedicated comment field, and also don't cause the parser to raise an error for having a field which it didn't expect.
- korijn 7y agoThe key difference is the target audience. The format's "UX", so to speak, is optimized for the target audience (machine or human). @jrockway makes an excellent point.
- mark-r 7y ago> Also for completeness it would be better to differentiate between int and float as float is imprecise due to rounding errors. Not true for most numbers. A 64-bit IEEE float can handle any 53-bit integer with no error at all.
- zeroimpl 7y agoIt can represent less than 0.1% of all 64-bit ints. That doesn’t seem like most to me.
- mark-r 7y agoI didn't claim 64-bit ints, I claimed 53-bit ints. How often do you use ints in the range from 54 to 64 bits? I'm willing to bet it's close to never. That's what I meant by "most". You have to consider the distribution.
- ben0x539 7y agorandomly generated IDs as 64 bit ints, pretty often. I mean, tweet IDs are up there too.
- ben0x539 7y agoAlmost all numbers aren't 53-bit integers (or smaller) :P
- mark-r 7y agoSince everybody feels like getting all pedantic on me, let me clarify what I was trying to say. Most of the integers you encounter in real life will be 53 bits or less, and can be held in an IEEE-754 64-bit float without introducing any error. It is not worth introducing a separate integer type for the exceptions, particularly when the language used as the basis of the format (Javascript) does not have an integer type.
- eyelidlessness 7y agoJSON is also missing a date type. Sure, you can use a string serialization but that's ambiguous. It's also non-extensible so these types it's missing cannot be added without assumptions or overhead from convention. Transit is a good solution to this.
- kenoyer130 7y agoTHIS! Not having a defined Date format has caused so much confusion with consumers of our API since every json API might have a different ISO date format (or some homegrown madness).
- eyelidlessness 7y agoIn APIs I've integrated in recent years, I've seen: 1. ISO UTC 2. ISO local 3. Epoch 4. 2018-04-20 5. Jan 01 2016 11:58am Probably others that I can't recall. It's a damn mess.
- anamexis 7y agoTo be fair, 1, 2 and 4 are all just ISO.
- eyelidlessness 7y agoThat's true. But 2 is substantially less useful than 1 and I encountered 4 mixed with 3.
- Jestar342 7y ago`/Date(1234567890)/` That will probably cause a twitch to any dev that used Asp.NET up until a few years ago.
- thawaway1837 7y agoI’m twitching away...
- rcfox 7y agoJSON doesn't use floating point numbers, it uses signed decimal numbers. It's perfectly valid to store a number in JSON that isn't representable in a n-bit IEEE 754 floating point number.
- temporallobe 7y agoIt’s “incomplete” because JS doesn’t natively have the concept of bytes or the other things you mention. JSON is JavaScript Object Notation, so there’s no reason it would support features beyond what JavaScript itself can. Of course JSON is now commonly used by other languages and platforms, but its origins in JavaScript are the source of its limitations.
- lokedhs 7y agoJSON is also missing an integer type. Read the spec. Basically all it says is that the numeric type is a float, but doesn't say anything about its precision. I'm amazed that there haven't been any security vulnerabilities found yet that takes advantage of different number formats in different JSON implementations. You'd think that a data serialisation format should get at least numbers right. All the ones that came before (and after) did, yet we're still stuck with JSON.
- hombre_fatal 7y agoIs that like saying it doesn't have a username type? How could you ever know this bit of data is a username if it doesn't have a username type?
- lokedhs 7y agoI think there is a rather large difference between not having a username type vs. not having an integer type. As it turns out, the only way you can reliably store an integer in JSON is to use a string field. This is incidentally also the correct way to store a username in JSON.
- leu-mas 7y agoAlso, never understood why JSON allows for duplicate keys at the same level. Behavior is implemented differently across languages. Some throwing an error on parsing of duplicate keys while others simply overwrite the first value encountered with the second.
- NoZZz 7y agoASN.1 already exists!!!