8 ms·
I was looking into binary JSON formats recently (there are a ton), and there are some important differences to note. Especially for my application - you can't d
by IshKebab 7y ago
I was looking into binary JSON formats recently (there are a ton), and there are some important differences to note. Especially for my application - you can't do what Amazon calls a "sparse read". Say I have a 10 GB JSON file and I only want to read one key, no dice. You have to parse the entire file.
"But, maybe some of these binary JSON formats are smarter!" you think. Well, almost all of them aren't. Only two are: BSON, and Amazon Ion. Unfortunately BSON limits the message size to 2 GB.
Amazon Ion is also the only format that actually deduplicates object keys. Definitely the most capable and well-designed of these formats. Unfortunately it is also the most complicated.
Sadly a lot of these formats make questionable choices, like storing numbers in big endian format (why?), using explicit `uint8`/`uint16`/`uint32` sizes rather than something like Protobuf's varint, etc. And there are also a load of them that are nearly identical. You really have to dig deep to find the critical flaws.
- ludocode 7y agoBinn also does binary length prefixing of all values, allowing you to skip through it. It's also a lot simpler than Amazon Ion and doesn't have the awful mistakes of BSON so it might do the trick for you. It's not at all popular though. https://github.com/liteserver/binn https://github.com/liteserver/binn The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. You have to basically encode a message inside-out to calculate the nested sizes of everything. This is going to be hugely slow and memory-intensive if you're encoding a 10 GB file, and it's useless for messages on the scale of kilobytes so there isn't any point. MessagePack on the other hand can be encoded in one pass from start to finish as long as you know the element counts of your maps and arrays beforehand. > storing numbers in big endian format (why?) Embedded processors tended to be big-endian, like older PowerPC and older ARM. These formats are designed for embedded so it (probably?) improved performance on those processors. This is less true now since virtually all modern ARM processors and probably most other embedded processors now run in little-endian mode. Ultimately what it comes down to is that these formats are designed for the opposite of your use case. I don't know what you're using a 10 GB JSON file for but there must be a better storage solution for you than a schemaless serialization format.
- IshKebab 7y ago> The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. Yeah this is true, except for BSON because it uses fixed-size length prefixes, so you can just go back and fill them in later. Presumably that's why they used fixed-size lengths. The downsides are it is less space efficient and limited to 2GB. In any case Amazon make the very good point that formats are read more often than they are written. It makes sense to optimise for the read case. > Embedded processors tended to be big-endian, like older PowerPC and older ARM. Nobody uses PowerPC anymore, and ARM hasn't been big endian for ages. Also MessagePack isn't designed for embedded systems and it still uses big endian. I don't think that's the reason. I suspect it's from a misguided belief that "network byte order" still matters. And I totally agree, a schema-based format makes way more sense for my use case - changing is difficult though.
- IshKebab 7y agoI looked into Binn, but unfortunately it has a 2GB file size limit too.