4 ms·

Binn also does binary length prefixing of all values, allowing you to skip through it. It's also a lot simpler than Amazon Ion and doesn't have the awful mistak

Binn also does binary length prefixing of all values, allowing you to skip through it. It's also a lot simpler than Amazon Ion and doesn't have the awful mistakes of BSON so it might do the trick for you. It's not at all popular though. https://github.com/liteserver/binn https://github.com/liteserver/binn The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. You have to basically encode a message inside-out to calculate the nested sizes of everything. This is going to be hugely slow and memory-intensive if you're encoding a 10 GB file, and it's useless for messages on the scale of kilobytes so there isn't any point. MessagePack on the other hand can be encoded in one pass from start to finish as long as you know the element counts of your maps and arrays beforehand. > storing numbers in big endian format (why?) Embedded processors tended to be big-endian, like older PowerPC and older ARM. These formats are designed for embedded so it (probably?) improved performance on those processors. This is less true now since virtually all modern ARM processors and probably most other embedded processors now run in little-endian mode. Ultimately what it comes down to is that these formats are designed for the opposite of your use case. I don't know what you're using a 10 GB JSON file for but there must be a better storage solution for you than a schemaless serialization format.

IshKebab 7y ago
> The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. Yeah this is true, except for BSON because it uses fixed-size length prefixes, so you can just go back and fill them in later. Presumably that's why they used fixed-size lengths. The downsides are it is less space efficient and limited to 2GB. In any case Amazon make the very good point that formats are read more often than they are written. It makes sense to optimise for the read case. > Embedded processors tended to be big-endian, like older PowerPC and older ARM. Nobody uses PowerPC anymore, and ARM hasn't been big endian for ages. Also MessagePack isn't designed for embedded systems and it still uses big endian. I don't think that's the reason. I suspect it's from a misguided belief that "network byte order" still matters. And I totally agree, a schema-based format makes way more sense for my use case - changing is difficult though.
IshKebab 7y ago
I looked into Binn, but unfortunately it has a 2GB file size limit too.

IshKebab 7y ago

> The reason most formats don't length-prefix everything is because it makes it costly to encode in both time and space. Yeah this is true, except for BSON because it uses fixed-size length prefixes, so you can just go back and fill them in later. Presumably that's why they used fixed-size lengths. The downsides are it is less space efficient and limited to 2GB. In any case Amazon make the very good point that formats are read more often than they are written. It makes sense to optimise for the read case. > Embedded processors tended to be big-endian, like older PowerPC and older ARM. Nobody uses PowerPC anymore, and ARM hasn't been big endian for ages. Also MessagePack isn't designed for embedded systems and it still uses big endian. I don't think that's the reason. I suspect it's from a misguided belief that "network byte order" still matters. And I totally agree, a schema-based format makes way more sense for my use case - changing is difficult though.

IshKebab 7y ago

I looked into Binn, but unfortunately it has a 2GB file size limit too.