3 ms·
I love flatbuffers but they're only worthwhile in a very small problem space. If your main concern is "faster than JSON" then you're better off using Protocol
by afavour 4y ago
I love flatbuffers but they're only worthwhile in a very small problem space.
If your main concern is "faster than JSON" then you're better off using Protocol Buffers simply because they're way more popular and better supported. FlatBuffers are cool because they let you decode on demand. Say you have an array of 10,000 complex objects. With JSON or Protocol Buffers you're going to need to decode and load into memory all 10,000 before you're able to access the one you want. But with FlatBuffers you can decode item X without touching 99% of the rest of the data. Quicker and much more memory efficient.
But it's not simple to implement. You have to write a schema then turn that schema into source files in your target language. There's an impressive array of target languages but it's a custom executable and that adds complexity to any build. Then the generated API is difficult to use (in JS at least) because of course an array isn't a JavaScript array, it's an object with decoder helpers.
It's also quite easy to trip yourself up in terms of performance by decoding the same data over and over again rather than re-using the first decode like you would with JSON or PB. So you have to think about which decoded items to store in memory, where, for how long, etc... I kind of think of it as the data equivalent of a programming language with manual memory management. Definitely has a place. But the majority of projects are going to be fine with automatic memory management.
- packetlost 4y agoThis. They're only really worthwhile in performance sensitive contexts or when you only need a small portion of a larger data-structure.
- mort96 4y ago> But it's not simple to implement. You have to write a schema then turn that schema into source files in your target language. There's an impressive array of target languages but it's a custom executable and that adds complexity to any build. Then the generated API is difficult to use Worth noting that all these things are true for protobuf as well.
- lmm 4y agoLess so. Many languages have a native implementation of protobuf that uses that language to build rather than a binary (e.g. pbandk), and will generate relatively idiomatic code.
- anonymoushn 4y agoAre there protobuf throughput benchmarks somewhere? I haven't been able to verify that they're faster than JSON. Edit: I was able to find these at https://github.com/hnakamur/protobuf-deb/blob/master/docs/performance.md#parsing-performance https://github.com/hnakamur/protobuf-deb/blob/master/docs/pe... but these numbers don't seem conclusive. Protobuf decode throughput for most schemas tested is much slower than JSON, but protobufs will probably also be a bit smaller. One would have to compare decode throughput for the same documents serialized both ways rather than just looking at a table.
- edwincheese 4y agoHere is a benchmark of Protobuf, FlatBuffers, and Cap'n Proto across Go and Rust. It doesn't include JSON tho https://github.com/kcchu/buffer-benchmarks https://github.com/kcchu/buffer-benchmarks
- dub 4y agoWhile I haven't benchmarked JSON vs protobuf, I've observed that JSON.stringify() can be shockingly inefficient when you have something like a multi-megabyte binary object that's been serialized to a base64 and dropped with an object. As in, multiple hundreds of megabytes of memory needed to run JSON.stringify({"content": <4-megabyte Buffer that's been base64-encoded>}) in node
- qorrect 4y agoWhat kind of sicko embeds a whole binary in JSON ?
- mrlonglong 4y agoAsshole coders, perhaps.
- ycombobreaker 4y agoIsn't HTTP POST content similarly encoded? Likewise with small embedded images in CSS, though I am rusty on that topic. Likewise with binary email attachments in SMTP (though this may be uuencoded, same net effect). The particular example of a trivial message that is mostly-binary just sounds like a useful test case, more than anything else.
- ilogik 4y agothere are json stream decoders that allow you to deocde large objects without using a lot of memory, but they are harder to use
- karmakaze 4y agoIf you only want faster than JSON then a 'binary json's like MessagePack or JSONB (is that the PostgreSQL one?) avoids dealing with schema. If you want schema it's not like JSON.
- elcritch 4y agoMsgPack is great. However I’ve been curious about Amazon’s Ion. Mainly since the namespace stuff allow you to define tags upfront into ints. MagPack transfers field names as strings still unless you crest your own string to int conversion.
- BerislavLopac 4y agoI think that Amazon Ion is gravely underappreciated. [0] [0] https://amazon-ion.github.io https://amazon-ion.github.io