21 ms·
Ask HN: What happened to flatbuffers? Are they being used?
A few years ago, their was a talk about flatbuffers[0] being a memory efficient and quicker method than JSON.
Anyone have any real world experience with it?
[0] https://github.com/google/flatbuffers https://github.com/google/flatbuffers
- doomlaser 4y agoApache Arrow uses flatbuffers in the .arrow format: https://arrow.apache.org/docs/format/Columnar.html https://arrow.apache.org/docs/format/Columnar.html
- politician 4y agoYes, flatbuffers are fantastic. Let me know if you have any specific questions; happy to respond.
- wzvici 4y ago1. Are they easier to use than JSON? 2. Is it just putting strings into a big array? 3. Do you have an examples of them being used?
- politician 4y ago1. It's not easier to use them than JSON when just getting started. However, the pay off is the strong typing and zero-copy access that they offer to folks that need to support clients on multiple architectures. 2. No, writers can directly embed structs and primitive data types into binary buffers at runtime through an API generated from an IDL file. Readers use direct memory access to pull values out of the buffers. If you set it up right, this can result in a massive perf boost by eliminating the encoding and decoding steps. 3. Facebook uses them in their mobile app. Another commenter mentioned use of the in the Arrow format. The flatbuffers website isn't the best, but clearly documents the flatbuffers IDL.
- wzvici 4y agoIs there any good writeup about using them that is easy to follow? With drawbacks on using them in prod?
- politician 4y agoThe google documentation has a minimal tutorial. There are implementations for all of the major languages. The level of documentation in the ecosystem, though, is poor. My best recommendation for you is to jump in and get the tutorial/hello world example working in a language you're comfortable with. They aren't hard to use, but they aren't the easiest thing either. Once you get the gist of the API through the tutorial, the other important topics that come up immediately are version control; git repo design; headers and framing. In production, they've been bulletproof. As long as you account for the compile time issues (schema versioning, repo, headers and framing, etc.).
- ardit33 4y agoWe were using protobufs at Spotify and ditched them for simple JSON calls on the client side. No one complained, and never going back to having anything like that on the client side if I can. Just too many drawbacks. For server to server, they might be fine, but to client just stick with JSON. (which when compressed is pretty efficient).
- wzvici 4y agoWhy is it bad in server to client?
- pphysch 4y agoSetting up TLS in gRPC is annoying if the alternative is using an already-existing HTTPS endpoint.
- anuragsoni 4y agoThat sounds like a problem with GRPC not protobufs.
- boardwaalk 4y agoHow is it annoying? To be fair, we’re fronting our gRPC service with a AWS LB that terminates TLS (so our gRPC is plaintext), so we don’t deal with certs as direct dependencies of our server.
- cmrdporcupine 4y agoBeing able to debug through a simple curl or browser devtools is golden. Also browser has JSON parsing built in. Less dependencies. Easier tooling overall. In my experience people overuse protobuf. But I also worked at Google, where it's the hammer in constant search of any nail it can find. At the very least, endpoints should provide the option to provide a JSON form through content representation negotiation.
- franciscop 4y ago
- yorwba 4y agoIs using flatbuffers as the on-disk storage format for an application a hare-brained idea? If yes, is it a less hare-brained idea than using the ctypes Python module to mmap a file as a C struct? That's what I'm currently doing to get 10x speedup relative to SQLite for an application bottlenecked on disk bandwidth, but it's unergonomic to say the least. Flatbuffers look like a way to get the same performance with better ergonomics, but maybe there's a catch. (E.g. I thought the same thing about Apache Arrow before, but then I realized it's basically read-only. I don't expect to need to resize my tables often, but I do need to be able to twiddle individual values inside the file.)
- nerpderp82 4y ago> ctypes Python module to mmap a file as a C struct Tell me more! Is your data larger than memory? You need persistence? You might take a look at Aerospike, even on a single node if you need low latency persistence.
- politician 4y agoIt really depends on what you're storing and how you need to access the content to meet performance requirements. A simple example of where flatbuffers shines is in TCP flows or in Kafka where each message is a flatbuffer. In Kafka, the message size and type can be included in metadata. In TCP, framing is your responsibility. Serializing a message queue to a flat file is reasonable and natural. Regarding files-as-C-structs: That isn't (necessarily) harebrained (if you can trust the input), Microsoft Word .DOC files were just memory dumps. However, twiddling bits inside a flatbuffer isn't recommended per the documentation; rather, the guidance is to replace the entire buffer. If you don't want to manage the IO yourself, then a key/value store that maps indices to flatbuffers is entirely possible. I'd suggest a look at Redis.
- maximilianburke 4y agoI don't think it's hare-brained, I think it'd be great. No less hare-brained than storing stuff to disk in any other format like json or yaml. That said, the ergonomics are absolutely awful for modifying existing objects; you can't modify an existing object, you need to serialize a whole new object. There's also a schemaless version (flexbuffers) which retains a number of the flatbuffers benefits (zero-copy access to data, compact binary representation), but is also a lot easier to use for ad-hoc serialization and deserialization; you can `loads`/`dumps` the flexbuffer objects, for example.
- tinglymintyfrsh 4y agoThey're used a lot in video games and embedded systems. They're not something you see advertised. grpc and Thrift are mostly backend service interconnects in lieu of RESTful. Capnproto is also awesome.
- joeld42 4y agoyeah they're used a lot. I think the difference is json is good for data or apis you want to be easily shared, flatbuffers (or protobuf or captnproto) are good for data that stays internal. That's just a guideline and there are plenty of exceptions but it's a starting point to thinking about it.
- jjtheblunt 4y agoDid you look at Wikipedia, which has 2 links including to one free software package using them? https://en.wikipedia.org/wiki/FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers
- trgn 4y agoThey (and similar technologies) are used where it matters. Games, data visualization, ... numerically heavy applications mainly. On a side-note; JSON has been somewhat of a curse. The developer ergonomics of it are so good, that web devs completely disregard how they should layout their data. You know, sending a table as a bunch of nested arrays, that sort of thing. Yuck. In web apps, data is essentially unusable until it has been unmarshalled. Fine for small things, horrible for data-heavy apps, which really so many apps are now. Sometimes I wonder if it will change. I'm optimistic that the popularity of mem-efficient formats like this will establish a new base paradigm of data transfer, and be adopted broadly on the web.
- sdeframond 4y ago> web devs completely disregard how they should layout their data How should data be laid out?
- trgn 4y agoimho, a few suggestions: wrt numerical data See if domain allows for more constraint typing (e.g. ints as actual ints). Will push devs naturally to binary formats. Consider column orientation layout. This will also help with more advanced compression (e.g. delta encoding). When sticking with JSON, avoid large nested hierarchies that would spam the heap when unmarchalling (ie. prefer [1,2,4,5] over [[1,2],[4,5]]. In general, for large payloads, see if you can avoid deserialization of the payload altogether, and just scan through it. Often times the program just ends up copying values from one place (the file) to another (buffer; DOM-objects), so there's really no need to create to allocate the entire data in heap as many individual objects. That's a hit the user will always feel twice: once at parse (freezing), once at garbage-collection (framerate hickups). You could technically do that with stream parsing of JSON, but then you need special libraries anyway. And once moving to stream-based parsing, you may as well choose a format which has a lot of other advantages as well (e.g. rich type system, column-layouts). wrt text It matters less (?). Nonetheless, scannable formats are good here too (e.g. the whole reason we e.g. have line delimited json, to bypass JSONs main limitation). These are just some general workable ideas. ymmv, ianal, etc..
- nerpderp82 4y agohttps://capnproto.org/ https://capnproto.org/ is a superior system due to its ability to handle malicious input.
- Aardappel 4y agoFlatBuffers has a Verifier that can be run in possibly hostile contexts.
- the-alchemist 4y agoI always wondered if it's as awesome as it's seemed. Never got a chance to personally use it...
- nerpderp82 4y agoThis looks like a cool post to follow along with, https://levelup.gitconnected.com/microservices-in-rust-with-graphql-and-capn-proto-e125bec23eae https://levelup.gitconnected.com/microservices-in-rust-with-...
- afavour 4y agoI love flatbuffers but they're only worthwhile in a very small problem space. If your main concern is "faster than JSON" then you're better off using Protocol Buffers simply because they're way more popular and better supported. FlatBuffers are cool because they let you decode on demand. Say you have an array of 10,000 complex objects. With JSON or Protocol Buffers you're going to need to decode and load into memory all 10,000 before you're able to access the one you want. But with FlatBuffers you can decode item X without touching 99% of the rest of the data. Quicker and much more memory efficient. But it's not simple to implement. You have to write a schema then turn that schema into source files in your target language. There's an impressive array of target languages but it's a custom executable and that adds complexity to any build. Then the generated API is difficult to use (in JS at least) because of course an array isn't a JavaScript array, it's an object with decoder helpers. It's also quite easy to trip yourself up in terms of performance by decoding the same data over and over again rather than re-using the first decode like you would with JSON or PB. So you have to think about which decoded items to store in memory, where, for how long, etc... I kind of think of it as the data equivalent of a programming language with manual memory management. Definitely has a place. But the majority of projects are going to be fine with automatic memory management.
- packetlost 4y agoThis. They're only really worthwhile in performance sensitive contexts or when you only need a small portion of a larger data-structure.
- mort96 4y ago> But it's not simple to implement. You have to write a schema then turn that schema into source files in your target language. There's an impressive array of target languages but it's a custom executable and that adds complexity to any build. Then the generated API is difficult to use Worth noting that all these things are true for protobuf as well.
- lmm 4y agoLess so. Many languages have a native implementation of protobuf that uses that language to build rather than a binary (e.g. pbandk), and will generate relatively idiomatic code.
- pantsforbirds 4y agoSimilarly does anyone use https://capnproto.org/ https://capnproto.org/? It's a project I was really interested in a few years back, but I haven't heard much in the way of it lately.
- matdehaast 4y agoAFAIK the whole cloudflare workers ecosystem is built using it. The capnproto creator is lead on the workers project
- jgrahamc 4y agoCorrect. It's extensively used at Cloudflare.
- pantsforbirds 4y agoI've played with using it to interopt some python data science code with some C++ code efficiently without writing the project in Cython or using a tool like Pybind11. It worked pretty well in my test scenario, but I'm not sure how great of an idea that truly is. Is the capn'n proto use case similar to something like ZeroMQ or NNG? I'm still not fully sure.
- jhawk28 4y agoZeroMQ/NNG are the transport layer. Cap'n Proto is mostly data serialization.
- kentonv 4y agoHey that's me. And yeah, if you don't hear much about Cap'n Proto, it's not because it isn't advancing, but more because I don't really have any reason to advertise it. Cap'n Proto's goal right now -- for me at least -- is not to take over the world or anything, but rather to support the needs of my main project, which is Cloudflare Workers. If other people find it useful to, great! But that's not my focus.
- 4y ago
- sosodev 4y agoI'm using flatbuffers as the basis of communication for my multiplayer game. They're really quite pleasant to work with after you get into the flow of it.
- gavinray 4y agoFlatbuffers is used in Arrow, which is a ringing endorsement
- covom 4y agoTensorFlow Lite (tflite) uses flatbuffers. This format, and vendor-specific forks of it, ship on hundreds of millions of phones and other embedded devices.
- mikaraento 4y agoIt's used for several ML-related projects, including as the model format for TensorFlow Lite (TFLite). The TFLite format also has long-term support as part of Google Play Services. The main attraction is ability to pass large amounts of data without having to serialize/deserialize all of it to access fields. (I work for Google but don't speak for it.)
- felixguendling 4y agoIf you're looking for something faster but C++ specific, more compact in serialized size, more efficient in serialization you can try cista: https://github.com/felixguendling/cista https://github.com/felixguendling/cista (Disclaimer: I'm the author, always happy for feedback, eg in the GitHub issues)
- gwbas1c 4y agoYes, I work on a product that uses Flatbuffers to control stormwater: https://optirtc.com/ https://optirtc.com/ Basically, we use a rather bandwith-constrained link between our services running in the cloud, and Particle-based IoT devices deployed in many locations. Some locations are remote, some are urban. I personally haven't had to touch the Flatbuffers code since I joined the company two years ago. It's written and hasn't needed to be maintained.
- jhawk28 4y agoI think https://capnproto.org https://capnproto.org has gained more traction than flatbuffers.
- maximilianburke 4y agoYep! Our platform uses flatbuffers as the primary format for both IPC, including for web responses, and for object persistence. It's a phenomenal format; I'm super happy with it.
- apendleton 4y agoI experimented with them (also with capnproto) at my last job for a usecase involving dense numerical data where being able to randomly seek within a dataset would have been really helpful for speed reasons, but found that as compared to protobuf, these formats were unacceptably bulky (lots of extra padding for word alignment, etc.), and the added cost even just to read the extra data from disk mostly negated the savings from avoiding the explicit decode step, plus would have had significant implications in terms of storage cost, etc. I ended up writing a custom wire format that allowed for seeking with less wasted space. Seems like a neat idea, but as another commenter said, the usecases where it's the best choice seem pretty narrow.
- mcint 4y agoI don't have experience with this. 1) SQLite with BLOB storage gives you binary benefits for file layout and database solutions to metadata, versioning, & indexing into large structure. 2) FlexBuffers look like a more flexible solution within the FlatBuffers library. FlatBuffers was designed around schemas, because when you want maximum performance and data consistency, strong typing is helpful. There are however times when you want to store data that doesn't fit a schema, because you can't know ahead of time what all needs to be stored. For this, FlatBuffers has a dedicated format, called FlexBuffers. This is a binary format that can be used in conjunction with FlatBuffers (by storing a part of a buffer in FlexBuffers format), or also as its own independent serialization format. https://google.github.io/flatbuffers/flexbuffers.html https://google.github.io/flatbuffers/flexbuffers.html https://stackoverflow.com/a/47799699/1020467 https://stackoverflow.com/a/47799699/1020467 3) Might see previous discussion of serialization formats on hn: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=serialization+formats&sort=byPopularity&type=all https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- tlrss 4y agoFlatGeoBuf [1] is an encoding for geographic data (vector features, i.e. points lines polygons and so on) written around flatbuffers that is increasingly well supported in geospatial software (GDAL, MapServer) and people reporting some experiments and demos on the @flatgeobuf Twitter. [1] https://flatgeobuf.org/ https://flatgeobuf.org/