7 ms·

Yes, flatbuffers are fantastic. Let me know if you have any specific questions; happy to respond.

wzvici 4y ago
1. Are they easier to use than JSON? 2. Is it just putting strings into a big array? 3. Do you have an examples of them being used?
- politician 4y ago
  1. It's not easier to use them than JSON when just getting started. However, the pay off is the strong typing and zero-copy access that they offer to folks that need to support clients on multiple architectures. 2. No, writers can directly embed structs and primitive data types into binary buffers at runtime through an API generated from an IDL file. Readers use direct memory access to pull values out of the buffers. If you set it up right, this can result in a massive perf boost by eliminating the encoding and decoding steps. 3. Facebook uses them in their mobile app. Another commenter mentioned use of the in the Arrow format. The flatbuffers website isn't the best, but clearly documents the flatbuffers IDL.
  - wzvici 4y ago
    Is there any good writeup about using them that is easy to follow? With drawbacks on using them in prod?
    - politician 4y ago
      The google documentation has a minimal tutorial. There are implementations for all of the major languages. The level of documentation in the ecosystem, though, is poor. My best recommendation for you is to jump in and get the tutorial/hello world example working in a language you're comfortable with. They aren't hard to use, but they aren't the easiest thing either. Once you get the gist of the API through the tutorial, the other important topics that come up immediately are version control; git repo design; headers and framing. In production, they've been bulletproof. As long as you account for the compile time issues (schema versioning, repo, headers and framing, etc.).
- ardit33 4y ago
  We were using protobufs at Spotify and ditched them for simple JSON calls on the client side. No one complained, and never going back to having anything like that on the client side if I can. Just too many drawbacks. For server to server, they might be fine, but to client just stick with JSON. (which when compressed is pretty efficient).
  - wzvici 4y ago
    Why is it bad in server to client?
    - pphysch 4y ago
      Setting up TLS in gRPC is annoying if the alternative is using an already-existing HTTPS endpoint.
      - anuragsoni 4y ago
        That sounds like a problem with GRPC not protobufs.
      - boardwaalk 4y ago
        How is it annoying? To be fair, we’re fronting our gRPC service with a AWS LB that terminates TLS (so our gRPC is plaintext), so we don’t deal with certs as direct dependencies of our server.
    - cmrdporcupine 4y ago
      Being able to debug through a simple curl or browser devtools is golden. Also browser has JSON parsing built in. Less dependencies. Easier tooling overall. In my experience people overuse protobuf. But I also worked at Google, where it's the hammer in constant search of any nail it can find. At the very least, endpoints should provide the option to provide a JSON form through content representation negotiation.
      - franciscop 4y ago
yorwba 4y ago
Is using flatbuffers as the on-disk storage format for an application a hare-brained idea? If yes, is it a less hare-brained idea than using the ctypes Python module to mmap a file as a C struct? That's what I'm currently doing to get 10x speedup relative to SQLite for an application bottlenecked on disk bandwidth, but it's unergonomic to say the least. Flatbuffers look like a way to get the same performance with better ergonomics, but maybe there's a catch. (E.g. I thought the same thing about Apache Arrow before, but then I realized it's basically read-only. I don't expect to need to resize my tables often, but I do need to be able to twiddle individual values inside the file.)
- nerpderp82 4y ago
  > ctypes Python module to mmap a file as a C struct Tell me more! Is your data larger than memory? You need persistence? You might take a look at Aerospike, even on a single node if you need low latency persistence.
- politician 4y ago
  It really depends on what you're storing and how you need to access the content to meet performance requirements. A simple example of where flatbuffers shines is in TCP flows or in Kafka where each message is a flatbuffer. In Kafka, the message size and type can be included in metadata. In TCP, framing is your responsibility. Serializing a message queue to a flat file is reasonable and natural. Regarding files-as-C-structs: That isn't (necessarily) harebrained (if you can trust the input), Microsoft Word .DOC files were just memory dumps. However, twiddling bits inside a flatbuffer isn't recommended per the documentation; rather, the guidance is to replace the entire buffer. If you don't want to manage the IO yourself, then a key/value store that maps indices to flatbuffers is entirely possible. I'd suggest a look at Redis.
- maximilianburke 4y ago
  I don't think it's hare-brained, I think it'd be great. No less hare-brained than storing stuff to disk in any other format like json or yaml. That said, the ergonomics are absolutely awful for modifying existing objects; you can't modify an existing object, you need to serialize a whole new object. There's also a schemaless version (flexbuffers) which retains a number of the flatbuffers benefits (zero-copy access to data, compact binary representation), but is also a lot easier to use for ad-hoc serialization and deserialization; you can `loads`/`dumps` the flexbuffer objects, for example.

1. Are they easier to use than JSON? 2. Is it just putting strings into a big array? 3. Do you have an examples of them being used?

politician 4y ago
1. It's not easier to use them than JSON when just getting started. However, the pay off is the strong typing and zero-copy access that they offer to folks that need to support clients on multiple architectures. 2. No, writers can directly embed structs and primitive data types into binary buffers at runtime through an API generated from an IDL file. Readers use direct memory access to pull values out of the buffers. If you set it up right, this can result in a massive perf boost by eliminating the encoding and decoding steps. 3. Facebook uses them in their mobile app. Another commenter mentioned use of the in the Arrow format. The flatbuffers website isn't the best, but clearly documents the flatbuffers IDL.
- wzvici 4y ago
  Is there any good writeup about using them that is easy to follow? With drawbacks on using them in prod?
  - politician 4y ago
    The google documentation has a minimal tutorial. There are implementations for all of the major languages. The level of documentation in the ecosystem, though, is poor. My best recommendation for you is to jump in and get the tutorial/hello world example working in a language you're comfortable with. They aren't hard to use, but they aren't the easiest thing either. Once you get the gist of the API through the tutorial, the other important topics that come up immediately are version control; git repo design; headers and framing. In production, they've been bulletproof. As long as you account for the compile time issues (schema versioning, repo, headers and framing, etc.).
ardit33 4y ago
We were using protobufs at Spotify and ditched them for simple JSON calls on the client side. No one complained, and never going back to having anything like that on the client side if I can. Just too many drawbacks. For server to server, they might be fine, but to client just stick with JSON. (which when compressed is pretty efficient).
- wzvici 4y ago
  Why is it bad in server to client?
  - pphysch 4y ago
    Setting up TLS in gRPC is annoying if the alternative is using an already-existing HTTPS endpoint.
    - anuragsoni 4y ago
      That sounds like a problem with GRPC not protobufs.
    - boardwaalk 4y ago
      How is it annoying? To be fair, we’re fronting our gRPC service with a AWS LB that terminates TLS (so our gRPC is plaintext), so we don’t deal with certs as direct dependencies of our server.
  - cmrdporcupine 4y ago
    Being able to debug through a simple curl or browser devtools is golden. Also browser has JSON parsing built in. Less dependencies. Easier tooling overall. In my experience people overuse protobuf. But I also worked at Google, where it's the hammer in constant search of any nail it can find. At the very least, endpoints should provide the option to provide a JSON form through content representation negotiation.
    - franciscop 4y ago

yorwba 4y ago

Is using flatbuffers as the on-disk storage format for an application a hare-brained idea? If yes, is it a less hare-brained idea than using the ctypes Python module to mmap a file as a C struct? That's what I'm currently doing to get 10x speedup relative to SQLite for an application bottlenecked on disk bandwidth, but it's unergonomic to say the least. Flatbuffers look like a way to get the same performance with better ergonomics, but maybe there's a catch. (E.g. I thought the same thing about Apache Arrow before, but then I realized it's basically read-only. I don't expect to need to resize my tables often, but I do need to be able to twiddle individual values inside the file.)

nerpderp82 4y ago
> ctypes Python module to mmap a file as a C struct Tell me more! Is your data larger than memory? You need persistence? You might take a look at Aerospike, even on a single node if you need low latency persistence.
politician 4y ago
It really depends on what you're storing and how you need to access the content to meet performance requirements. A simple example of where flatbuffers shines is in TCP flows or in Kafka where each message is a flatbuffer. In Kafka, the message size and type can be included in metadata. In TCP, framing is your responsibility. Serializing a message queue to a flat file is reasonable and natural. Regarding files-as-C-structs: That isn't (necessarily) harebrained (if you can trust the input), Microsoft Word .DOC files were just memory dumps. However, twiddling bits inside a flatbuffer isn't recommended per the documentation; rather, the guidance is to replace the entire buffer. If you don't want to manage the IO yourself, then a key/value store that maps indices to flatbuffers is entirely possible. I'd suggest a look at Redis.
maximilianburke 4y ago
I don't think it's hare-brained, I think it'd be great. No less hare-brained than storing stuff to disk in any other format like json or yaml. That said, the ergonomics are absolutely awful for modifying existing objects; you can't modify an existing object, you need to serialize a whole new object. There's also a schemaless version (flexbuffers) which retains a number of the flatbuffers benefits (zero-copy access to data, compact binary representation), but is also a lot easier to use for ad-hoc serialization and deserialization; you can `loads`/`dumps` the flexbuffer objects, for example.