3 ms·

Do you know if flatbuffers can be used as data structures in applications? That seems to be a shortcoming of protobuf, all the ser/de code makes it suboptimal f

by kortex 7y ago

Do you know if flatbuffers can be used as data structures in applications? That seems to be a shortcoming of protobuf, all the ser/de code makes it suboptimal for in-app data transport.

kentonv 7y ago
Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. E.g. in Cap'n Proto, all objects in a message are allocated sequentially within the larger message buffer. If you remove an object or change its size (e.g. overwrite a string field with a new value of different length), the new value needs to be allocated on the end and the memory space for the old value cannot be reused -- it is wasted. I'm not super-familiar with FlatBuffers, but I believe it uses a model where messages must strictly be constructed in bottom-up order, such that all pointers point in the same direction. This seems to imply that you can't modify a message at all after construction, but I haven't actually played with it so I could be mistaken. (I'm the author of Cap'n Proto.)
- kortex 7y ago
  > Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. Oh, I never thought of it that way, but that makes perfect sense. I guess I just assumed messages would be collections of pointers and buffers living "somewhere" in memory, but of course the actual layout can make a ton of difference. I guess there is an implicit rule that if you are dealing with inbound structures in a read-only fashion, passing around the serial structure is OK for when the field access cost is minor compared to the copy cost, but if you want to mutate it, or doing lots of access operations where that isn't trivial, it makes sense to copy into your own data structure.

by kortex 7y ago

Do you know if flatbuffers can be used as data structures in applications? That seems to be a shortcoming of protobuf, all the ser/de code makes it suboptimal for in-app data transport.

kentonv 7y ago

Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. E.g. in Cap'n Proto, all objects in a message are allocated sequentially within the larger message buffer. If you remove an object or change its size (e.g. overwrite a string field with a new value of different length), the new value needs to be allocated on the end and the memory space for the old value cannot be reused -- it is wasted. I'm not super-familiar with FlatBuffers, but I believe it uses a model where messages must strictly be constructed in bottom-up order, such that all pointers point in the same direction. This seems to imply that you can't modify a message at all after construction, but I haven't actually played with it so I could be mistaken. (I'm the author of Cap'n Proto.)

kortex 7y ago
> Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. Oh, I never thought of it that way, but that makes perfect sense. I guess I just assumed messages would be collections of pointers and buffers living "somewhere" in memory, but of course the actual layout can make a ton of difference. I guess there is an implicit rule that if you are dealing with inbound structures in a read-only fashion, passing around the serial structure is OK for when the field access cost is minor compared to the copy cost, but if you want to mutate it, or doing lots of access operations where that isn't trivial, it makes sense to copy into your own data structure.