14 ms·

FlexBuffers

325 points by excerionsforte 6y ago

sillysaurusx 6y ago
https://google.github.io/flatbuffers/flatbuffers_white_paper.html https://google.github.io/flatbuffers/flatbuffers_white_paper... has the “why”, if anyone is curious.
uluyol 6y ago
So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)
- sophiebits 6y ago
  Sounds like some implementations of MessagePack may be in the same category.
  - nitrogen 6y ago
    MessagePack/msgpack is great for that middle ground where JSON ser/des is too slow, but you don't have enough engineers to justify the maintenance burden of a heavier-weight schemaful protocol. I did end up writing a simple schema verifier for Ruby (ClassyHash, on GitHub) in one of the jobs where I used msgpack, but I no longer have access to maintain it. My benchmarks showed msgpack+classyhash was faster than native JSON (didn't test oj I think) and other serialization formats, and faster than all the other popular Ruby schema validators at the time. Tldr: msgpack rocks, use it instead of JSON for internal services
- cortesoft 6y ago
  Aren't the newer versions of protobuf zero-copy?
  - yablak 6y ago
    Now that absl::Cord has been opensourced, I'd expect a release of protobuf that takes advantage of it. Up to you to use the API though.
  - kentonv 6y ago
    No, not in the way that Cap'n Proto and FlatBuffers are. Protobuf can support zero-copy of strings and byte arrays embedded in the message. Cap'n Proto and FlatBuffers support zero-copy of the entire message structure. (Disclosure: I'm the author of Cap'n Proto and Protobuf v2.)
    - rurban 6y ago
      Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.
      - kentonv 6y ago
        FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special language upfront). FlexBuffers is more like JSON in that all types are dynamic. Personally I'm a strong believer that schemas are highly desirable, but some people argue that schema-less serializations let you get stuff done faster. I think it's very analogous to the argument between type-safe languages vs. dynamically-typed languages. Obviously there are a lot of smart people on both sides of these arguments.
        vlovich123 6y ago
        In theory, you could build something like flexbuffers for cap’n’proto if someone was really motivated, no?
- ChristianBundy 6y ago
  > Any others? Would BIPF fall into this category? It's a serialization format for JSON: https://github.com/dominictarr/bipf https://github.com/dominictarr/bipf
- jeffbee 6y ago
  Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any referenced buffer alive and deletes it when it's not referenced. In any case, it's always been possible to have zero-copy protobuf, you just don't get that from the "Hello, World!" protobuf tutorial.
  - malkia 6y ago
    Indeed, there were several projects that can replace the generator (really, you can come up with something completely else). One of the latest I've seen is https://perfetto.dev/docs/design-docs/protozero https://perfetto.dev/docs/design-docs/protozero
  - kentonv 6y ago
    This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message buffer, rather than having to allocate a copy of the bytes at parse time. Cap'n Proto and FlatBuffers implement a much stronger form of zero-copy. With them, it's not just strings and byte buffers that are zero-copy, it's the entire data structure. With these systems, once you have the bytes of a message mapped into memory, you do not need to do any "parse" step at all before you start using the message. For example, if you have a multi-gigabyte file formatted with one of these, you can mmap() it, and then you can traverse the message tree to read any bytes of the message except the chain of pointers (parent to child) leading to that one datum. Aside from the mmap() call, you can do all this without even allocating any memory at all. That is absolutely not possible with Protobuf, because Protobuf encoding is a list of tag-value pairs each of which has variable width. In order to read any particular value, you must, at the very least, linearly scan through the tag-values until you find the one you want. But in practice, you usually want to read more than one value, at which point the only way to avoid O(n^2) time complexity while keeping things sane is to parse the entire message tree into a different set of in-memory data structures allocated on the heap. That is not "zero-copy" by Cap'n Proto's definition. (Disclosure: I am the author of Cap'n Proto and Protobuf v2.)
- acjohnson55 6y ago
  Where does Apache Arrow fit in? Schema-ful, zero-copy? https://arrow.apache.org https://arrow.apache.org
  - maximilianburke 6y ago
    Arrow would be schema-ful, zero-copy, yeah. Parquet isn't though.
    - filmor 6y ago
      Arrow is based on Flatbuffers, Parquet is based on Thrift.
- throw0101a 6y ago
  So in this/your view, Protobuf and CBOR would be used in different scenarios?
  - ChristianBundy 6y ago
    Yeah, protobuf requires that both parties agree on a schema whereas CBOR is self-describing like XML or JSON.
    - JoshTriplett 6y ago
      CBOR also has a schema format, but yes, it's typically used schemaless.
      - boardwaalk 6y ago
        I hadn't heard of this, so I had to look it up: https://tools.ietf.org/html/rfc8610 https://tools.ietf.org/html/rfc8610
- leakybucket 6y ago
  We're still early in development, but the ZNG format is one we're working on for schema-less, semi-structured, zero-copy operations: https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1-introduction https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1... We currently use this to store security log data, but think it's an interesting midpoint between having no schema at all vs requiring schema registries to do useful work.
- earthboundkid 6y ago
  https://github.com/rentzsch/lich https://github.com/rentzsch/lich is schemaless and potentially zero copy.
- t0nyh0 6y ago
  Apache Avro is also another data serialization format. Schema-Ful, unsure about zero copying or copying. https://avro.apache.org/ https://avro.apache.org/
  - fsaintjacques 6y ago
    Not zero copy, requires deserialization.
- blattimwind 6y ago
  The msgpack parser handles strings zero copy as well. Some tiny JSON parsers for embedded stuff do that as well.
- deadmutex 6y ago
  One thing I wish people knew more about is SBE, it seems to be super fast by design: https://speice.io/2019/09/binary-format-shootout.html https://speice.io/2019/09/binary-format-shootout.html
  - jcul 6y ago
    As far as I know, SBE was designed with financial protocols in mind, though possiblity more order entry (FIX-SBE) than market data. I'm not saying it is the case but it's possible that it's more suited to the IEX market data that the article uses for the performance test than capnproto or flatbuffers. I will consider it in future though, while I'm familiar with SBE it's not one I'd have thought of when thinking about serialisation.
    - dm3 6y ago
      SBE is a zero-copy schema-full serialization format. I don't think there's anything which limits the format to the financial domain. For example, here's [1] a toy example of a schema describing a car. [1] https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-samples/src/main/resources/example-schema.xml https://github.com/real-logic/simple-binary-encoding/blob/ma...
- occamrazor 6y ago
  Apache Arrow is schema-ful, zero-copy (it is both an in-memory and serialization format)
- tannhaeuser 6y ago
  While it's true that XML is a generic serialization of angle-bracket markup that doesn't need a schema for serialization (which was exactly the main motivation for its "invention" as a SGML subset), the reason it's being used in inter-party or long-term/loosely-coupled service payload serialization is because it has fairly powerful and well-established schema languages (XML DTDs and XML Schema) for validation. This is unlike JSON which thrives in ad-hoc serialization for tightly-coupled back-ends and web front-ends (and only in that scenario IMHO). So I don't think XML belongs into the "schema-less, copying" category.
  - randombytes6869 6y ago
    Very true. IMO many people that hate XML config files just haven't used an IDE that validates schema. Its super nice to have auto-complete and property validation on config files, something not offered by JSON or YAML. A good reason to stick with XML for complicated configs. Its one of reasons I don't mind maven. Yeah there's 1000+ line XML config file, but maven DTD is so tight nearly any syntax issue will be flagged. Something easy to appreciate when you're used to giant config files that don't get validated until runtime.
    - Negitivefrags 6y ago
      Visual studio does schema validation for JSON and gives errors inline. There is a big list of supported schemas built in and you can define your own. Never actually looked at the format personally.
      - nojvek 6y ago
        Json schema is what vscode uses. It’s a standard nowadays. Just as effective as XMLSchema. https://jsonschema.net/ https://jsonschema.net/ http://www.schemastore.org/json/ http://www.schemastore.org/json/
    - tannhaeuser 6y ago
      While I generally agree, I'm not sure maven's pom.xml (aka porn.xml), of all things, is a paragon of good markup design ;) For one, maven actively forbids/rejects use of ordinary XML entity references, and invents its own text expansion instead (so strictly speaking pom.xml isn't even XML proper). Then using XML for a relatively simple EAV format seems like overkill. But yeah, over a decade ago the maven developers had great plans to open up the format for alternative serializations/DSLs; thankfully they didn't, if only because they realized it isn't worth the maintenance effort. I'll add that a format for describing software builds is probably the wrong place to let a thousand blossoms bloom, something that you realize soon enough if you've worked as freelancer in Java-ish project's for any amount of time, and where every other project is feeling the urge to use the oh-so-great gradle as an alternative to pom.xml. My comment was only aimed at service payload serialization; as to whether markup makes for a good config file format, I'm not entirely sure. It certainly is better than inventing an ad-hoc format IMO, but OTOH there are a couple of not-quite-SGML/XML config formats such as Apache's httpd.conf, or in fact maven's that just give XML/SGML a bad name IMHO, because they generally inherit the downsides of markup without bringing its benefits (such as being able to assemble a configuration from fragments using regular entity expansion).
- mhd 6y ago
  There's also ASN.1, the dreadful ancient giant of the scene.
- kstenerud 6y ago
  I'm building Concise Encoding, which is schema-less, with a 1:1 compatible text and binary representation. The binary format supports zero copy for string and binary values. The reference implementation (in go) is 90% complete, enough to marshal objects except for recursive support. https://github.com/kstenerud/concise-encoding/blob/master/README.md#concise-encoding https://github.com/kstenerud/concise-encoding/blob/master/RE...
- IshKebab 6y ago
  Some formats optionally support schemas, or bundle the schema with the file. I think Avro does that.
- tankenmate 6y ago
  XDR (schema-ful, zero-copy (if done correctly)), but it's older and even less cool. [EDIT] and I forgot to mention re-entrant as well (again if done correctly).
- deleted 6y ago
  [deleted]
- convolvatron 6y ago
  but i really do wonder why this is such a matter of debate. if performance and specific semantics are an issue...just use the standard tricks and write bytes into a buffer and push it onto the wire. if performance isn't an issue, then just use any of these tools. unless the tooling cost and representational issues make it easier to to just use bytes. all for abstractions..but people seem to to be blind to the idea that there's a perfectly good one a short step down from capn proto
- cbsmith 6y ago
  Except in truth Protobuf, Thrift, Avro, etc. all support doing the same "schemaless" decoding as FlexBuffers.
ccktlmazeltov 6y ago
there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?
- mmastrac 6y ago
  We use FlatBuffers (the parent project) in our code @ FS... that means it has to work in C++, Rust, Java, and TypeScript. It's mostly the schema processing that requires a C++ compiler to make it work. It's a pretty stellar cross-platform serialization toolkit and the zero-copy support for reading is no joke.
  - maximilianburke 6y ago
    We've been using it across multiple languages too as the data serialization format between server and (web) client. It's fantastic.
- jchook 6y ago
  AFAIK some languages can use C++ fairly easily, including D and Rust
  - kccqzy 6y ago
    These languages can use C fairly easily. To use C++, you must wrap the C++ interface in `extern "C"` interfaces. It's very hard to use the full spectrum of C++ features. Consider that you're writing a custom struct/data type called MyStruct in your language. Now, can you use std::vector<MyStruct>? This requires your other language to read the entire vector header and instantiate a new type based on both your definition of your custom type and the vector template. The vector instantiation will ask questions like whether or not your type is copy constructible and move constructible, and if so, use these implementations. Designing this into your other language requires a lot of compromises. And that's just templates. ADL poses an even bigger difficulty. Again when you write your custom type in your language and define a swap function for it, will C++ standard algorithms that need swapping find your implementation via ADL? Recall that swapping is invoked via having `using std::swap;` and then invoking the bare function `swap` which triggers ADL. Again, designing this into your other language requires a lot of compromises. The only language that I know of that comes close to having full interoperability with C++ is Objective-C, and it does so by creating a new Frankenstein language called Objective-C++. It has two different kinds of classes, two different exception mechanisms, etc.
8organicbits 6y ago
> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about what errors you'd want to handle. Unexpectedly getting a zero when you get garbage data may cause issues. Thinking about it, zero is often used as a success code: ERROR_SUCCESS is 0x0 on Windows. So garbage in, ERROR_SUCCESS out?? Something like: value, err = root.AsInt64(); makes it clear you have an error path. You can still ignore `err`, like when you are moving fast and breaking things or it's somehow certain to always succeed, but it's clear that there's an unhandled error path.
- typon 6y ago
  If you use cpp17, you can use the built-in std::optional to solve this problem.
  - taeric 6y ago
    Strictly, not the same. One returns a value and an error. One returns a value or nothing. Having the error can signal both if a value could not be retrieved, or if it could be retrieved, but was coerced, as an example.
- iso8859-1 6y ago
  But look at the encoding, they compact the int down to the amount of bytes it needs for storage. There is no space to encode the type, and there is no way to distinguish encoded uint from an int. You must layer your own schema on top.
  - 8organicbits 6y ago
    I believe this is the docs that explain their type byte. That information is stored: https://google.github.io/flatbuffers/flatbuffers_internals.html https://google.github.io/flatbuffers/flatbuffers_internals.h... > A type byte is made up of 2 components (see flexbuffers.h for exact values): > * 2 lower bits representing the bit-width of the child (8, 16, 32, 64). This is only used if the child is accessed over an offset, such as a child vector. It is ignored for inline types. > * 6 bits representing the actual type (see flexbuffers.h). > Thus, in this example 4 means 8 bit child (value 0, unused, since the value is in-line), type SL_INT (value 1). Type aware methods like `.isInt()` wouldn't make sense if they couldn't determine the underlying type.
armitron 6y ago
I'd rather use MessagePack. Clean, simple, small.
- iso8859-1 6y ago
  It has worse performance though. Not zero-copy.
  - ludocode 6y ago
    It depends what you mean by zero-copy. It can't be traversed without parsing the structure, but strings and binary blobs can be used in-place without copying. (This is also possible with JSON even, for example with RapidJSON's in-situ parser.) This has worse performance for decoding, but far better performance for encoding. Encoding any traversible format is guaranteed to be expensive because you have to encode it inside-out, calculating the nested sizes of everything as you go. For messages that are only encoded once and read once, for example network messages, a format that is traversible in-place is a poor choice.
    - Dylan16807 6y ago
      That's too general. Sometimes there's [almost] nothing to calculate, so no slowdown. Sometimes the traversible format can be used as your working data structure and then encoding is a no-op.
86J8oyZv 6y ago
So, is this basically Android's Bundle semantics made standard and cross-platform? Are there differences implementation-wise?
person_of_color 6y ago
What does zero-copy mean in this context?
- sillysaurusx 6y ago
  Normally, zero-copy refers to the fact that data can be decoded without copying it in memory. I.e. a pointer cast is zero-copy because there's no need to copy the memory elsewhere; that memory can simply be reinterpreted as the target type. I'm not sure if that's the same meaning in this context, however.
Aardappel 6y ago
I made this thing! AMA :)
- excerionsforte 6y ago
  When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ https://serde.rs/, which seems to be a recent addition. I was looking for an alternative to messagepack.
  - Aardappel 6y ago
    "market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.
    - webmaven 6y ago
      Rather than "market", you could "promote" or "evangelize" instead...
    - excerionsforte 6y ago
      Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers. There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)
      - Aardappel 6y ago
        Even that kind of "marketing" takes time :) But thanks for helping out.
- mavam 6y ago
  Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?
m0zg 6y ago
I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.
- vore 6y ago
  It looks like https://github.com/google/riegeli https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
  - m0zg 6y ago
    Far too complicated for what it's supposed to do. Google's RecordIO basically just concatenates messages together, with a few additional provisions for compression and handling small messages. In contrast, Riegeli depends on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md https://github.com/google/riegeli/blob/master/doc/riegeli_re.... This could explain its lack of market penetration (only 221 stars at the time of this writing). The authors apparently don't care that it's not popular, since they don't even provide a code example on how to use it. It also apparently uses HighwayHash for integrity checking, which is overkill - such things should use crc32c, for which most modern CPUs have a dedicated, fast instruction.
  - davidwtbuxton 6y ago
    Also this https://github.com/xeno14/recordio https://github.com/xeno14/recordio
- jeffbee 6y ago
  TensorFlow's TFRecord is RecordIO.
je42 6y ago
I wonder how it compares to messagepack's zero copy serializer: https://blog.treasuredata.com/blog/2011/11/21/messagepack-the-missing-serializer/ https://blog.treasuredata.com/blog/2011/11/21/messagepack-th... ?
vbezhenar 6y ago
Java example is very weird.
secondcoming 6y ago
I'm looking to replace our current Boost.Serialization code with something else. I have the following requirements: - serialisation cannot be intrusive (i.e. I don't want a class definition to be generated from a schema file) - zero copy - some sort of versioning so any accidental message version mismatch between sender and receiver doesn't cause a crash / undefined behviour. - many language support Protobufs are intrusive, so that rules them out. JSON is just too verbose and slow. MessagePack looked like the most promising path to pursue at the moment. Flatbuffers look quite similar.
- awild 6y ago
  My predecessor has employed smile in all our m2m communication. As far as I have understood it, it aims to be a binary representation of json. I cannot speak for its speed but I trust him to a certain extent.

https://google.github.io/flatbuffers/flatbuffers_white_paper.html https://google.github.io/flatbuffers/flatbuffers_white_paper... has the “why”, if anyone is curious.

uluyol 6y ago

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-copy: Cap'n'proto, Flatbuffers Schema-less, copying: Json (binary and other variants included), XML Schema-less, zero-copy: Flexbuffers (Any others? This seems new to me)

sophiebits 6y ago
Sounds like some implementations of MessagePack may be in the same category.
- nitrogen 6y ago
  MessagePack/msgpack is great for that middle ground where JSON ser/des is too slow, but you don't have enough engineers to justify the maintenance burden of a heavier-weight schemaful protocol. I did end up writing a simple schema verifier for Ruby (ClassyHash, on GitHub) in one of the jobs where I used msgpack, but I no longer have access to maintain it. My benchmarks showed msgpack+classyhash was faster than native JSON (didn't test oj I think) and other serialization formats, and faster than all the other popular Ruby schema validators at the time. Tldr: msgpack rocks, use it instead of JSON for internal services
cortesoft 6y ago
Aren't the newer versions of protobuf zero-copy?
- yablak 6y ago
  Now that absl::Cord has been opensourced, I'd expect a release of protobuf that takes advantage of it. Up to you to use the API though.
- kentonv 6y ago
  No, not in the way that Cap'n Proto and FlatBuffers are. Protobuf can support zero-copy of strings and byte arrays embedded in the message. Cap'n Proto and FlatBuffers support zero-copy of the entire message structure. (Disclosure: I'm the author of Cap'n Proto and Protobuf v2.)
  - rurban 6y ago
    Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.
    - kentonv 6y ago
      FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special language upfront). FlexBuffers is more like JSON in that all types are dynamic. Personally I'm a strong believer that schemas are highly desirable, but some people argue that schema-less serializations let you get stuff done faster. I think it's very analogous to the argument between type-safe languages vs. dynamically-typed languages. Obviously there are a lot of smart people on both sides of these arguments.
      - vlovich123 6y ago
        In theory, you could build something like flexbuffers for cap’n’proto if someone was really motivated, no?
ChristianBundy 6y ago
> Any others? Would BIPF fall into this category? It's a serialization format for JSON: https://github.com/dominictarr/bipf https://github.com/dominictarr/bipf
jeffbee 6y ago
Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any referenced buffer alive and deletes it when it's not referenced. In any case, it's always been possible to have zero-copy protobuf, you just don't get that from the "Hello, World!" protobuf tutorial.
- malkia 6y ago
  Indeed, there were several projects that can replace the generator (really, you can come up with something completely else). One of the latest I've seen is https://perfetto.dev/docs/design-docs/protozero https://perfetto.dev/docs/design-docs/protozero
- kentonv 6y ago
  This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message buffer, rather than having to allocate a copy of the bytes at parse time. Cap'n Proto and FlatBuffers implement a much stronger form of zero-copy. With them, it's not just strings and byte buffers that are zero-copy, it's the entire data structure. With these systems, once you have the bytes of a message mapped into memory, you do not need to do any "parse" step at all before you start using the message. For example, if you have a multi-gigabyte file formatted with one of these, you can mmap() it, and then you can traverse the message tree to read any bytes of the message except the chain of pointers (parent to child) leading to that one datum. Aside from the mmap() call, you can do all this without even allocating any memory at all. That is absolutely not possible with Protobuf, because Protobuf encoding is a list of tag-value pairs each of which has variable width. In order to read any particular value, you must, at the very least, linearly scan through the tag-values until you find the one you want. But in practice, you usually want to read more than one value, at which point the only way to avoid O(n^2) time complexity while keeping things sane is to parse the entire message tree into a different set of in-memory data structures allocated on the heap. That is not "zero-copy" by Cap'n Proto's definition. (Disclosure: I am the author of Cap'n Proto and Protobuf v2.)
acjohnson55 6y ago
Where does Apache Arrow fit in? Schema-ful, zero-copy? https://arrow.apache.org https://arrow.apache.org
- maximilianburke 6y ago
  Arrow would be schema-ful, zero-copy, yeah. Parquet isn't though.
  - filmor 6y ago
    Arrow is based on Flatbuffers, Parquet is based on Thrift.
throw0101a 6y ago
So in this/your view, Protobuf and CBOR would be used in different scenarios?
- ChristianBundy 6y ago
  Yeah, protobuf requires that both parties agree on a schema whereas CBOR is self-describing like XML or JSON.
  - JoshTriplett 6y ago
    CBOR also has a schema format, but yes, it's typically used schemaless.
    - boardwaalk 6y ago
      I hadn't heard of this, so I had to look it up: https://tools.ietf.org/html/rfc8610 https://tools.ietf.org/html/rfc8610
leakybucket 6y ago
We're still early in development, but the ZNG format is one we're working on for schema-less, semi-structured, zero-copy operations: https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1-introduction https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1... We currently use this to store security log data, but think it's an interesting midpoint between having no schema at all vs requiring schema registries to do useful work.
earthboundkid 6y ago
https://github.com/rentzsch/lich https://github.com/rentzsch/lich is schemaless and potentially zero copy.
t0nyh0 6y ago
Apache Avro is also another data serialization format. Schema-Ful, unsure about zero copying or copying. https://avro.apache.org/ https://avro.apache.org/
- fsaintjacques 6y ago
  Not zero copy, requires deserialization.
blattimwind 6y ago
The msgpack parser handles strings zero copy as well. Some tiny JSON parsers for embedded stuff do that as well.
deadmutex 6y ago
One thing I wish people knew more about is SBE, it seems to be super fast by design: https://speice.io/2019/09/binary-format-shootout.html https://speice.io/2019/09/binary-format-shootout.html
- jcul 6y ago
  As far as I know, SBE was designed with financial protocols in mind, though possiblity more order entry (FIX-SBE) than market data. I'm not saying it is the case but it's possible that it's more suited to the IEX market data that the article uses for the performance test than capnproto or flatbuffers. I will consider it in future though, while I'm familiar with SBE it's not one I'd have thought of when thinking about serialisation.
  - dm3 6y ago
    SBE is a zero-copy schema-full serialization format. I don't think there's anything which limits the format to the financial domain. For example, here's [1] a toy example of a schema describing a car. [1] https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-samples/src/main/resources/example-schema.xml https://github.com/real-logic/simple-binary-encoding/blob/ma...
occamrazor 6y ago
Apache Arrow is schema-ful, zero-copy (it is both an in-memory and serialization format)
tannhaeuser 6y ago
While it's true that XML is a generic serialization of angle-bracket markup that doesn't need a schema for serialization (which was exactly the main motivation for its "invention" as a SGML subset), the reason it's being used in inter-party or long-term/loosely-coupled service payload serialization is because it has fairly powerful and well-established schema languages (XML DTDs and XML Schema) for validation. This is unlike JSON which thrives in ad-hoc serialization for tightly-coupled back-ends and web front-ends (and only in that scenario IMHO). So I don't think XML belongs into the "schema-less, copying" category.
- randombytes6869 6y ago
  Very true. IMO many people that hate XML config files just haven't used an IDE that validates schema. Its super nice to have auto-complete and property validation on config files, something not offered by JSON or YAML. A good reason to stick with XML for complicated configs. Its one of reasons I don't mind maven. Yeah there's 1000+ line XML config file, but maven DTD is so tight nearly any syntax issue will be flagged. Something easy to appreciate when you're used to giant config files that don't get validated until runtime.
  - Negitivefrags 6y ago
    Visual studio does schema validation for JSON and gives errors inline. There is a big list of supported schemas built in and you can define your own. Never actually looked at the format personally.
    - nojvek 6y ago
      Json schema is what vscode uses. It’s a standard nowadays. Just as effective as XMLSchema. https://jsonschema.net/ https://jsonschema.net/ http://www.schemastore.org/json/ http://www.schemastore.org/json/
  - tannhaeuser 6y ago
    While I generally agree, I'm not sure maven's pom.xml (aka porn.xml), of all things, is a paragon of good markup design ;) For one, maven actively forbids/rejects use of ordinary XML entity references, and invents its own text expansion instead (so strictly speaking pom.xml isn't even XML proper). Then using XML for a relatively simple EAV format seems like overkill. But yeah, over a decade ago the maven developers had great plans to open up the format for alternative serializations/DSLs; thankfully they didn't, if only because they realized it isn't worth the maintenance effort. I'll add that a format for describing software builds is probably the wrong place to let a thousand blossoms bloom, something that you realize soon enough if you've worked as freelancer in Java-ish project's for any amount of time, and where every other project is feeling the urge to use the oh-so-great gradle as an alternative to pom.xml. My comment was only aimed at service payload serialization; as to whether markup makes for a good config file format, I'm not entirely sure. It certainly is better than inventing an ad-hoc format IMO, but OTOH there are a couple of not-quite-SGML/XML config formats such as Apache's httpd.conf, or in fact maven's that just give XML/SGML a bad name IMHO, because they generally inherit the downsides of markup without bringing its benefits (such as being able to assemble a configuration from fragments using regular entity expansion).
mhd 6y ago
There's also ASN.1, the dreadful ancient giant of the scene.
kstenerud 6y ago
I'm building Concise Encoding, which is schema-less, with a 1:1 compatible text and binary representation. The binary format supports zero copy for string and binary values. The reference implementation (in go) is 90% complete, enough to marshal objects except for recursive support. https://github.com/kstenerud/concise-encoding/blob/master/README.md#concise-encoding https://github.com/kstenerud/concise-encoding/blob/master/RE...
IshKebab 6y ago
Some formats optionally support schemas, or bundle the schema with the file. I think Avro does that.
tankenmate 6y ago
XDR (schema-ful, zero-copy (if done correctly)), but it's older and even less cool. [EDIT] and I forgot to mention re-entrant as well (again if done correctly).
deleted 6y ago
[deleted]
convolvatron 6y ago
but i really do wonder why this is such a matter of debate. if performance and specific semantics are an issue...just use the standard tricks and write bytes into a buffer and push it onto the wire. if performance isn't an issue, then just use any of these tools. unless the tooling cost and representational issues make it easier to to just use bytes. all for abstractions..but people seem to to be blind to the idea that there's a perfectly good one a short step down from capn proto
cbsmith 6y ago
Except in truth Protobuf, Thrift, Avro, etc. all support doing the same "schemaless" decoding as FlexBuffers.

ccktlmazeltov 6y ago

there's a lot of C++ around this FlexBuffer thing so I'm not sure how relevant it is outside of C++. Any idea?

mmastrac 6y ago
We use FlatBuffers (the parent project) in our code @ FS... that means it has to work in C++, Rust, Java, and TypeScript. It's mostly the schema processing that requires a C++ compiler to make it work. It's a pretty stellar cross-platform serialization toolkit and the zero-copy support for reading is no joke.
- maximilianburke 6y ago
  We've been using it across multiple languages too as the data serialization format between server and (web) client. It's fantastic.
jchook 6y ago
AFAIK some languages can use C++ fairly easily, including D and Rust
- kccqzy 6y ago
  These languages can use C fairly easily. To use C++, you must wrap the C++ interface in `extern "C"` interfaces. It's very hard to use the full spectrum of C++ features. Consider that you're writing a custom struct/data type called MyStruct in your language. Now, can you use std::vector<MyStruct>? This requires your other language to read the entire vector header and instantiate a new type based on both your definition of your custom type and the vector template. The vector instantiation will ask questions like whether or not your type is copy constructible and move constructible, and if so, use these implementations. Designing this into your other language requires a lot of compromises. And that's just templates. ADL poses an even bigger difficulty. Again when you write your custom type in your language and define a swap function for it, will C++ standard algorithms that need swapping find your implementation via ADL? Recall that swapping is invoked via having `using std::swap;` and then invoking the bare function `swap` which triggers ADL. Again, designing this into your other language requires a lot of compromises. The only language that I know of that comes close to having full interoperability with C++ is Objective-C, and it does so by creating a new Frankenstein language called Objective-C++. It has two different kinds of classes, two different exception mechanisms, etc.

8organicbits 6y ago

> if you supply a buffer that actually contains a float, or a string with numbers in it, it will convert it for you on the fly as well, or return 0 if it can't. If instead you actually want to know what is inside the buffer before you access it, you can call root.GetType() or root.IsInt() etc. I've started to prefer functions that are explicit about their error cases and have interfaces that make it obvious about what errors you'd want to handle. Unexpectedly getting a zero when you get garbage data may cause issues. Thinking about it, zero is often used as a success code: ERROR_SUCCESS is 0x0 on Windows. So garbage in, ERROR_SUCCESS out?? Something like: value, err = root.AsInt64(); makes it clear you have an error path. You can still ignore `err`, like when you are moving fast and breaking things or it's somehow certain to always succeed, but it's clear that there's an unhandled error path.

typon 6y ago
If you use cpp17, you can use the built-in std::optional to solve this problem.
- taeric 6y ago
  Strictly, not the same. One returns a value and an error. One returns a value or nothing. Having the error can signal both if a value could not be retrieved, or if it could be retrieved, but was coerced, as an example.
iso8859-1 6y ago
But look at the encoding, they compact the int down to the amount of bytes it needs for storage. There is no space to encode the type, and there is no way to distinguish encoded uint from an int. You must layer your own schema on top.
- 8organicbits 6y ago
  I believe this is the docs that explain their type byte. That information is stored: https://google.github.io/flatbuffers/flatbuffers_internals.html https://google.github.io/flatbuffers/flatbuffers_internals.h... > A type byte is made up of 2 components (see flexbuffers.h for exact values): > * 2 lower bits representing the bit-width of the child (8, 16, 32, 64). This is only used if the child is accessed over an offset, such as a child vector. It is ignored for inline types. > * 6 bits representing the actual type (see flexbuffers.h). > Thus, in this example 4 means 8 bit child (value 0, unused, since the value is in-line), type SL_INT (value 1). Type aware methods like `.isInt()` wouldn't make sense if they couldn't determine the underlying type.

armitron 6y ago

I'd rather use MessagePack. Clean, simple, small.

iso8859-1 6y ago
It has worse performance though. Not zero-copy.
- ludocode 6y ago
  It depends what you mean by zero-copy. It can't be traversed without parsing the structure, but strings and binary blobs can be used in-place without copying. (This is also possible with JSON even, for example with RapidJSON's in-situ parser.) This has worse performance for decoding, but far better performance for encoding. Encoding any traversible format is guaranteed to be expensive because you have to encode it inside-out, calculating the nested sizes of everything as you go. For messages that are only encoded once and read once, for example network messages, a format that is traversible in-place is a poor choice.
  - Dylan16807 6y ago
    That's too general. Sometimes there's [almost] nothing to calculate, so no slowdown. Sometimes the traversible format can be used as your working data structure and then encoding is a no-op.

86J8oyZv 6y ago

So, is this basically Android's Bundle semantics made standard and cross-platform? Are there differences implementation-wise?

person_of_color 6y ago

What does zero-copy mean in this context?

sillysaurusx 6y ago
Normally, zero-copy refers to the fact that data can be decoded without copying it in memory. I.e. a pointer cast is zero-copy because there's no need to copy the memory elsewhere; that memory can simply be reinterpreted as the target type. I'm not sure if that's the same meaning in this context, however.

Aardappel 6y ago

I made this thing! AMA :)

excerionsforte 6y ago
When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ https://serde.rs/, which seems to be a recent addition. I was looking for an alternative to messagepack.
- Aardappel 6y ago
  "market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.
  - webmaven 6y ago
    Rather than "market", you could "promote" or "evangelize" instead...
  - excerionsforte 6y ago
    Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers. There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)
    - Aardappel 6y ago
      Even that kind of "marketing" takes time :) But thanks for helping out.
mavam 6y ago
Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?

m0zg 6y ago

I wish Google open sourced RecordIO instead (or in addition). People reinvent this particular bicycle, poorly, pretty much in every project where engineers are smart enough to introduce a _structured_ application log.

vore 6y ago
It looks like https://github.com/google/riegeli https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
- m0zg 6y ago
  Far too complicated for what it's supposed to do. Google's RecordIO basically just concatenates messages together, with a few additional provisions for compression and handling small messages. In contrast, Riegeli depends on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_records_file_format.md https://github.com/google/riegeli/blob/master/doc/riegeli_re.... This could explain its lack of market penetration (only 221 stars at the time of this writing). The authors apparently don't care that it's not popular, since they don't even provide a code example on how to use it. It also apparently uses HighwayHash for integrity checking, which is overkill - such things should use crc32c, for which most modern CPUs have a dedicated, fast instruction.
- davidwtbuxton 6y ago
  Also this https://github.com/xeno14/recordio https://github.com/xeno14/recordio
jeffbee 6y ago
TensorFlow's TFRecord is RecordIO.

je42 6y ago

I wonder how it compares to messagepack's zero copy serializer: https://blog.treasuredata.com/blog/2011/11/21/messagepack-the-missing-serializer/ https://blog.treasuredata.com/blog/2011/11/21/messagepack-th... ?

vbezhenar 6y ago

Java example is very weird.

secondcoming 6y ago

I'm looking to replace our current Boost.Serialization code with something else. I have the following requirements: - serialisation cannot be intrusive (i.e. I don't want a class definition to be generated from a schema file) - zero copy - some sort of versioning so any accidental message version mismatch between sender and receiver doesn't cause a crash / undefined behviour. - many language support Protobufs are intrusive, so that rules them out. JSON is just too verbose and slow. MessagePack looked like the most promising path to pursue at the moment. Flatbuffers look quite similar.

awild 6y ago
My predecessor has employed smile in all our m2m communication. As far as I have understood it, it aims to be a binary representation of json. I cannot speak for its speed but I trust him to a certain extent.