10 ms·

So here's how I think this fits into all the other different types of data serialization: Schema-ful, copying: Protobuf, Thrift, plenty more Schema-ful, zero-

• sophiebits 6y ago
  Sounds like some implementations of MessagePack may be in the same category.
  • nitrogen 6y ago
    MessagePack/msgpack is great for that middle ground where JSON ser/des is too slow, but you don't have enough engineers to justify the maintenance burden of a heavier-weight schemaful protocol. I did end up writing a simple schema verifier for Ruby (ClassyHash, on GitHub) in one of the jobs where I used msgpack, but I no longer have access to maintain it. My benchmarks showed msgpack+classyhash was faster than native JSON (didn't test oj I think) and other serialization formats, and faster than all the other popular Ruby schema validators at the time. Tldr: msgpack rocks, use it instead of JSON for internal services
• cortesoft 6y ago
  Aren't the newer versions of protobuf zero-copy?
  • yablak 6y ago
    Now that absl::Cord has been opensourced, I'd expect a release of protobuf that takes advantage of it. Up to you to use the API though.
  • kentonv 6y ago
    No, not in the way that Cap'n Proto and FlatBuffers are. Protobuf can support zero-copy of strings and byte arrays embedded in the message. Cap'n Proto and FlatBuffers support zero-copy of the entire message structure. (Disclosure: I'm the author of Cap'n Proto and Protobuf v2.)
    • rurban 6y ago
      Kenton, so what are the main difference to your Cap'n Proto? My guess are that Flatbuffers are more flexible and slower to read. And 5 years to late to the party.
      • kentonv 6y ago
        FlatBuffers has been around almost as long as Cap'n Proto. I wrote this comparison back in 2014, but it may be outdated now: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... This HN article, though, is about FlexBuffers. FlexBuffers appears to be based on FlatBuffers, but does not use schemas. Cap'n Proto, FlatBuffers, and Protobuf are all schema-driven (you must define your message types in a special language upfront). FlexBuffers is more like JSON in that all types are dynamic. Personally I'm a strong believer that schemas are highly desirable, but some people argue that schema-less serializations let you get stuff done faster. I think it's very analogous to the argument between type-safe languages vs. dynamically-typed languages. Obviously there are a lot of smart people on both sides of these arguments.
        • vlovich123 6y ago
          In theory, you could build something like flexbuffers for cap’n’proto if someone was really motivated, no?
          • kentonv 6y ago
            I suppose you could layer something on top. But I think it would be tricky to come up with something with satisfying performance properties. A naive encoding of JSON into Cap'n Proto would result in messages that are much larger than JSON messages, because of all the pointers and padding and text field names. I haven't looked into exactly how FlexBuffers work but off the top of my head I suspect that leveraging FlatBuffer's "virtual table" technique probably helps here. In (normal, schema-ful) Cap'n Proto, fields within a struct have fixed offset, meaning that unused fields still take space. As I understand it, FlatBuffers tries to avoid this by adding an extra layer of indirection -- each struct has a sort of "virtual table" which stores the offsets of each field, where some fields might not be present at all. If multiple structs in a message happen to end up with the same virtual table, then the virtual table is only written once. Totally speculating here since, again, I haven't actually looked at FlexBuffers, but if I were building something isomorphic to JSON on top of FlatBuffers, I'd probably look into extending the virtual tables to index fields by name rather than number. So if you have two structures with the same set of field names, they can share a virtual table, and those field names only have to appear once. That'd be a pretty great way to compress JSON. Back in Cap'n Proto, we don't have these vtables. For data with fixed schemas, my opinion is that these vtables seem like they require more bookkeeping than they are worth. But for dynamic schemas they seem like a much bigger win. So if you wanted to encode dynamic schemas layered on top of Cap'n Proto, you'd probably have to come up with some similar vtable thing yourself.
• ChristianBundy 6y ago
  > Any others? Would BIPF fall into this category? It's a serialization format for JSON: https://github.com/dominictarr/bipf https://github.com/dominictarr/bipf
• jeffbee 6y ago
  Copying is more a facet of the implementation than the architecture, and relates strongly to the language and runtime. There's no reason that protobuf needs to copy. The only reason most C++ protobuf libraries copy is because ownership in C++ is hard and that makes zero-copy hard to use safely. By contrast it's easier to write a protobuf codec in Go that just aliases everything, because the Go runtime keeps any referenced buffer alive and deletes it when it's not referenced. In any case, it's always been possible to have zero-copy protobuf, you just don't get that from the "Hello, World!" protobuf tutorial.
  • malkia 6y ago
    Indeed, there were several projects that can replace the generator (really, you can come up with something completely else). One of the latest I've seen is https://perfetto.dev/docs/design-docs/protozero https://perfetto.dev/docs/design-docs/protozero
  • kentonv 6y ago
    This is incorrect -- it is not possible to implement Protobuf in a way that achieves the notion of "zero-copy" that Cap'n Proto and FlatBuffers achieve. This probably comes down to a disagreement on what "zero-copy" means. Some people use the term "zero-copy" to mean only that when the message contains a string or byte array, the parsed representation of those specific fields will point back into the original message buffer, rather than having to allocate a copy of the bytes at parse time. Cap'n Proto and FlatBuffers implement a much stronger form of zero-copy. With them, it's not just strings and byte buffers that are zero-copy, it's the entire data structure. With these systems, once you have the bytes of a message mapped into memory, you do not need to do any "parse" step at all before you start using the message. For example, if you have a multi-gigabyte file formatted with one of these, you can mmap() it, and then you can traverse the message tree to read any bytes of the message except the chain of pointers (parent to child) leading to that one datum. Aside from the mmap() call, you can do all this without even allocating any memory at all. That is absolutely not possible with Protobuf, because Protobuf encoding is a list of tag-value pairs each of which has variable width. In order to read any particular value, you must, at the very least, linearly scan through the tag-values until you find the one you want. But in practice, you usually want to read more than one value, at which point the only way to avoid O(n^2) time complexity while keeping things sane is to parse the entire message tree into a different set of in-memory data structures allocated on the heap. That is not "zero-copy" by Cap'n Proto's definition. (Disclosure: I am the author of Cap'n Proto and Protobuf v2.)
• acjohnson55 6y ago
  Where does Apache Arrow fit in? Schema-ful, zero-copy? https://arrow.apache.org https://arrow.apache.org
  • maximilianburke 6y ago
    Arrow would be schema-ful, zero-copy, yeah. Parquet isn't though.
    • filmor 6y ago
      Arrow is based on Flatbuffers, Parquet is based on Thrift.
• throw0101a 6y ago
  So in this/your view, Protobuf and CBOR would be used in different scenarios?
  • ChristianBundy 6y ago
    Yeah, protobuf requires that both parties agree on a schema whereas CBOR is self-describing like XML or JSON.
    • JoshTriplett 6y ago
      CBOR also has a schema format, but yes, it's typically used schemaless.
      • boardwaalk 6y ago
        I hadn't heard of this, so I had to look it up: https://tools.ietf.org/html/rfc8610 https://tools.ietf.org/html/rfc8610
• leakybucket 6y ago
  We're still early in development, but the ZNG format is one we're working on for schema-less, semi-structured, zero-copy operations: https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1-introduction https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1... We currently use this to store security log data, but think it's an interesting midpoint between having no schema at all vs requiring schema registries to do useful work.
• earthboundkid 6y ago
  https://github.com/rentzsch/lich https://github.com/rentzsch/lich is schemaless and potentially zero copy.
• t0nyh0 6y ago
  Apache Avro is also another data serialization format. Schema-Ful, unsure about zero copying or copying. https://avro.apache.org/ https://avro.apache.org/
  • fsaintjacques 6y ago
    Not zero copy, requires deserialization.
• blattimwind 6y ago
  The msgpack parser handles strings zero copy as well. Some tiny JSON parsers for embedded stuff do that as well.
• deadmutex 6y ago
  One thing I wish people knew more about is SBE, it seems to be super fast by design: https://speice.io/2019/09/binary-format-shootout.html https://speice.io/2019/09/binary-format-shootout.html
  • jcul 6y ago
    As far as I know, SBE was designed with financial protocols in mind, though possiblity more order entry (FIX-SBE) than market data. I'm not saying it is the case but it's possible that it's more suited to the IEX market data that the article uses for the performance test than capnproto or flatbuffers. I will consider it in future though, while I'm familiar with SBE it's not one I'd have thought of when thinking about serialisation.
    • dm3 6y ago
      SBE is a zero-copy schema-full serialization format. I don't think there's anything which limits the format to the financial domain. For example, here's [1] a toy example of a schema describing a car. [1] https://github.com/real-logic/simple-binary-encoding/blob/master/sbe-samples/src/main/resources/example-schema.xml https://github.com/real-logic/simple-binary-encoding/blob/ma...
• occamrazor 6y ago
  Apache Arrow is schema-ful, zero-copy (it is both an in-memory and serialization format)
• tannhaeuser 6y ago
  While it's true that XML is a generic serialization of angle-bracket markup that doesn't need a schema for serialization (which was exactly the main motivation for its "invention" as a SGML subset), the reason it's being used in inter-party or long-term/loosely-coupled service payload serialization is because it has fairly powerful and well-established schema languages (XML DTDs and XML Schema) for validation. This is unlike JSON which thrives in ad-hoc serialization for tightly-coupled back-ends and web front-ends (and only in that scenario IMHO). So I don't think XML belongs into the "schema-less, copying" category.
  • randombytes6869 6y ago
    Very true. IMO many people that hate XML config files just haven't used an IDE that validates schema. Its super nice to have auto-complete and property validation on config files, something not offered by JSON or YAML. A good reason to stick with XML for complicated configs. Its one of reasons I don't mind maven. Yeah there's 1000+ line XML config file, but maven DTD is so tight nearly any syntax issue will be flagged. Something easy to appreciate when you're used to giant config files that don't get validated until runtime.
    • Negitivefrags 6y ago
      Visual studio does schema validation for JSON and gives errors inline. There is a big list of supported schemas built in and you can define your own. Never actually looked at the format personally.
      • nojvek 6y ago
        Json schema is what vscode uses. It’s a standard nowadays. Just as effective as XMLSchema. https://jsonschema.net/ https://jsonschema.net/ http://www.schemastore.org/json/ http://www.schemastore.org/json/
    • tannhaeuser 6y ago
      While I generally agree, I'm not sure maven's pom.xml (aka porn.xml), of all things, is a paragon of good markup design ;) For one, maven actively forbids/rejects use of ordinary XML entity references, and invents its own text expansion instead (so strictly speaking pom.xml isn't even XML proper). Then using XML for a relatively simple EAV format seems like overkill. But yeah, over a decade ago the maven developers had great plans to open up the format for alternative serializations/DSLs; thankfully they didn't, if only because they realized it isn't worth the maintenance effort. I'll add that a format for describing software builds is probably the wrong place to let a thousand blossoms bloom, something that you realize soon enough if you've worked as freelancer in Java-ish project's for any amount of time, and where every other project is feeling the urge to use the oh-so-great gradle as an alternative to pom.xml. My comment was only aimed at service payload serialization; as to whether markup makes for a good config file format, I'm not entirely sure. It certainly is better than inventing an ad-hoc format IMO, but OTOH there are a couple of not-quite-SGML/XML config formats such as Apache's httpd.conf, or in fact maven's that just give XML/SGML a bad name IMHO, because they generally inherit the downsides of markup without bringing its benefits (such as being able to assemble a configuration from fragments using regular entity expansion).
• mhd 6y ago
  There's also ASN.1, the dreadful ancient giant of the scene.
• kstenerud 6y ago
  I'm building Concise Encoding, which is schema-less, with a 1:1 compatible text and binary representation. The binary format supports zero copy for string and binary values. The reference implementation (in go) is 90% complete, enough to marshal objects except for recursive support. https://github.com/kstenerud/concise-encoding/blob/master/README.md#concise-encoding https://github.com/kstenerud/concise-encoding/blob/master/RE...
• IshKebab 6y ago
  Some formats optionally support schemas, or bundle the schema with the file. I think Avro does that.
• tankenmate 6y ago
  XDR (schema-ful, zero-copy (if done correctly)), but it's older and even less cool. [EDIT] and I forgot to mention re-entrant as well (again if done correctly).
• deleted 6y ago
  [deleted]
• convolvatron 6y ago
  but i really do wonder why this is such a matter of debate. if performance and specific semantics are an issue...just use the standard tricks and write bytes into a buffer and push it onto the wire. if performance isn't an issue, then just use any of these tools. unless the tooling cost and representational issues make it easier to to just use bytes. all for abstractions..but people seem to to be blind to the idea that there's a perfectly good one a short step down from capn proto
• cbsmith 6y ago
  Except in truth Protobuf, Thrift, Avro, etc. all support doing the same "schemaless" decoding as FlexBuffers.