7 ms·

This is the wrong argument. Who cares about the type system of a binary packaging format? The joy is how these messages can be used as rows in storage systems a

This is the wrong argument. Who cares about the type system of a binary packaging format? The joy is how these messages can be used as rows in storage systems as well as RPC. Complicating the type system limits the domain applicability and increases the porting cost. No. Protobuffers are shit coz they don't support zero copy and you have to deserialize the whole thing even if you are interested in one field or an outer envelope, causing memory churn in your JVMs. Cap'n'proto and flat buffers attack this real problem. The expressivity of the type system is a minor issue, hence no credible competition. Note grpc abandoned required fields! Nothing is required over a decade, backward compatibility is important! Required should be enforced at application later not the binary packing layer. It is a property of the version of the code processing the blob, not the blob representation itself. Ex googler and equally happy openAPI spec user.

doublement 7y ago
This is going to sound sarcastic but it's not: Can we get back to just putting the members of C structures into network byte order and sending that over the wire in binary, à la 1995?
- mikece 7y ago
  Is that less of a configuration mess than WCF was? JSON isn't "The Magical Elixir" of data exchange and I'm more than open to something better but at least we (in the .NET community) have moved past the WCF configuration nightmares.
  - bob1029 7y ago
    WCF is an unmitigated dumpster fire. We have actually written a non-WCF client that uses a raw HttpClient implementation with StringBuilder to compose SOAP envelopes around cached XMLSerializers in order to talk to other WCF services. First request delay went from 1-2 seconds down to a few milliseconds. Memory overhead is negligible now. Prior, you could watch task manager and immediately recognize when WCF is "warming up". Additionally, the XML serializer in .NET seems almost pathologically determined to ruin everything you seek to accomplish. By comparison, JSON contracts are an absolute joy to work with. We still practice strong-typing on both sides of the wire (we control both ends), and have pretty much nothing to complain about. If you are concerned with space overhead w/ JSON, simply passing it through gzip can get you down to a very reasonable place for 99% of use cases. I understand that there are arguments to be made against JSON for extremely performance sensitive applications, but I would counter-argue that these are extremely rare in practice.
    - heavenlyblue 7y ago
      Isn’t the problem with WCF rather than XML?
- btown 7y ago
  This is more or less what https://capnproto.org/ https://capnproto.org/ does.
  - omginternets 7y ago
    From the capnproto docs: >Isn’t this all horribly insecure? >No no no! To be clear, we’re NOT just casting a buffer pointer to a struct pointer and calling it a day. Isn't this a direct contradiction to your claim? Or have I misunderstood them?
    - deleted 7y ago
      [deleted]
    - lidHanteyk 7y ago
      Capn is better than C at struct layout. We are not, under any circumstances, going back to the 90s. We are moving forward and learning from mistakes.
    - jtolmar 7y ago
      IIRC: capnproto generates messages that you could deserialize by casting them to the right struct, but refrains from actually doing it that way. Instead it generates a bunch of accessor methods that parse the data, as if you were reading something that's not basically a c-struct, like a protobuff.
      - kentonv 7y ago
        That's basically correct. Cap'n Proto generates classes with inline accessor methods that do roughly the same pointer arithmetic that the compiler would generate for struct access. There's a couple subtle differences: * The struct is allowed to be shorter than expected, in which case fields past the end are assumed to have their schema-defined default values. This is what allows you to add new fields over time while remaining forwards- and backwards-compatible. * Pointers are in a non-native format. They are offset-based (rather than absolute) and contain some extra type information (such as the size of the target, needed for the previous point). Following a pointer requires validating it for security. (Disclosure: I'm the author of Cap'n Proto.)
- izacus 7y ago
  This is practically exactly what Protobuffers are. Except that they actually are defined clearly enough for multiple services written in multiple languages can work with them.
  - cma 7y ago
    I thought that's what capt'n proto was, not protobuffers.
  - CoolGuySteve 7y ago
    Definitely not, protobuf's strange wire format becomes apparent if you ever look at the hexdump of one or the profiler output of your favourite protobuffer-decoding C/C++ application. They're actually kind of performance heavy for no benefit.
    - imtringued 7y ago
      I have once looked at a benchmark that compared protobuffer, message pack, json and a variety of other serialization formats. In terms of reducing bytes per message gzipped json was ahead of all of them at the cost of increased CPU time for gzip. Protobuffer did pretty poorly, the only benefit was decreased CPU usage. I'm sure you could use some other compression algorithm like LZMA to get both good compression and good performance for JSON messages.
      - CoolGuySteve 7y ago
        I use LZ4 (with "best" compression) for packet captures and replay with great results. I get about a 37% compression ratio with extremely fast decoding, like 10 million packets per second off an SSD. It was better than snappy, gzip, and bz2 for the trade-off of compression time, decompression time and file size. As for protobuf: flatbuffers, capn proto, HDF5, and plain C structs all deliver much, much faster decoding time. It's really not the best answer for any serialization at this point but it's still inexplicably popular.
      - touisteur 7y ago
- amacbride 7y ago
  XDR FTW!
- pantalaimon 7y ago
  This gets hairy the moment you want to add new fields.
  - CoolGuySteve 7y ago
    Both protobuf and plain C structs are append only formats if you put the message-type and size at the start of the C-struct.
    - marcan_42 7y ago
      C structs do not compose extensively. Protobufs do. You can't put variable length data into a struct, and hence you can't put extensible structs into it either.
      - doublement 7y ago
        You can put variable length data into a struct: https://en.wikipedia.org/wiki/Flexible_array_member https://en.wikipedia.org/wiki/Flexible_array_member
        xyzzyz 7y ago
        Only one field can be variable length, and it must be last. I'll pass.
      - CoolGuySteve 7y ago
        You definitely can but it's not as obvious, make a separate message type for list elements and append them on the wire. If you only have one list at the tail, you can use a flexible array[] at the end but it's finicky to deal with if you need more than one. You can build large hierarchical structures of messages with lists contained therein. It's pretty much how .mov/.mp4/many, many media container formats work. The technique dates back to the Amiga days.
  - doublement 7y ago
    Yeah I know that pain, there needs to be a consistent header with a version in all the messages.
- BubRoss 7y ago
  Why even put them in network byte order? Every modern system is little endian, if you standardize on that, only exotic systems would have to deserialize anything.
  - doublement 7y ago
    Because when someone builds a hugely popular exotic system in the future, because it is one (1) cent cheaper, you'd end up with code that has to check to see if it's running on such a system.
    - BubRoss 7y ago
      This doesn't make any sense for multiple reasons, but especially because you wouldn't be checking anything in the first place. A big endian system would would reorder bytes and a little endian system would just use it directly from memory without another copy or reordering anything.
  - bserfaty 7y ago
    Ha - I just had this exact argument yesterday. Why indeed?
  - toast0 7y ago
    There's not a library pattern for host to little endian, or little endian to host, like we have with hton and ntoh. Which makes it more likely to be messed up.
  - syncsynchalt 7y ago
    If you force the most common system to translate byte order, then you'll have some confidence that your code is performing the translation correctly. If instead you rely on hoping that everyone added the correct no-op translation calls everywhere, you'll find your code doesn't work as soon as you port it to another CPU. This is a nice side effect of network byte order being the opposite of the dominant cpu order, though obviously it was never intended.
- nabla9 7y ago
  The memory layout of a C struct is ABI and compiler dependent. Some compilers conform to same ABI in same system or similar system and work almost exactly the same, so you may grow old thinking that's how it is until it's too late. I think gcc, clang and Intel work almost the same in Linux and OSX.
  - doublement 7y ago
    Indeed, that's why I specified putting the members of the C structure on the wire, not the structure as a whole, so it's just basic types in network byte order (i.e. consistent endian-ness) being sent.
    - itronitron 7y ago
      I've worked on an application where that was the standard data transfer scheme, and then while working with protobuf on another project felt that after looking under protobuf's covers it was doing something very similar but wrapping an entire API around it.
  - CoolGuySteve 7y ago
    No, not really. #pragma pack and/or __attribute__((packed)) have been supported for eons now and guarantee the alignment of struct members between compilers. In newer C++ specs, you can also static assert that the struct is a POD type to statically ensure that there's no accidental vtable pointer. This argument pops up every time someone mentions this and every time it's completely uninformed.
    - cyphar 7y ago
      Though it should be noted that packed structures cause compilers to produce absolutely garbage code when accessing them (because most of the accesses become unaligned) and it becomes incredibly memory-unsafe (as in "your program may crash or corrupt memory") to take pointers of fields inside the struct because they are (usually) presumed to be aligned by the compiler. Explicit alignment doesn't suffer from this problem nearly as badly (yeah, you might have to add some padding but that's hardly the end of the world -- and if you have explicit padding fields you can reuse them in the future).
- innagadadavida 7y ago
  Protobufs have a pretty nice variable encoding integer wire format. This gives you the flexibility of saving space without doing compression. While zero copy is nice, you cannot make it work when using compression.
  - rapsey 7y ago
    That variable integer encoding is however slow for encoding/decoding. The space savings are of questionable worth.
    - seriesf 7y ago
      They are the same size as UTF-8 numbers but much slower to decode. I think the more-bit format is the only glaring mistake in proto that can never be fixed.
- caffeine 7y ago
  https://github.com/real-logic/simple-binary-encoding https://github.com/real-logic/simple-binary-encoding is a good way to do this
- ping_pong 7y ago
  XDR and ONC/RPC for the win!
- salgernon 7y ago
  I would like to submit apples archaic “Rez”[1] as a great language for declaring binary formats. It was designed to be able to describe c and pascal structures. [1] http://preserve.mactech.com/articles/mactech/Vol.14/14.09/RezIsYourFriend/index.html http://preserve.mactech.com/articles/mactech/Vol.14/14.09/Re...
- rbanffy 7y ago
  > Can we get back to just putting the members of C structures into network byte order and sending that over the wire in binary, à la 1995? I hope not (I know you are being sarcastic). We should use something that's trivial to implement correctly, as well as easy to read and to debug.
- sagarm 7y ago
  The wire encoding for protos is much more compact than the in-memory representation, especially for sparsely populated messages (very common especially in mature systems). You'd still have to figure out some way to serialize nested messages. Note that you can have recursive message definitions.
deleted 7y ago
[deleted]
memling 7y ago
> Protobuffers are shit coz they don't support zero copy and you have to deserialize the whole thing even if you are interested in one field or an outer envelope, causing memory churn in your JVMs. Cap'n'proto and flat buffers attack this real problem. The expressivity of the type system is a minor issue, hence no credible competition. I'm not terribly familiar with protobuffers, but I'm a little surprised by this. Some ASN.1 encodings (BER, CER, DER), by contrast, use nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. (This is, by the way, not that uncommon.) > Required should be enforced at application later not the binary packing layer. It is a property of the version of the code processing the blob, not the blob representation itself. This might depend a bit. Suppose you wanted to make sure that messages could be read (if not exactly decoded) without a schema. The structure of the message could, in principle, include this information in the form of bit-field preambles that indicate the number of fields, extensibility, and so forth. I don't suppose that's strictly necessary for most applications: embedding message structure in message content seems like a bit of an anti-pattern, but I bet you could come up with a use case that makes sense in some context.
- kentonv 7y ago
  > nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontally through an array, you need to inspect the tag/length of each element in order to skip to the next element. So you can't rapidly seek through a massive array. In order to allow such seeking, you either need statically-sized elements (which is often too restrictive), or you need pointers to data represented out-of-line. Cap'n Proto uses pointers. Using pointers is pretty uncommon among serialization formats, which is strange considering how ubiquitous they are for in-memory data structures. (Disclosure: I'm the author of Cap'n Proto, and also Protobuf v2.)
  - cryptonector 7y ago
    "ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.
m0th87 7y ago
I maintain a couple of protobuf-based libraries, and the issue I've seen with its anemic type system is that it inevitably creates an impedance mismatch between itself and the host language's type system. To make the library usable, you end up having to wrap the autogenerated code in a bunch of boilerplate, which defeats one of the major selling points of gRPC/protobuf in the first place.
- apeace 7y ago
  Exactly. I use Typescript with strict null checks. I would love to directly use the Proto objects I get from gRPC calls, but since the type of every string field is actually string|null, I have to do some validation and then turn it into an object with a regular string field (or else check that it’s not null every time I use it). I get that forcing this validation is a good thing, especially for bigger/distributed teams. But I’m the guy who wrote the backend and I know it will return an error if it cannot set some value in that string field. Therefore I consider it “required”, and I resent the Protobuf authors’ insistence that I am wrong to ever use that concept.
  - kentonv 7y ago
    That sounds like a quirk (I'd argue, a misfeature) of the specific Protobuf implementation you are using. In most implementations, the getter methods for optional fields will return a default value if the field isn't set. If you want to explicitly check for presence, you call a separate "has" method. There are several reasons for this design: * Convenience of avoiding null checks when you know the field is always set. * (Sometimes) Easy backwards-compatibility -- new fields can be declared with a default value that is appropriate when dealing with older senders. * Security: It shouldn't be trivially easy for a malicious client to omit a field that the server is expecting will be there, causing the server to crash or throw an exception. The C++ and Java implementations of Protobuf, at least, have always worked this way. It sounds like the TypeScript implementation you are using does not, unfortunately. (Dislosure: I wrote the C++ and Java implementations of proto2.)
mixedCase 7y ago
> Who cares about the type system of a binary packaging format? The people who have to write them and map them to actual domain data structures. Monomorphizing by hand and working around oneof+repeated's crap is an absolute joke. Tools in widespread use should do better.
yegle 7y ago
If you are looking for 0 copy alternative, look at https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- kortex 7y ago
  Do you know if flatbuffers can be used as data structures in applications? That seems to be a shortcoming of protobuf, all the ser/de code makes it suboptimal for in-app data transport.
  - kentonv 7y ago
    Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. E.g. in Cap'n Proto, all objects in a message are allocated sequentially within the larger message buffer. If you remove an object or change its size (e.g. overwrite a string field with a new value of different length), the new value needs to be allocated on the end and the memory space for the old value cannot be reused -- it is wasted. I'm not super-familiar with FlatBuffers, but I believe it uses a model where messages must strictly be constructed in bottom-up order, such that all pointers point in the same direction. This seems to imply that you can't modify a message at all after construction, but I haven't actually played with it so I could be mistaken. (I'm the author of Cap'n Proto.)
    - kortex 7y ago
      > Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. Oh, I never thought of it that way, but that makes perfect sense. I guess I just assumed messages would be collections of pointers and buffers living "somewhere" in memory, but of course the actual layout can make a ton of difference. I guess there is an implicit rule that if you are dealing with inbound structures in a read-only fashion, passing around the serial structure is OK for when the field access cost is minor compared to the copy cost, but if you want to mutate it, or doing lots of access operations where that isn't trivial, it makes sense to copy into your own data structure.
erik_seaberg 7y ago
> these messages can be used as rows in storage systems Storage needs a strong schema language, much more so than RPC does, because every mistake becomes permanent. A schema should only allow messages that make sense within the problem domain, and Maguire is right that protobuf is not good at this.

doublement 7y ago

This is going to sound sarcastic but it's not: Can we get back to just putting the members of C structures into network byte order and sending that over the wire in binary, à la 1995?

mikece 7y ago
Is that less of a configuration mess than WCF was? JSON isn't "The Magical Elixir" of data exchange and I'm more than open to something better but at least we (in the .NET community) have moved past the WCF configuration nightmares.
- bob1029 7y ago
  WCF is an unmitigated dumpster fire. We have actually written a non-WCF client that uses a raw HttpClient implementation with StringBuilder to compose SOAP envelopes around cached XMLSerializers in order to talk to other WCF services. First request delay went from 1-2 seconds down to a few milliseconds. Memory overhead is negligible now. Prior, you could watch task manager and immediately recognize when WCF is "warming up". Additionally, the XML serializer in .NET seems almost pathologically determined to ruin everything you seek to accomplish. By comparison, JSON contracts are an absolute joy to work with. We still practice strong-typing on both sides of the wire (we control both ends), and have pretty much nothing to complain about. If you are concerned with space overhead w/ JSON, simply passing it through gzip can get you down to a very reasonable place for 99% of use cases. I understand that there are arguments to be made against JSON for extremely performance sensitive applications, but I would counter-argue that these are extremely rare in practice.
  - heavenlyblue 7y ago
    Isn’t the problem with WCF rather than XML?
btown 7y ago
This is more or less what https://capnproto.org/ https://capnproto.org/ does.
- omginternets 7y ago
  From the capnproto docs: >Isn’t this all horribly insecure? >No no no! To be clear, we’re NOT just casting a buffer pointer to a struct pointer and calling it a day. Isn't this a direct contradiction to your claim? Or have I misunderstood them?
  - deleted 7y ago
    [deleted]
  - lidHanteyk 7y ago
    Capn is better than C at struct layout. We are not, under any circumstances, going back to the 90s. We are moving forward and learning from mistakes.
  - jtolmar 7y ago
    IIRC: capnproto generates messages that you could deserialize by casting them to the right struct, but refrains from actually doing it that way. Instead it generates a bunch of accessor methods that parse the data, as if you were reading something that's not basically a c-struct, like a protobuff.
    - kentonv 7y ago
      That's basically correct. Cap'n Proto generates classes with inline accessor methods that do roughly the same pointer arithmetic that the compiler would generate for struct access. There's a couple subtle differences: * The struct is allowed to be shorter than expected, in which case fields past the end are assumed to have their schema-defined default values. This is what allows you to add new fields over time while remaining forwards- and backwards-compatible. * Pointers are in a non-native format. They are offset-based (rather than absolute) and contain some extra type information (such as the size of the target, needed for the previous point). Following a pointer requires validating it for security. (Disclosure: I'm the author of Cap'n Proto.)
izacus 7y ago
This is practically exactly what Protobuffers are. Except that they actually are defined clearly enough for multiple services written in multiple languages can work with them.
- cma 7y ago
  I thought that's what capt'n proto was, not protobuffers.
- CoolGuySteve 7y ago
  Definitely not, protobuf's strange wire format becomes apparent if you ever look at the hexdump of one or the profiler output of your favourite protobuffer-decoding C/C++ application. They're actually kind of performance heavy for no benefit.
  - imtringued 7y ago
    I have once looked at a benchmark that compared protobuffer, message pack, json and a variety of other serialization formats. In terms of reducing bytes per message gzipped json was ahead of all of them at the cost of increased CPU time for gzip. Protobuffer did pretty poorly, the only benefit was decreased CPU usage. I'm sure you could use some other compression algorithm like LZMA to get both good compression and good performance for JSON messages.
    - CoolGuySteve 7y ago
      I use LZ4 (with "best" compression) for packet captures and replay with great results. I get about a 37% compression ratio with extremely fast decoding, like 10 million packets per second off an SSD. It was better than snappy, gzip, and bz2 for the trade-off of compression time, decompression time and file size. As for protobuf: flatbuffers, capn proto, HDF5, and plain C structs all deliver much, much faster decoding time. It's really not the best answer for any serialization at this point but it's still inexplicably popular.
    - touisteur 7y ago
amacbride 7y ago
XDR FTW!
pantalaimon 7y ago
This gets hairy the moment you want to add new fields.
- CoolGuySteve 7y ago
  Both protobuf and plain C structs are append only formats if you put the message-type and size at the start of the C-struct.
  - marcan_42 7y ago
    C structs do not compose extensively. Protobufs do. You can't put variable length data into a struct, and hence you can't put extensible structs into it either.
    - doublement 7y ago
      You can put variable length data into a struct: https://en.wikipedia.org/wiki/Flexible_array_member https://en.wikipedia.org/wiki/Flexible_array_member
      - xyzzyz 7y ago
        Only one field can be variable length, and it must be last. I'll pass.
    - CoolGuySteve 7y ago
      You definitely can but it's not as obvious, make a separate message type for list elements and append them on the wire. If you only have one list at the tail, you can use a flexible array[] at the end but it's finicky to deal with if you need more than one. You can build large hierarchical structures of messages with lists contained therein. It's pretty much how .mov/.mp4/many, many media container formats work. The technique dates back to the Amiga days.
- doublement 7y ago
  Yeah I know that pain, there needs to be a consistent header with a version in all the messages.
BubRoss 7y ago
Why even put them in network byte order? Every modern system is little endian, if you standardize on that, only exotic systems would have to deserialize anything.
- doublement 7y ago
  Because when someone builds a hugely popular exotic system in the future, because it is one (1) cent cheaper, you'd end up with code that has to check to see if it's running on such a system.
  - BubRoss 7y ago
    This doesn't make any sense for multiple reasons, but especially because you wouldn't be checking anything in the first place. A big endian system would would reorder bytes and a little endian system would just use it directly from memory without another copy or reordering anything.
- bserfaty 7y ago
  Ha - I just had this exact argument yesterday. Why indeed?
- toast0 7y ago
  There's not a library pattern for host to little endian, or little endian to host, like we have with hton and ntoh. Which makes it more likely to be messed up.
- syncsynchalt 7y ago
  If you force the most common system to translate byte order, then you'll have some confidence that your code is performing the translation correctly. If instead you rely on hoping that everyone added the correct no-op translation calls everywhere, you'll find your code doesn't work as soon as you port it to another CPU. This is a nice side effect of network byte order being the opposite of the dominant cpu order, though obviously it was never intended.
nabla9 7y ago
The memory layout of a C struct is ABI and compiler dependent. Some compilers conform to same ABI in same system or similar system and work almost exactly the same, so you may grow old thinking that's how it is until it's too late. I think gcc, clang and Intel work almost the same in Linux and OSX.
- doublement 7y ago
  Indeed, that's why I specified putting the members of the C structure on the wire, not the structure as a whole, so it's just basic types in network byte order (i.e. consistent endian-ness) being sent.
  - itronitron 7y ago
    I've worked on an application where that was the standard data transfer scheme, and then while working with protobuf on another project felt that after looking under protobuf's covers it was doing something very similar but wrapping an entire API around it.
- CoolGuySteve 7y ago
  No, not really. #pragma pack and/or __attribute__((packed)) have been supported for eons now and guarantee the alignment of struct members between compilers. In newer C++ specs, you can also static assert that the struct is a POD type to statically ensure that there's no accidental vtable pointer. This argument pops up every time someone mentions this and every time it's completely uninformed.
  - cyphar 7y ago
    Though it should be noted that packed structures cause compilers to produce absolutely garbage code when accessing them (because most of the accesses become unaligned) and it becomes incredibly memory-unsafe (as in "your program may crash or corrupt memory") to take pointers of fields inside the struct because they are (usually) presumed to be aligned by the compiler. Explicit alignment doesn't suffer from this problem nearly as badly (yeah, you might have to add some padding but that's hardly the end of the world -- and if you have explicit padding fields you can reuse them in the future).
innagadadavida 7y ago
Protobufs have a pretty nice variable encoding integer wire format. This gives you the flexibility of saving space without doing compression. While zero copy is nice, you cannot make it work when using compression.
- rapsey 7y ago
  That variable integer encoding is however slow for encoding/decoding. The space savings are of questionable worth.
  - seriesf 7y ago
    They are the same size as UTF-8 numbers but much slower to decode. I think the more-bit format is the only glaring mistake in proto that can never be fixed.
caffeine 7y ago
https://github.com/real-logic/simple-binary-encoding https://github.com/real-logic/simple-binary-encoding is a good way to do this
ping_pong 7y ago
XDR and ONC/RPC for the win!
salgernon 7y ago
I would like to submit apples archaic “Rez”[1] as a great language for declaring binary formats. It was designed to be able to describe c and pascal structures. [1] http://preserve.mactech.com/articles/mactech/Vol.14/14.09/RezIsYourFriend/index.html http://preserve.mactech.com/articles/mactech/Vol.14/14.09/Re...
rbanffy 7y ago
> Can we get back to just putting the members of C structures into network byte order and sending that over the wire in binary, à la 1995? I hope not (I know you are being sarcastic). We should use something that's trivial to implement correctly, as well as easy to read and to debug.
sagarm 7y ago
The wire encoding for protos is much more compact than the in-memory representation, especially for sparsely populated messages (very common especially in mature systems). You'd still have to figure out some way to serialize nested messages. Note that you can have recursive message definitions.

deleted 7y ago

[deleted]

memling 7y ago

> Protobuffers are shit coz they don't support zero copy and you have to deserialize the whole thing even if you are interested in one field or an outer envelope, causing memory churn in your JVMs. Cap'n'proto and flat buffers attack this real problem. The expressivity of the type system is a minor issue, hence no credible competition. I'm not terribly familiar with protobuffers, but I'm a little surprised by this. Some ASN.1 encodings (BER, CER, DER), by contrast, use nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. (This is, by the way, not that uncommon.) > Required should be enforced at application later not the binary packing layer. It is a property of the version of the code processing the blob, not the blob representation itself. This might depend a bit. Suppose you wanted to make sure that messages could be read (if not exactly decoded) without a schema. The structure of the message could, in principle, include this information in the form of bit-field preambles that indicate the number of fields, extensibility, and so forth. I don't suppose that's strictly necessary for most applications: embedding message structure in message content seems like a bit of an anti-pattern, but I bet you could come up with a use case that makes sense in some context.

kentonv 7y ago
> nested tag-length-value triads. This allows you to skip parts of the message that aren't interesting. Protobuf does that too. But in order to seek horizontally through an array, you need to inspect the tag/length of each element in order to skip to the next element. So you can't rapidly seek through a massive array. In order to allow such seeking, you either need statically-sized elements (which is often too restrictive), or you need pointers to data represented out-of-line. Cap'n Proto uses pointers. Using pointers is pretty uncommon among serialization formats, which is strange considering how ubiquitous they are for in-memory data structures. (Disclosure: I'm the author of Cap'n Proto, and also Protobuf v2.)
- cryptonector 7y ago
  "ASN.1/DER sucks -- everyone knows that. Let's build a new, better thing. How shall we encode things? Oh oh oh! I know! Let's have a type tag, a length, and a value encoding. Perfect! So so much better than ASN.1/DER!!!" Those who don't study the past... Look, ASN.1 is a crappy (but not awful!) schema language, but it supports many many encodings, and some of them are dumb, stupid, and bad, like BER, DER, and CER, and some are clever (PER), and some are awesome (OER), and it even supports things like XML (XER) and even JSON. So what's so bad about ASN.1? Not much, really, just that the first generation of encoding rules (BER/DER/CER) for it were. But no, people don't look. They jump without looking, and then they re-create things, and do it badly. The only thing since ASN.1 that doesn't suck is XDR. That's because XDR very much resembles ASN.1's PER/OER, but with 4-byte units, so XDR is very ergonomic. EDIT: I should also mention flatbuffers as not sucking.

m0th87 7y ago

I maintain a couple of protobuf-based libraries, and the issue I've seen with its anemic type system is that it inevitably creates an impedance mismatch between itself and the host language's type system. To make the library usable, you end up having to wrap the autogenerated code in a bunch of boilerplate, which defeats one of the major selling points of gRPC/protobuf in the first place.

apeace 7y ago
Exactly. I use Typescript with strict null checks. I would love to directly use the Proto objects I get from gRPC calls, but since the type of every string field is actually string|null, I have to do some validation and then turn it into an object with a regular string field (or else check that it’s not null every time I use it). I get that forcing this validation is a good thing, especially for bigger/distributed teams. But I’m the guy who wrote the backend and I know it will return an error if it cannot set some value in that string field. Therefore I consider it “required”, and I resent the Protobuf authors’ insistence that I am wrong to ever use that concept.
- kentonv 7y ago
  That sounds like a quirk (I'd argue, a misfeature) of the specific Protobuf implementation you are using. In most implementations, the getter methods for optional fields will return a default value if the field isn't set. If you want to explicitly check for presence, you call a separate "has" method. There are several reasons for this design: * Convenience of avoiding null checks when you know the field is always set. * (Sometimes) Easy backwards-compatibility -- new fields can be declared with a default value that is appropriate when dealing with older senders. * Security: It shouldn't be trivially easy for a malicious client to omit a field that the server is expecting will be there, causing the server to crash or throw an exception. The C++ and Java implementations of Protobuf, at least, have always worked this way. It sounds like the TypeScript implementation you are using does not, unfortunately. (Dislosure: I wrote the C++ and Java implementations of proto2.)

mixedCase 7y ago

> Who cares about the type system of a binary packaging format? The people who have to write them and map them to actual domain data structures. Monomorphizing by hand and working around oneof+repeated's crap is an absolute joke. Tools in widespread use should do better.

yegle 7y ago

If you are looking for 0 copy alternative, look at https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/

kortex 7y ago
Do you know if flatbuffers can be used as data structures in applications? That seems to be a shortcoming of protobuf, all the ser/de code makes it suboptimal for in-app data transport.
- kentonv 7y ago
  Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. E.g. in Cap'n Proto, all objects in a message are allocated sequentially within the larger message buffer. If you remove an object or change its size (e.g. overwrite a string field with a new value of different length), the new value needs to be allocated on the end and the memory space for the old value cannot be reused -- it is wasted. I'm not super-familiar with FlatBuffers, but I believe it uses a model where messages must strictly be constructed in bottom-up order, such that all pointers point in the same direction. This seems to imply that you can't modify a message at all after construction, but I haven't actually played with it so I could be mistaken. (I'm the author of Cap'n Proto.)
  - kortex 7y ago
    > Zero-copy formats usually turn out to be worse for use as in-app data structures, because they have to carefully control memory allocation and layout in a way that makes it hard to mutate an already-constructed message. Oh, I never thought of it that way, but that makes perfect sense. I guess I just assumed messages would be collections of pointers and buffers living "somewhere" in memory, but of course the actual layout can make a ton of difference. I guess there is an implicit rule that if you are dealing with inbound structures in a read-only fashion, passing around the serial structure is OK for when the field access cost is minor compared to the copy cost, but if you want to mutate it, or doing lots of access operations where that isn't trivial, it makes sense to copy into your own data structure.

erik_seaberg 7y ago

> these messages can be used as rows in storage systems Storage needs a strong schema language, much more so than RPC does, because every mistake becomes permanent. A schema should only allow messages that make sense within the problem domain, and Maguire is right that protobuf is not good at this.