3 ms·

Just gonna add this here: https://news.ycombinator.com/item?id=23589117 https://news.ycombinator.com/item?id=23589117 (mostly for the followup discussion, see f

Just gonna add this here: https://news.ycombinator.com/item?id=23589117 https://news.ycombinator.com/item?id=23589117 (mostly for the followup discussion, see for example https://news.ycombinator.com/item?id=23589357 https://news.ycombinator.com/item?id=23589357). There's tradeoffs. "Packed encoding type is generally superior" for some workflows (write-once serialization across the wire), but certainly not across the board.

cryptonector 6y ago
There are tradeoffs, but I think binary, non-TLV encodings are superior. TLV encodings are notoriously difficult to encode but add no real value on the decode side. With TLV encodings you get a measure of implied extensibility that requires explicit support in binary, non-TLV encodings. But it turns out that you want to have explicit extensibility even in TLV cases anyways, since it helps the code generator, and for security reasons. Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto, TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs. Perhaps the best example I have of trade-offs is PostgreSQL's JSONB, which is optimized to be compressible and fast to traverse. To my knowledge that's the only binary JSON encoding that does that. (Specifically, it encodes lengths of array and object elements, and of object keys, but every N it encodes an offset, and the lengths are of the value encodings which are themselves concatenated. This allows similar lengths to be compressible, while the never-repeating offsets are not but also they're few and far between, and it also allows for fast indexing.)
- joshuamorton 6y ago
  > TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs. This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf). > Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings.
  - kentonv 6y ago
    > zero-copy can be a disadvantage in write-many applications To clarify, the disadvantage stems from the fact that to get zero-copy, you really need all the parts of a particular message to be in contiguous memory (or at least a small number of contiguous segments), which creates a memory allocator design challenge: now instead of one huge heap you have many small heaps and fragmentation potentially wastes space on the wire. So far I've made no attempt to solve this problem in Cap'n Proto; new objects are simply always added to the end of the message, which works fine for write-once and append-only scenarios but not for general mutable structures. I think this is an orthogonal issue to the TLV vs. fixed offsets debate.
    - joshuamorton 6y ago
      > I think this is an orthogonal issue to the TLV vs. fixed offsets debate I agree, I don't quite think they're related. But I think some of the original claim that tlv is worse is based on the potential for zero copy. That is advantageous in some common situations, but it's not always the case.
    - cryptonector 6y ago
      Right, this is a problem, but it's orthogonal to TLV, but it's TLV I was arguing against, not necessarily for cap'n proto. You can use OER without having to allocate all sub-parts of the structure from one flat buffer, with no "edits" that fragment the buffer, and no memmoves, and you won't get zero-copy, but you'll still get a more efficient codec than any TLV encoding. I just don't see any case where TLV is appealing, but TLV is appalling :) I think we're on the same page. EDIT: BTW, thanks for the clarification -- I thought that was the issue you meant, but what you linked to was lengthy and I didn't want to have to wade through all of it. EDIT: Also, even if zero-copy schemes are inefficient for some uses on the encoder side, they can still be very efficient on the decode side, as you need essentially no memory allocations to decode (though you still need to traverse the encoded data).
  - cryptonector 6y ago
    > This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf). Or maybe it's an argument against the kind of designs that tempt devs to hand-code them. > See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings. I don't buy that as a counter argument to the point about TLV's disadvantages relative to PER/OER and flatbuffers/cap'n proto. You might prefer OER to cap'n proto in some cases, or vice versa, but not TLV to either.

cryptonector 6y ago

There are tradeoffs, but I think binary, non-TLV encodings are superior. TLV encodings are notoriously difficult to encode but add no real value on the decode side. With TLV encodings you get a measure of implied extensibility that requires explicit support in binary, non-TLV encodings. But it turns out that you want to have explicit extensibility even in TLV cases anyways, since it helps the code generator, and for security reasons. Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto, TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs. Perhaps the best example I have of trade-offs is PostgreSQL's JSONB, which is optimized to be compressible and fast to traverse. To my knowledge that's the only binary JSON encoding that does that. (Specifically, it encodes lengths of array and object elements, and of object keys, but every N it encodes an offset, and the lengths are of the value encodings which are themselves concatenated. This allows similar lengths to be compressible, while the never-repeating offsets are not but also they're few and far between, and it also allows for fast indexing.)

joshuamorton 6y ago
> TLV is bloaty and redundant, and redundancy is one mother of serious bugs in hand-rolled codecs. This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf). > Even ignoring the zero-copy advantages of flatbufs and Cap'n Proto See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings.
- kentonv 6y ago
  > zero-copy can be a disadvantage in write-many applications To clarify, the disadvantage stems from the fact that to get zero-copy, you really need all the parts of a particular message to be in contiguous memory (or at least a small number of contiguous segments), which creates a memory allocator design challenge: now instead of one huge heap you have many small heaps and fragmentation potentially wastes space on the wire. So far I've made no attempt to solve this problem in Cap'n Proto; new objects are simply always added to the end of the message, which works fine for write-once and append-only scenarios but not for general mutable structures. I think this is an orthogonal issue to the TLV vs. fixed offsets debate.
  - joshuamorton 6y ago
    > I think this is an orthogonal issue to the TLV vs. fixed offsets debate I agree, I don't quite think they're related. But I think some of the original claim that tlv is worse is based on the potential for zero copy. That is advantageous in some common situations, but it's not always the case.
  - cryptonector 6y ago
    Right, this is a problem, but it's orthogonal to TLV, but it's TLV I was arguing against, not necessarily for cap'n proto. You can use OER without having to allocate all sub-parts of the structure from one flat buffer, with no "edits" that fragment the buffer, and no memmoves, and you won't get zero-copy, but you'll still get a more efficient codec than any TLV encoding. I just don't see any case where TLV is appealing, but TLV is appalling :) I think we're on the same page. EDIT: BTW, thanks for the clarification -- I thought that was the issue you meant, but what you linked to was lengthy and I didn't want to have to wade through all of it. EDIT: Also, even if zero-copy schemes are inefficient for some uses on the encoder side, they can still be very efficient on the decode side, as you need essentially no memory allocations to decode (though you still need to traverse the encoded data).
- cryptonector 6y ago
  > This is an argument against hand-generating TLV binary packets, not an argument against using generated code (as in protobuf). Or maybe it's an argument against the kind of designs that tempt devs to hand-code them. > See kenton's comment I linked, the zero-copy can be a disadvantage in write-many applications, since if you have a variable-length field (a string or submessage) and you can end up with super inefficient binary encodings. I don't buy that as a counter argument to the point about TLV's disadvantages relative to PER/OER and flatbuffers/cap'n proto. You might prefer OER to cap'n proto in some cases, or vice versa, but not TLV to either.