3 ms·

I'm looking at: http://tutorials.jenkov.com/iap/ion-vs-other-formats.html http://tutorials.jenkov.com/iap/ion-vs-other-formats.html As the author of Protobuf v

by kentonv 11y ago

I'm looking at: http://tutorials.jenkov.com/iap/ion-vs-other-formats.html http://tutorials.jenkov.com/iap/ion-vs-other-formats.html As the author of Protobuf v2 (the version that was open sourced by Google), I object to some of the "no"s in the protobuf column. (Note: I no longer work on Protobuf, and I did not invent the format. I do work on and did invent Cap'n Proto.) > Protobuf apparently isn't great at encoding raw bytes either (according to their own website). Protobuf can handle raw bytes just fine, using the "bytes" type. There is no special encoding done on bytes; parsing and encoding is done by memcpy(). I'm curious to know what part of the web site you interpret as saying otherwise. It's entirely possible that the web site contains confusing language, but a citation would have been a good idea here. > Schema / Class Id > Self describing The Protobuf libraries have extensive support for manipulating dynamic schemas and transmitting schemas over the wire. See the "Descriptor" and "DynamicMessage" APIs. This is mentioned on the web site: https://developers.google.com/protocol-buffers/docs/techniques#self-description https://developers.google.com/protocol-buffers/docs/techniqu... > Even if these compact objects do not contain any property names, they are still self describing enough that you can see where fields start and end, plus their data type, without an external schema. You cannot do that with Protobuf (as far as we know). You absolutely can do that with Protobuf. This is what the "protoc --decode_raw" flag does, and it should be clear enough from reading the encoding. https://developers.google.com/protocol-buffers/docs/encoding https://developers.google.com/protocol-buffers/docs/encoding > Cyclic references While it's true that Protobuf doesn't support these, I hope you've considered the denial-of-service vulnerabilities they tend to create if the receiver is not expecting them. Please ensure that cyclic references are only allowed in cases where the app opted into it. Relatedly, overlapping references / backreferences ("Copy" in your table) potentially leads to an amplification attack where a small message on the wire turns out to be much, much larger when traversed. If applications cannot defend themselves from huge payloads by setting a message size limit, then you'll need to give them some other way. > All of the formats (except perhaps Protobuf) supports arbitrary hierarchical navigation of the encoded data, without first converting it to objects. Protobuf supports this, and in fact should be an unqualified "Yes" rather than "Yes(*)" like the others. Protobuf encoding is very similar to ION's. Sub-messages are length-delimited, which seems to be exactly the advantage you're claiming that ION has. Note that none of these formats support random access in the way that Cap'n Proto does. In summary, I believe Protobuf deserves a "yes" in: "Raw bytes", "Good at raw bytes", "Schema / Class Id", "Arbitrary hierarchical navigation", and "Self describing".

VStack 11y ago
If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes. Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated. The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs). We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.
- kentonv 11y ago
  > Sending schema over the wire is not a good solution for anything else than point-to-point communication. OK, let's back up a moment. I am not entirely sure what "Schema / Class Id" in your table means. Your table claims Protobuf doesn't support it, but the text below is unclear on what you think Protobuf doesn't support. You frequently use the term "self-describing", but this could have two meanings: 1) Like JSON, where the names of all fields appear in the message, so that a human can read the message easily without external information. 2) Limited self-description in which field values can be identified and parsed, but their names are not available (perhaps replaced by numeric tags or indexes). Protobuf can support (1) by including the schema in the payload. I agree this is not commonly useful. Protobuf supports (2) natively, by virtue of being a TLV format (just like ION). Re-reading the page, it sounds like you are assuming the Protobuf format cannot be deciphered at all without the schema, but this simply isn't true. If you meant something else, please explain. > The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. I wrote that documentation. It doesn't mean what you think (my fault, perhaps). What it's saying is that the top-level message is a series of tag-value pairs with no explicit indication of where that series ends (on the assumption that you already know, e.g. based on EOF). Thus, if you concatenate two whole messages without adding any delimiter then it will look like one big message containing all the fields from both. However, each field within the message is clearly delimited and sub-messages are length-delimited therefore skipable. > We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. ION is a TLV encoding like Protobuf. Cap'n Proto is fixed offsets + pointers. These are vastly different styles of encoding that enable different modes of use. You can certainly debate which is better but I don't think it's correct to describe the formats as "pretty close", unless you consider all binary formats to be "pretty close" to each other.

by kentonv 11y ago

VStack 11y ago

If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes. Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated. The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs). We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.

kentonv 11y ago
> Sending schema over the wire is not a good solution for anything else than point-to-point communication. OK, let's back up a moment. I am not entirely sure what "Schema / Class Id" in your table means. Your table claims Protobuf doesn't support it, but the text below is unclear on what you think Protobuf doesn't support. You frequently use the term "self-describing", but this could have two meanings: 1) Like JSON, where the names of all fields appear in the message, so that a human can read the message easily without external information. 2) Limited self-description in which field values can be identified and parsed, but their names are not available (perhaps replaced by numeric tags or indexes). Protobuf can support (1) by including the schema in the payload. I agree this is not commonly useful. Protobuf supports (2) natively, by virtue of being a TLV format (just like ION). Re-reading the page, it sounds like you are assuming the Protobuf format cannot be deciphered at all without the schema, but this simply isn't true. If you meant something else, please explain. > The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. I wrote that documentation. It doesn't mean what you think (my fault, perhaps). What it's saying is that the top-level message is a series of tag-value pairs with no explicit indication of where that series ends (on the assumption that you already know, e.g. based on EOF). Thus, if you concatenate two whole messages without adding any delimiter then it will look like one big message containing all the fields from both. However, each field within the message is clearly delimited and sub-messages are length-delimited therefore skipable. > We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level. ION is a TLV encoding like Protobuf. Cap'n Proto is fixed offsets + pointers. These are vastly different styles of encoding that enable different modes of use. You can certainly debate which is better but I don't think it's correct to describe the formats as "pretty close", unless you consider all binary formats to be "pretty close" to each other.