4 ms·

While there's a lot of issues with JSON, this one also applies to any other interchange format that supports nesting, including the much beloved XML. Protobuf m

While there's a lot of issues with JSON, this one also applies to any other interchange format that supports nesting, including the much beloved XML. Protobuf might also have this, idk if it does any static analysis for infinite depth.

q3k 5y ago
The problem doesn't really exist in Protobuf, as protobuf (de)serialization is performed based on an IDL definition of the message type. Whatever that IDL specifies, a corresponding typed definition and (de)serialization function will be generated for your programming language, and that implementation will ignore any fields that weren't part of the IDL. The (de)serializing code is statically generated ahead of time, and is treated like any other code that operates on potentially nested data structures. What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored. Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.
- tehbeard 5y ago
  Thank you for the 2nd + 3rd paragraphs, those were parts of protobufs design I wasn't really aware of from a cursory glance. I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.
  - q3k 5y ago
    > I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages. That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :) I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.
- charleslmunger 5y ago
  To clarify, tags unknown to the IDL are propagated without parsing, not ignored.
ChrisMarshallNY 5y ago
> the much beloved XML I can't quite resolve "beloved" and "XML" in the same sentence... That said, I have used XML a lot, pretty much because of XML Schema. I don't like it. No sir. Not one bit. Uh-uh... But there's really no viable substitute. When I design an API, I generally start with an object model, and use native converters to create XML and JSON from it. I will provide an XML Schema with the XML variant. I often have to do this by hand, which sucks. There are tools to create Schema from dumps, but these are pretty limited. I may use them to "get me in the ballpark," but there's always lots of elbow grease. I'll use the XML for testing, but I will usually use the JSON format for the actual implementation.
- nocman 5y ago
  > I can't quite resolve "beloved" and "XML" in the same sentence... You mean, it's possible to take that as being NON-sarcastic??? If so, I share your lack of resolution.

q3k 5y ago

The problem doesn't really exist in Protobuf, as protobuf (de)serialization is performed based on an IDL definition of the message type. Whatever that IDL specifies, a corresponding typed definition and (de)serialization function will be generated for your programming language, and that implementation will ignore any fields that weren't part of the IDL. The (de)serializing code is statically generated ahead of time, and is treated like any other code that operates on potentially nested data structures. What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored. Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.

tehbeard 5y ago
Thank you for the 2nd + 3rd paragraphs, those were parts of protobufs design I wasn't really aware of from a cursory glance. I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.
- q3k 5y ago
  > I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages. That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :) I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.
charleslmunger 5y ago
To clarify, tags unknown to the IDL are propagated without parsing, not ignored.

ChrisMarshallNY 5y ago

> the much beloved XML I can't quite resolve "beloved" and "XML" in the same sentence... That said, I have used XML a lot, pretty much because of XML Schema. I don't like it. No sir. Not one bit. Uh-uh... But there's really no viable substitute. When I design an API, I generally start with an object model, and use native converters to create XML and JSON from it. I will provide an XML Schema with the XML variant. I often have to do this by hand, which sucks. There are tools to create Schema from dumps, but these are pretty limited. I may use them to "get me in the ballpark," but there's always lots of elbow grease. I'll use the XML for testing, but I will usually use the JSON format for the actual implementation.

nocman 5y ago
> I can't quite resolve "beloved" and "XML" in the same sentence... You mean, it's possible to take that as being NON-sarcastic??? If so, I share your lack of resolution.