5 ms·

> but it's not really a problem with the format itself, beyond the fact that clearer specs might disambiguate things and lead to less deviation. It is a proble

by q3k 5y ago

> but it's not really a problem with the format itself, beyond the fact that clearer specs might disambiguate things and lead to less deviation. It is a problem, because it's not a spec that can be implemented reliably. Different parsers behave differently on various corner cases not only because of implementation blunders, but also because the standard(s) just let them do whatever. This spectacularly breaks systems that use more than one parser implementation, each slightly implementing the standard slightly differently. One part of some processing/parsing pipeline will let some payload through, while another one will reject it, or even parse it differently.

horsawlarway 5y ago
I disagree (at least mostly). This is a case where the spec is intentionally loose to allow compatibility with a much larger number of machines and use cases. You'll notice most of the cases where the behavior is implementation defined have resource requirements. example: how deep you want to allow nesting depends a LOT on the capabilities of the machine running the code. A sane value for a modern browser is going to be unworkable on an arduino/ESP32/embedded other. Also... if these ambiguities bother you, you probably haven't read the full http spec either. It's riddled with cases where behavior is implementation defined, for exactly the same reasons (resources are required, and you can't assume everyone has the same amount available). Want to take a guess at the maximum length for a url?
- q3k 5y ago
  > This is a case where the spec is intentionally loose to allow compatibility with a much larger number of machines and use cases. There's plenty that could've been specified with little detriment to small systems: strings are UTF-8 with a well-defined escape sequence set, numbers are always IEEE-754 doubles, messages cannot be nested by more than 128 levels (or some other arbitrary number in this range), repeated fields are not permitted, everything non-compliant must fail the entire parse. Then the only thing left to handle is a maximum serialized size (which can be explicitly implementation or user defined). Set the maximum string length to maximum payload length defined earlier and you're golden. That is then your only difference between implementations. This will work on your Ryzen server and on your ESP8266 or ESP32, and can even be handled on your washing machine microcontroller^W^W^WArduino (with a slowdown for dealing with floating point numbers, but you already have to deal with that). Finally, the spec isn't loose because of some design choice to allow interoperability with more machines: it's loose because it was historically loose (see: JSON business card 'specification' chutzpah, which itself is based on a mess of a programming language that is/was JS), and before it could be formalized to something sensible it got implemented haphazardly by different languages. That doomed the format to forever be underdefined, as anything more strict would render existing implementations non-compliant.
  - horsawlarway 5y ago
    But that attempt at strictness harms implementation value. Even your own requirement set that you've claimed will work on everything is... bad - Sure I can parse every number as a double, if I'm willing to spend at least 64bits on every number in the payload. I just finished building a PH Autodoser for a hydroponics system I run - it sends JSON payloads with sensor data and receives JSON commands to do things like dispense PHDown/PHUp solution, toggle on water cooling. I have very little spare working memory on the device doing the actual monitoring. having to hold 64 bits per number would push me into having to buy a more expensive microcontroller. Instead - I have an informal contract that almost all fields are plain unsigned bytes (0 to 255) which works fine for my use-case, requiring just 1/8th the space. And to go the other direction - I have a desktop running some financial software, I pass around json payloads there, but a double is NOT ENOUGH. I want a BigInt field for numbers there instead, because rounding errors that would be a-ok for a ph sensor are absolutely not ok for calculating financial data. ---- Basically - I want the flexibility to chose the correct interpretation for my data. And this: "everything non-compliant must fail the entire parse." Is just fucking insanity. It's the literal antithesis of the robustness principle: "be conservative in what you send, be liberal in what you accept"
    - q3k 5y ago
      > Instead - I have an informal contract that almost all fields are plain unsigned bytes (0 to 255) which works fine for my use-case, requiring just 1/8th the space. Right, but that informal contract is at the detriment of everyone else having to also specify the expected limits of numbers they work with. It makes your particular usecase easier, but it doesn't make the standard better in the grand scheme of things. > And to go the other direction - I have a desktop running some financial software, I pass around json payloads there, but a double is NOT ENOUGH. I want a BigInt field for numbers there instead, because rounding errors that would be a-ok for a ph sensor are absolutely not ok for calculating financial data. And JSON doesn't guarantee you that, you have to shop around for languages and implementations that permit this. If you then have to make work with an implementation that always deserializes to doubles (which is a compliant behaviour) or bytes (which is a compliant behaviour), you're screwed. Again, this might work for the simple case of you controlling both ends of the serialization, but it's terrible for trying to work with an end that you don't control (ie. when actually using JSON as an interchange format). > And this: "everything non-compliant must fail the entire parse." Is just fucking insanity. It's the literal antithesis of the robustness principle: "be conservative in what you send, be liberal in what you accept" The Robustness Principle followed blindly is known to be harmful when dealing with long-term standards, evolving implementations and the human element of software engineering [1]. My opinion is that an interchange format's job is to transfer some data reliably and atomically: the deserialized data should be either be 100% correct or the deserialization should be rejected. Anything else can and will lead to bugs, and bugs that are then difficult to solve (as at that point it's difficult to agree whether the serialization was not conservative enough, or the deserialization not liberal enough). [1] - https://datatracker.ietf.org/doc/html/draft-iab-protocol-maintenance https://datatracker.ietf.org/doc/html/draft-iab-protocol-mai...

by q3k 5y ago

horsawlarway 5y ago

I disagree (at least mostly). This is a case where the spec is intentionally loose to allow compatibility with a much larger number of machines and use cases. You'll notice most of the cases where the behavior is implementation defined have resource requirements. example: how deep you want to allow nesting depends a LOT on the capabilities of the machine running the code. A sane value for a modern browser is going to be unworkable on an arduino/ESP32/embedded other. Also... if these ambiguities bother you, you probably haven't read the full http spec either. It's riddled with cases where behavior is implementation defined, for exactly the same reasons (resources are required, and you can't assume everyone has the same amount available). Want to take a guess at the maximum length for a url?

q3k 5y ago
> This is a case where the spec is intentionally loose to allow compatibility with a much larger number of machines and use cases. There's plenty that could've been specified with little detriment to small systems: strings are UTF-8 with a well-defined escape sequence set, numbers are always IEEE-754 doubles, messages cannot be nested by more than 128 levels (or some other arbitrary number in this range), repeated fields are not permitted, everything non-compliant must fail the entire parse. Then the only thing left to handle is a maximum serialized size (which can be explicitly implementation or user defined). Set the maximum string length to maximum payload length defined earlier and you're golden. That is then your only difference between implementations. This will work on your Ryzen server and on your ESP8266 or ESP32, and can even be handled on your washing machine microcontroller^W^W^WArduino (with a slowdown for dealing with floating point numbers, but you already have to deal with that). Finally, the spec isn't loose because of some design choice to allow interoperability with more machines: it's loose because it was historically loose (see: JSON business card 'specification' chutzpah, which itself is based on a mess of a programming language that is/was JS), and before it could be formalized to something sensible it got implemented haphazardly by different languages. That doomed the format to forever be underdefined, as anything more strict would render existing implementations non-compliant.
- horsawlarway 5y ago
  But that attempt at strictness harms implementation value. Even your own requirement set that you've claimed will work on everything is... bad - Sure I can parse every number as a double, if I'm willing to spend at least 64bits on every number in the payload. I just finished building a PH Autodoser for a hydroponics system I run - it sends JSON payloads with sensor data and receives JSON commands to do things like dispense PHDown/PHUp solution, toggle on water cooling. I have very little spare working memory on the device doing the actual monitoring. having to hold 64 bits per number would push me into having to buy a more expensive microcontroller. Instead - I have an informal contract that almost all fields are plain unsigned bytes (0 to 255) which works fine for my use-case, requiring just 1/8th the space. And to go the other direction - I have a desktop running some financial software, I pass around json payloads there, but a double is NOT ENOUGH. I want a BigInt field for numbers there instead, because rounding errors that would be a-ok for a ph sensor are absolutely not ok for calculating financial data. ---- Basically - I want the flexibility to chose the correct interpretation for my data. And this: "everything non-compliant must fail the entire parse." Is just fucking insanity. It's the literal antithesis of the robustness principle: "be conservative in what you send, be liberal in what you accept"
  - q3k 5y ago
    > Instead - I have an informal contract that almost all fields are plain unsigned bytes (0 to 255) which works fine for my use-case, requiring just 1/8th the space. Right, but that informal contract is at the detriment of everyone else having to also specify the expected limits of numbers they work with. It makes your particular usecase easier, but it doesn't make the standard better in the grand scheme of things. > And to go the other direction - I have a desktop running some financial software, I pass around json payloads there, but a double is NOT ENOUGH. I want a BigInt field for numbers there instead, because rounding errors that would be a-ok for a ph sensor are absolutely not ok for calculating financial data. And JSON doesn't guarantee you that, you have to shop around for languages and implementations that permit this. If you then have to make work with an implementation that always deserializes to doubles (which is a compliant behaviour) or bytes (which is a compliant behaviour), you're screwed. Again, this might work for the simple case of you controlling both ends of the serialization, but it's terrible for trying to work with an end that you don't control (ie. when actually using JSON as an interchange format). > And this: "everything non-compliant must fail the entire parse." Is just fucking insanity. It's the literal antithesis of the robustness principle: "be conservative in what you send, be liberal in what you accept" The Robustness Principle followed blindly is known to be harmful when dealing with long-term standards, evolving implementations and the human element of software engineering [1]. My opinion is that an interchange format's job is to transfer some data reliably and atomically: the deserialized data should be either be 100% correct or the deserialization should be rejected. Anything else can and will lead to bugs, and bugs that are then difficult to solve (as at that point it's difficult to agree whether the serialization was not conservative enough, or the deserialization not liberal enough). [1] - https://datatracker.ietf.org/doc/html/draft-iab-protocol-maintenance https://datatracker.ietf.org/doc/html/draft-iab-protocol-mai...