2 ms·

My baptism by fire was when I had to write Protobuf parser and generator. It's the same exact story: format author didn't think it through, didn't realize some

My baptism by fire was when I had to write Protobuf parser and generator. It's the same exact story: format author didn't think it through, didn't realize some bad consequences are about to be had, made vague or unrealistic claims in documentation (or forgot to mention things in documentation). Since then, I've seen a lot of this stuff... I cannot think about a format suitable for archives / data transfer / configuration that was well designed. And by this I mean that the claims made by the author of the format are shown to be true in a formal way. It feels very sad that despite countless failures, which all result from the lack of a tool for doing this formal thing that can check the specification and tell you if it makes sense... nobody really works on that. It seems obvious that you'd want to have a tool that tells you how well you are doing, and programmers are very dedicated to tools that do formal verification (s.a. compile-time type checkers), but when it comes to something with far more reaching consequences is a "whatever".

GuB-42 3y ago
A way to deal with the problem is to make a reference implementation instead of a specification document. A document can also be written for convenience, but if code and documentation disagree, short of an obvious bug, the code is right and the docs need to change to reflect that. And it is actually a common practice. This way, nothing is vague or missing. If you wonder what is going to happen if a certain field has a certain value, run the code and see what happen, it may not be great, but you know exactly what to expect. With code, you also have access to all the developer tools: compilers, linters, test frameworks, debuggers, coverage analyzers, fuzzers,... and even provers. Ideally the reference implementation should be open source with a permissive license, be written with readability in mind, and come with a good test suite. The problem with zip as described in the article is that the specs are half-assed and the real test for a correct zip file is that it should work with pkzip. But we can't really use pkzip as a reference either because it is proprietary.
- crabbone 3y ago
  A way to deal with the problem? I think this is how the problem was created in the first place: the authors wrote the reference implementation, but then didn't test it enough, didn't think of all possible ways things can go wrong. Also, I'm not saying that this necessarily results in bad formats. Eg. I don't particularly like Thrift, but it's kind of OK in the sense that it doesn't do much weird stuff unexpectedly. I don't believe they had any formal way to verify that their plan is going to work, they were "lucky" because they had to work with Protobuf a lot and figured some of its shortfalls before they wrote their own. So, unless you have some sort of a mechanical way to verify that your goals are achieved by the format you are creating, it's going to work just in the same way how any given C program won't corrupt memory: you might get lucky, or after a lot of testing you'll conclude that it's very unlikely that the program will corrupt memory, but you can never be sure. And, really, I don't blame the authors. I'm not saying they were lazy, or didn't pay enough attention. It's just hard to do when you don't have a watchdog who will absolutely not allow any and all transgressions. I wouldn't count on myself to do that w/o such a watchdog -- I don't have that kind of confidence even after implementing many different formats used for similar purpose.

GuB-42 3y ago

A way to deal with the problem is to make a reference implementation instead of a specification document. A document can also be written for convenience, but if code and documentation disagree, short of an obvious bug, the code is right and the docs need to change to reflect that. And it is actually a common practice. This way, nothing is vague or missing. If you wonder what is going to happen if a certain field has a certain value, run the code and see what happen, it may not be great, but you know exactly what to expect. With code, you also have access to all the developer tools: compilers, linters, test frameworks, debuggers, coverage analyzers, fuzzers,... and even provers. Ideally the reference implementation should be open source with a permissive license, be written with readability in mind, and come with a good test suite. The problem with zip as described in the article is that the specs are half-assed and the real test for a correct zip file is that it should work with pkzip. But we can't really use pkzip as a reference either because it is proprietary.

crabbone 3y ago
A way to deal with the problem? I think this is how the problem was created in the first place: the authors wrote the reference implementation, but then didn't test it enough, didn't think of all possible ways things can go wrong. Also, I'm not saying that this necessarily results in bad formats. Eg. I don't particularly like Thrift, but it's kind of OK in the sense that it doesn't do much weird stuff unexpectedly. I don't believe they had any formal way to verify that their plan is going to work, they were "lucky" because they had to work with Protobuf a lot and figured some of its shortfalls before they wrote their own. So, unless you have some sort of a mechanical way to verify that your goals are achieved by the format you are creating, it's going to work just in the same way how any given C program won't corrupt memory: you might get lucky, or after a lot of testing you'll conclude that it's very unlikely that the program will corrupt memory, but you can never be sure. And, really, I don't blame the authors. I'm not saying they were lazy, or didn't pay enough attention. It's just hard to do when you don't have a watchdog who will absolutely not allow any and all transgressions. I wouldn't count on myself to do that w/o such a watchdog -- I don't have that kind of confidence even after implementing many different formats used for similar purpose.