3 ms·
Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of cour
by jacquesm 3y ago
Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of course, standing by the sidelines it is easy to criticize and I'm sure whoever worked on this wasn't stupid. But I've seen this error often enough now in practice that I think that it needs to be drilled into programmers heads more forcefully: stuff is only valid if you have just validated it. If you send it to someone else, if someone you trust sends it to you, if you store in a database and then retrieve it and so on then it is just input all over again and you probably should validate it for being well-formed. If you don't do that then you're a bitflip, migration or an update away from an error that will cause your system to go into an unstable state and the real problem is that you might just propagate the error downstream because you didn't identify it.
Input is hard. Judging what constitutes 'input' in the first place can be harder.
- lgeorget 3y agoFrom what I gathered from the article, the input WAS valid. It's the software that was unable to handle a specific case of valid input.
- jacquesm 3y agoThat's fine, and is exactly the kind of case that I was thinking of: your software has a different idea of what is valid than an upstream piece of software, so from your perspective it is invalid. So you need to pull this message out of the stream, sideline it so it can be looked at by someone qualified enough to make the call of what's the case (because it could well be either way) and processing for all other messages should continue as normal. After all the only reason you can say with confidence that it in fact was valid is because someone looked at it! You can only do that well after the fact. A message switch [1] that I worked on had to deal with messages sources from 100's of different parties and while in principle everybody was working from the same spec (CCITT [2]) every day some malformed messages would land in the 'error' queue. Usually the problem was on the side of the sender, but sometimes (fortunately rarely) it wasn't and then the software would be improved to be able to handle that case correctly as well. Given the size of the specs and the many variations on the protocols it wasn't weird at all to see parties get confused. What's surprising is that it happens as rarely as it does. The big takeaway here should be that even if something happens very rarely it should still not result in a massive cascade, the system should handle this gracefully. [1] https://www.kvsa.nl/en/ https://www.kvsa.nl/en/ [2] https://en.wikipedia.org/wiki/Group_4_compression https://en.wikipedia.org/wiki/Group_4_compression
- dboreham 3y agoExact same experience developing systems that process RFC-822 (and descendents) email messages.
- onetimeuse92304 3y agoThis really isn't about input. Whether it comes from outside or produce inside the application, the reality is that everything can have bugs. A correct input can cause a buggy application to fail. So while verifying input is obviously an important step, it is not even a beginning if you are really looking to building reliable software. What really is the heart of the matter is for the entire thing to be allowed to crash due to a problem with single transaction. What you really want to do is to have firewalls. For example, you want a separate module that runs individual transactions and a separate shell that orchestrates everything but has no or very limited contact with the individual transactions. As bad as giving up on processing a single aircraft is, allowing the problem to cascade to entire system is way worse. What's even more tragic about this monumental waste of resources is that the knowledge about how to do all of this is readily available. The aerospace and automotive industry have very high development standards along with people you can hire who know those standards and how to use them to write reliable software.
- jacquesm 3y agoYes, there are multiple problems here that interplay in a really bad way and that's one of them. But the input processing/validation step is the first point of contact with that particular flight plan and it should have never progressed beyond that state. It all hinges on a whole bunch of assumptions and each and every one of those should be dealt with structurally rather than by patching things over. Just from reading TFA I see a very long list of things that would need attention. Quick recap: - validate all input - ensure the system can never stall on any one record - the system will occasionally come across malformed input which needs a process - it won't be immediately clear whether the system or the input is at fault, which needs a process - testing will need to take these scenarios into account - negative tests will need to be created (such as: purposefully malformed input) - attempts should be made to force the system into undefined states using malformed and well formed input - a supervisor mechanism needs to be built into the system that checks overall system health And probably many more besides. But this is what I gather from the article is what they'll need at a minimum. Typically once you start digging into what it would take to implement any of these you'll run into new things that also need fixing. As for the last bit of your comment: I'm quite sure that those standards were in play for this particular piece of software, the question is whether or not they were properly applied and even then there are no guarantees against mistakes, they can and do happen. All that those standards manage to do is to reduce their frequency by catching the bulk of them. But some do slip through, and always will. Perfect software never is.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]