4 ms·

None of this really argues against fuzz testing; even with completely bogus/malformed flight plans, it shouldn't be possible for a dead letter to take down the

None of this really argues against fuzz testing; even with completely bogus/malformed flight plans, it shouldn't be possible for a dead letter to take down the entire system. And, since it's translating between an upstream and downstream format (and all the validation is done when ingesting the upstream), you probably want to be sure anything that is valid upstream is also valid downstream. It's true that fuzz testing is easiest when you can do it more at the unit level (fuzz this function implementing a core algorithm, say) but doing whole-system fuzz tests is perfectly fine too.

crabbone 3y ago
This is not against the principle of fuzz testing. This is to say that the author doesn't really know the reality of testing and is very quick to point fingers. It's easy to tell in retrospect that this particular aspect should've been tested. It's basically impossible to find such defects proactively.
- epolanski 3y ago
  I've read both messages and I'm still unsure on how fuzzy testing may have not brought up similar edge cases. We literally talking about a parser shutting down an entire system rather than reporting malformed data. Considering this is a "one in 15M cases" it seems to me that fuzzy testing would've caught this and probably more bugs in a short time span.
  - nimchimpsky 3y ago
    [dead]
- theptip 3y ago
  Easy for me to say in retrospect, but IMO this is a textbook example of where you should reach for fuzz testing; it’s basically protocol parsing, you have a well-known text format upstream and you need to ensure your system can parse all well-formed protocol messages and at very least not crash if a given message is invalid in your own system. Similarly with a message queue, handling dead letters is textbook stuff, and you must have system tests to verify that poison pills do not break your queue. I did not think the author was setting unreasonable expectations for the a priori testing regime. These are common best practices.

crabbone 3y ago

This is not against the principle of fuzz testing. This is to say that the author doesn't really know the reality of testing and is very quick to point fingers. It's easy to tell in retrospect that this particular aspect should've been tested. It's basically impossible to find such defects proactively.

epolanski 3y ago
I've read both messages and I'm still unsure on how fuzzy testing may have not brought up similar edge cases. We literally talking about a parser shutting down an entire system rather than reporting malformed data. Considering this is a "one in 15M cases" it seems to me that fuzzy testing would've caught this and probably more bugs in a short time span.
- nimchimpsky 3y ago
  [dead]
theptip 3y ago
Easy for me to say in retrospect, but IMO this is a textbook example of where you should reach for fuzz testing; it’s basically protocol parsing, you have a well-known text format upstream and you need to ensure your system can parse all well-formed protocol messages and at very least not crash if a given message is invalid in your own system. Similarly with a message queue, handling dead letters is textbook stuff, and you must have system tests to verify that poison pills do not break your queue. I did not think the author was setting unreasonable expectations for the a priori testing regime. These are common best practices.