4 ms·
That's because these are knowable errors; you can't talk about a bug if you don't know it might occur. Testing is not a panacea, by design you can only test fo
by Cthulhu_ 5y ago
That's because these are knowable errors; you can't talk about a bug if you don't know it might occur.
Testing is not a panacea, by design you can only test for what you know and predict. In addition to testing, your setup needs to be resilient enough that unpredictable bugs can be solved quickly and won't lead to data loss.
- pydry 5y agoI don't think we do this because integration and specification bugs are inherently "unknowable". I think it's more cultural than technical. I've also noticed a parallel phenomenon where teams double down on unit tests for branch conditions, coverage %, etc. when the bugs that are really hurting them predominantly cant even really be detected by unit tests.
- faizshah 5y agoI’m with you, we spend most of our time testing for bugs that we are more comfortable with but they all have some underlying assumption about how we integrate with outside systems. Defensive programming has often taken a backseat to delivering features and testing has more so become a rubber stamp for regressions rather than a test of resilience. A lot of modern software, especially web apps are very brittle, when something stops working the way it’s supposed to the whole app goes into an undefined failure mode instead of giving a user partial functionality.
- piyh 5y agoI inherited a distributed app that causes me a lot of off hours work. My goal build pipeline brings up the app in docker-compose, then randomly kills services while running integration tests. So many failures are because people don't consider the happy path in a distributed/highly modular system.
- thaumasiotes 5y ago> Testing is not a panacea, by design you can only test for what you know and predict. Fuzz testing is still a type of testing.
- AnimalMuppet 5y agoFuzz testing is based on us knowing that certain kinds of bugs can be triggered by certain kinds of input, even if we don't know the exact details.
- thaumasiotes 5y agoNo it isn't. Fuzz testing is based on us knowing that bugs probably exist, and no more than that. The entire concept is that you provide random input and see whether bugs occur. You're not looking for particular kinds of bugs.
- AnimalMuppet 5y agoWe know that programs sometimes expect data with certain relationships between different parts. For example, one part may be a length field, and the program expects that some other part of the data will have that length, and may fail to handle it correctly if the data does not conform to that assumption. Fuzzing is a... well, a slightly-better-than-nothing way of trying to activate that kind of bug. Could fuzzing activate other kinds of bugs? Sure. But it's not magic. It's mostly working on known kinds of bugs.
- thaumasiotes 5y agoYou have some strange ideas about what fuzzing is.
- AnimalMuppet 5y agoIf you know more, you could provide some clarity, rather than just insults.
- ThrowawayR2 5y agoFuzz testing requires a test oracle to tell you whether the output is good or bad. Otherwise, all you're checking for is crashes which is unlikely to yield much.