2 ms·
> Software testing doesn't work Got it right there! > We have some promising ideas. Some examples include strong type system, fuzzing, snapshot tests and sani
by feoren 4y ago
> Software testing doesn't work
Got it right there!
> We have some promising ideas. Some examples include strong type system, fuzzing, snapshot tests and sanitizers.
Okay, look. There's a lot of different times when a software error can be detected and fixed -- a whole spectrum. Let's be real generous about the categories of things "software error" can include here, including inefficiencies or building something the user doesn't actually want. The lower on this list we find errors, the more costly it is to fix them. Our goal is to push up where we detect errors as high as possible in the spectrum. It looks something like this (details and order may vary):
- Initial ideation ["Facebook for dogs? That idea sucks."]
- Requirements analysis ["That's not actually how we calculate that metric."]
- Conceptual system design ["Wait, these parts don't fit together like that."]
- Low-level design ["Crap, a loop won't work here, I need a different flow."]
- Brain-to-fingers typing ["Whoops I almost typed 'elesif'"]
- Immediately post-typing [Red squiggly line under 'elesif']
- Re-reading ["Wait that should be i-1, not i"]
- Compile time ["SYNTAX ERROR: Unknown identifier 'elesif'"]
- Unit test time ["Assertion failed: expected 7, got 8"]
- Code review time ["You didn't handle the case where N is negative"]
- Merge / integration test time ["Oh crap, David's commit broke my commit! How the hell do I merge this?"]
- Internal manual testing time ["You need to tighten up the graphics on level 3"]
- Production ["The users say the application keeps crashing when they run this report! Fix it!"]
- Years later/never ["Turns out the last 10 years of analysis that we used to make business decisions were wrong."]
Testing is actually quite low on the list, and there are lots of ways to move more errors higher up that list, such as strong typing, good abstractions, and really clean code. It's also tightly coupled to the implementation itself, so a conceptual error in the code can also exist in the test's assertions. I'd much rather write test that is obviously correct (errors caught at re-read or compile time) than write tests; only when you can't do you fall back to that lower-level option.