3 ms·

Software testing, and why I'm unhappy about it

20 points by pabs3 4y ago

turtleyacht 4y ago
Three things may help: * Mutation testing * Contract testing * Formal verification Other things that may help: * NLP models that automate from Excel test cases * Probabilistic models coordinating on build pipeline and test runs to determine coverage, and which to skip * High-coverage integration and API tests Companies are already selling these.
jdlshore 4y ago
The article concludes with “the state of the art is depressing,” but doesn’t mention any of the work of the XP/TDD community, which addresses the problems the author mentioned. I don’t think the state of the art is depressing at all; what’s depressing is that the state of the art isn’t widely known.
omgmajk 4y ago
The tests should always follow the requirements first and code second, if they fail you should look at the system requirements and deduce whether the tests are bad or the code is actually failing. If the tests are bad, re-write them to fit the requirements, if the code is bad... don't push your new code. It's very hard to figure out what the author really is unhappy with.
nextlevelwizard 4y ago
I do not agree with the "test failing is normal" or flaky tests parts at all. CI pipe should be green all of the time. Why would you add failing tests for missing features before the feature? New tests should be merged in with the feature/bug fix. If a test start failing marking it as failing is a huge smell. Yes it will stop everyone from pushing, but that is how it should be. Everyone should be mobilized to figure out why the tests are failing and either reverting commits or fixing shit until we are back in green and normal work can continue. Same goes for flaky (a.k.a. bad) tests. You shouldn't run any kind of heuristics on failure rates to figure out if regression has happened it should be obvious from the state of the test (output should be binary). If a test fails randomly it means that your test is bad and it needs to be fixed. Sometimes this means implementing better instrumentation, sometimes it means doing "hard" stuff that previous test writer skipped, sometimes it requires completely rethinking how a feature should be tested.

20 points by pabs3 4y ago

Three things may help: * Mutation testing * Contract testing * Formal verification Other things that may help: * NLP models that automate from Excel test cases * Probabilistic models coordinating on build pipeline and test runs to determine coverage, and which to skip * High-coverage integration and API tests Companies are already selling these.

jdlshore 4y ago

The article concludes with “the state of the art is depressing,” but doesn’t mention any of the work of the XP/TDD community, which addresses the problems the author mentioned. I don’t think the state of the art is depressing at all; what’s depressing is that the state of the art isn’t widely known.

omgmajk 4y ago

The tests should always follow the requirements first and code second, if they fail you should look at the system requirements and deduce whether the tests are bad or the code is actually failing. If the tests are bad, re-write them to fit the requirements, if the code is bad... don't push your new code. It's very hard to figure out what the author really is unhappy with.

nextlevelwizard 4y ago

I do not agree with the "test failing is normal" or flaky tests parts at all. CI pipe should be green all of the time. Why would you add failing tests for missing features before the feature? New tests should be merged in with the feature/bug fix. If a test start failing marking it as failing is a huge smell. Yes it will stop everyone from pushing, but that is how it should be. Everyone should be mobilized to figure out why the tests are failing and either reverting commits or fixing shit until we are back in green and normal work can continue. Same goes for flaky (a.k.a. bad) tests. You shouldn't run any kind of heuristics on failure rates to figure out if regression has happened it should be obvious from the state of the test (output should be binary). If a test fails randomly it means that your test is bad and it needs to be fixed. Sometimes this means implementing better instrumentation, sometimes it means doing "hard" stuff that previous test writer skipped, sometimes it requires completely rethinking how a feature should be tested.