3 ms·

In 20 years I've never worked on a project that had a test suite strong enough to merit blind faith. Usually the weak points are * realistic test input: either

In 20 years I've never worked on a project that had a test suite strong enough to merit blind faith. Usually the weak points are * realistic test input: either you carefully scrub actual production traffic and replay it (these require a ton of setup that doesn't age well), or you guess what you think users are doing and create dummy requests (and miss anything wacky you didn't know users do) * load & perf: it's rare for a team to spin up a prod-sized cluster just for a test, and before AWS it was almost unheard of to go out and buy twice the hardware you need, even when you were lucky enough not to be deploying on your customer's boxes. A change that tanks your cache hit rate can ruin your whole day yet look fine at trivial sizes. * visual design: Microsoft famously got bitten by moving away from manual testing that would catch dialogs made unusable by layout changes We usually had best-effort integration tests, but they never caught everything, and we always had to make tactical decisions about when to release what based on risk.

crdoconnor 10y ago
* Lack of realistic test input is usually a problem caused by over-reliance on unit testing or lack of monitoring feedback. * UX review and load & performance testing are usually things that we engage in regularly on trunk or the developer/code reviewer flags up ("this change has load implications/this change requires a UX review"). * "Microsoft famously got bitten by moving away from manual testing" - I'd never advocate eliminating manual testing entirely, just that it should be A) never repetitive, B) not a gatekeeper, C) always exploratory. >We usually had best-effort integration tests, but they never caught everything, Nothing ever catches everything but IMHO continuous delivery, short-lived branches, good monitoring and regular exploratory testing, load/perf testing on trunk is the best way to catch as much as possible. I've started experimenting with integration tests that generate reports with screenshots and story steps, as well, so that there's a tighter UX feedback.

crdoconnor 10y ago

* Lack of realistic test input is usually a problem caused by over-reliance on unit testing or lack of monitoring feedback. * UX review and load & performance testing are usually things that we engage in regularly on trunk or the developer/code reviewer flags up ("this change has load implications/this change requires a UX review"). * "Microsoft famously got bitten by moving away from manual testing" - I'd never advocate eliminating manual testing entirely, just that it should be A) never repetitive, B) not a gatekeeper, C) always exploratory. >We usually had best-effort integration tests, but they never caught everything, Nothing ever catches everything but IMHO continuous delivery, short-lived branches, good monitoring and regular exploratory testing, load/perf testing on trunk is the best way to catch as much as possible. I've started experimenting with integration tests that generate reports with screenshots and story steps, as well, so that there's a tighter UX feedback.