2 ms·
> Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong
by Daishiman 5y ago
> Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production.
"Reliably" isn't a binary indicator, but a spectrum of how frequently certain classes of bugs may appear in a system.
In the example that you were mentioning, it would appear that the amount of effort needed to maintain the e2e test suite was simply not worth it. How many man-hours were spent by your manager and staff ignoring the test suite? How critical was the bug (it would appear not much)? How much effort would have to be dedicated to get the e2e suite working will that won't be spent doing other classes of tests or feature development?
I'm not saying a well-maintained e2e suite doesn't work well or help to catch a lot of interesting production bugs. But I am saying that I think that for the vast majority of systems it's just not a good use of your time. Save your efforts and put more thought into the system design to avoid certain theoretical classes of errors and devote the rest of your time to better integration tests and that will likely serve more orgs better.
- aidenn0 5y ago> How many man-hours were spent by your manager and staff ignoring the test suite? I'm not sure what you mean by that. Ignoring the test-suite isn't something that you spend time doing. It was 6ish weeks with a team of 8ish people, so you could say "48 man-weeks" were spent ignoring it, but they were doing other things in that time, not just sitting at their desk proclaiming "I'm ignoring this test." Once the manager forced someone to fix the test it took less than one man-day to find the bug, and about 5 minutes to fix the bug once it was found.