3 ms·
if you are fine releasing broken things, by all means, skip testing. Automated tests are faster and more repeatable than manual ones and allows you to more conf
by sethammons 3y ago
if you are fine releasing broken things, by all means, skip testing. Automated tests are faster and more repeatable than manual ones and allows you to more confidently release code. I like knowing that when I deploy that things basically still work.
At my last gig, acceptance testing that were blackbox were required before code could be merged to main. Sure, issues happened. But by far and large, we could release code and let alerts and other monitoring work and not have to check everything by hand. We seldom utterly broke things. We could confidently merge code and have it in production in like 15 minutes.
At my current gig, we lack acceptance tests or good integration tests for $crappy_reasons and even though we are a public company worth billions of dollars, every week the site is broken and people can't log in. Reverts happen regularly. The test suite takes over an hour and it is mostly unit level with some weak integration tests. The system itself is never actually exercised before it is released to customers. It took me 18 hours of test running to validate a schema change because you can't run the integration tests locally and so you have to let the build server do its thing. If tests were a priority here and treated well, my 18 hours would have been like 2. With no testing, I would have broken customer data and had to clean up tables after the fact.
If you prioritize fast, high level tests, you can really unlock velocity. If you have shitty tests that take a long time, your gonna have a bad time.
- whatever1 3y agoWhy do they have to release to all of the customers at once? Have a small sample, if they report bugs look at it, ow move on with more features. Tests are are very weak at proving the correctness of your code anyway. So why bother, there will be always non covered cases
- sethammons 3y agoOur customers pay us and don't want to pay for a broken experience. I respect our customers too much to ship untested code. Heck, even if we were free, I'd expect far more users to experience bugs than report them. Why would those users and customers continue to trust the reliability of your software? Thus reducing future conversions and giving reason for churn.
- whatever1 3y agoYour tests make you feel like your code is correct. The fact is you have no clue if your tests cover 99, 9 or 0.9% of all potential cases. If your code is critical you need proof of correctness, not feelings.
- sethammons 3y agoNonsense. The argument isn't 100% proof based tests vs yolo shipping directly to prod. Our tests are based on user actions and expected error conditions (from bad input to network errors). We don't cover the combinatorial explosion of potential inputs but we don't need 100% mathematical proof based confidence. Tests boost the confidence from "i think the code works" with no tests to "i know this code works in these N cases." It is risk management. Could we miss an unknown percentage of potential errors? Of course. But the errors that happen become fewer and fewer as the deployment safety net grows. And we monitor bug reports, errors, and incidents over time to gauge the effectiveness. This has allowed us to confidently ship code that has earned our company, literally, hundreds of millions of dollars and has evolved over time. I know my tests mean the software works as expected because my tests are my expectations. Maybe there is a cache bug or a memory leak or a logic bug or a misunderstood business requirement, but I _know_ that when my code deploys, standard usage of it will absolutely work. If it didn't, then a test should have failed. I've been developing software for a couple of decades now and I have a good feel on what to test at what layer. If the argument is only "let my valued customers experience bugs and report them to me" vs tests before code release, I'm siding on tests before releasing.