3 ms·
I'm replying to this and also some of the sibling comments. No, automated development-time tests do not necessarily equate to higher quality, given that they:
by jiggawatts 5y ago
I'm replying to this and also some of the sibling comments.
No, automated development-time tests do not necessarily equate to higher quality, given that they:
- Take time away from other activities
- May provide a false positive, lulling developers into a false sense of confidence
- Increase the time taken for certain CI/CD operations, sometimes by hours or days!
- The presence of a large volume of "test" code means that certain refactoring operations take a lot longer, which is a real cost.
People go into some work environments where tests DID help, and they generalise that to ALL work environments.
Automated build-time tests are less important, and entire categories may be potentially unnecessary when:
- The product is written in a strongly-typed language that catches most errors at compile time.
- If high-level "linting" tools are used to further enhance the code quality before the executable ever runs.
- If feedback from the production environment is more relevant. E.g.: for web applications with performance issues being monitored by an APM, but no critical code stability concerns at build time. Think typical web apps with no real consequences to a crash, not finance applications where a small error could mean Real Money.
- Where time-to-market is more important than chasing 100% robustness at the expense of meeting release dates.
You often see people from a PHP or Python background go on and on about the importance of tests, but those are dynamically typed languages with lots of unexpected pitfalls that only testing will uncover. Meanwhile entire teams do just fine without any automated tests when using languages like C#, Java, or Rust.
- cweagans 5y agoTypes do a great job at assuring you’re getting generally the right kind of data in and out of different functions/methods, but they don’t assure that the program is actually doing the right thing. Strong typing just helps to assure that the program isn’t doing specific categories of the wrong thing (in the same way that Rust and Go make it difficult to write memory-unsafe code). I mean, Go and Rust both have testing capabilities built in. If it wasn’t necessary, why would they do that? Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.
- jiggawatts 5y agoConversely, I've actually never seen automated tests "in the wild". Every project I've come across has had zero tests, or close to it. I'm dealing with a migration of a legacy code base right now, today, that could probably use some tests! But even suggesting this is a complete non-starter: The time required to write the tests vastly exceeds the time and budget allocated to the migration project. I'll simply enable an APM tool that costs $50 a month and call it a day. If something crashes in UAT we'll catch it. If not, we'll definitely catch it in PRD. If not, then it's probably not worth dealing with! I've added tests to some of my projects in the past and caught virtually nothing with them. I always run my code "through its paces" at least a few times with a REPL, debugger, or some sort of tracing tool before it's ever committed to the repo. The kind of bugs I find in production would never be found by typical automated test suites. Things like logic errors due to misunderstanding the requirements, or deadlocks in a database that only manifest with an obscure multi-user workflow combination. As a random example, a significant issue in one codebase was that a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions. Will your tests find this issue? What if I told you nobody noticed for years, and that the processing where this manifested takes 4 hours? Will your dev team still be productive if changes require 4-hour tests every time they hit build or check in the code? To find a one-time issue that won't reoccur now that it's been fixed? Really? Small, fast tests often find nothing and are worthless because they don't test anything of interest. Big complicated tests are time consuming to write, still often find nothing, and slow down processes.
- albertopv 5y agoI worked on a project with no tests. We had to refactor a specific functionality changing database entities, augmenting algorithms and adding behaviour. Hundreds of possible cases, no way to be tested manually. I refactor all the code to be "testable", I wrote hundreds of tests. I wouldn't have done it without tests. In the same project, a single algorithm had a dozen of parameters, dozens of possible scenarios. An issue occured in prod. Luckily algorithm had over 60 tests, we added a red test, fixed, run all green and we were very confident of the release, same confidence not possible without tests.