3 ms·

Types do a great job at assuring you’re getting generally the right kind of data in and out of different functions/methods, but they don’t assure that the progr

Types do a great job at assuring you’re getting generally the right kind of data in and out of different functions/methods, but they don’t assure that the program is actually doing the right thing. Strong typing just helps to assure that the program isn’t doing specific categories of the wrong thing (in the same way that Rust and Go make it difficult to write memory-unsafe code). I mean, Go and Rust both have testing capabilities built in. If it wasn’t necessary, why would they do that? Beyond the most trivial of projects, I can’t think of a single project that I’ve worked on (or even looked at on GitHub) that has weighed the costs and benefits of automated testing and decided it just wasn’t worth it.

jiggawatts 5y ago
Conversely, I've actually never seen automated tests "in the wild". Every project I've come across has had zero tests, or close to it. I'm dealing with a migration of a legacy code base right now, today, that could probably use some tests! But even suggesting this is a complete non-starter: The time required to write the tests vastly exceeds the time and budget allocated to the migration project. I'll simply enable an APM tool that costs $50 a month and call it a day. If something crashes in UAT we'll catch it. If not, we'll definitely catch it in PRD. If not, then it's probably not worth dealing with! I've added tests to some of my projects in the past and caught virtually nothing with them. I always run my code "through its paces" at least a few times with a REPL, debugger, or some sort of tracing tool before it's ever committed to the repo. The kind of bugs I find in production would never be found by typical automated test suites. Things like logic errors due to misunderstanding the requirements, or deadlocks in a database that only manifest with an obscure multi-user workflow combination. As a random example, a significant issue in one codebase was that a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions. Will your tests find this issue? What if I told you nobody noticed for years, and that the processing where this manifested takes 4 hours? Will your dev team still be productive if changes require 4-hour tests every time they hit build or check in the code? To find a one-time issue that won't reoccur now that it's been fixed? Really? Small, fast tests often find nothing and are worthless because they don't test anything of interest. Big complicated tests are time consuming to write, still often find nothing, and slow down processes.
- albertopv 5y ago
  I worked on a project with no tests. We had to refactor a specific functionality changing database entities, augmenting algorithms and adding behaviour. Hundreds of possible cases, no way to be tested manually. I refactor all the code to be "testable", I wrote hundreds of tests. I wouldn't have done it without tests. In the same project, a single algorithm had a dozen of parameters, dozens of possible scenarios. An issue occured in prod. Luckily algorithm had over 60 tests, we added a red test, fixed, run all green and we were very confident of the release, same confidence not possible without tests.
  - chris11 5y ago
    +1. That's one of the first things I look for if I need to refactor something. I want tests to exist to verify refactoring doesn't change functionality.
- cweagans 5y ago
  4 hour tests are a nonstarter, but ponder this: how can you guarantee that bug will never happen again? You can be pretty sure. You can reason about the code that you’re now familiar with and infer (based on your current understanding) that something like that won’t happen again. But you can’t guarantee it. Some bugs don’t really deserve tests that run all the time - that I’ll grant you. I frequently have tests for stuff like that as a form of documentation for myself down the road. A year from now, will you remember the specific steps you took to fix your case sensitivity bug? I wouldn’t. But my VCS will. Future me will be very grateful if I write a “verify this annoying case sensitivity thing isn’t a problem” script somewhere so that if I’m seeing weird behavior in the future, I can be reasonably sure that it’s a different problem (or that my fix got reverted or overwritten). I also don’t find a lot of value in unit tests or having 100% code coverage. But that doesn’t invalidate automated testing altogether. I cannot tell you how often tests - even just simple end to end, happy path tests - have saved my ass. I’m willing to bet that you’re already testing your software manually, right? Like if you change (for the sake of example) a form, you’re probably going to submit that form a few times with different inputs. Somebody else down the road is going to do that too. Why wouldn’t you write down the different inputs you gave so that the next person has them available? Maybe that person doesn’t have the same context that you do. Maybe they just aren’t as thorough as you. Either way though: if you’ve already done the work to figure out how to fully exercise the form, why would you make someone else do that work again? The next logical step after writing that stuff down is to automate it if for no other reason than to avoid having smart, expensive people sitting around doing data entry when a computer can do it for them. That seems extremely uncontroversial to me and I have a hard time understanding why someone wouldn’t be on board with that.

jiggawatts 5y ago

Conversely, I've actually never seen automated tests "in the wild". Every project I've come across has had zero tests, or close to it. I'm dealing with a migration of a legacy code base right now, today, that could probably use some tests! But even suggesting this is a complete non-starter: The time required to write the tests vastly exceeds the time and budget allocated to the migration project. I'll simply enable an APM tool that costs $50 a month and call it a day. If something crashes in UAT we'll catch it. If not, we'll definitely catch it in PRD. If not, then it's probably not worth dealing with! I've added tests to some of my projects in the past and caught virtually nothing with them. I always run my code "through its paces" at least a few times with a REPL, debugger, or some sort of tracing tool before it's ever committed to the repo. The kind of bugs I find in production would never be found by typical automated test suites. Things like logic errors due to misunderstanding the requirements, or deadlocks in a database that only manifest with an obscure multi-user workflow combination. As a random example, a significant issue in one codebase was that a distinct sort was case-insensitive but should have been case-sensitive. It was dropping a few hundred items out of hundreds of millions. Will your tests find this issue? What if I told you nobody noticed for years, and that the processing where this manifested takes 4 hours? Will your dev team still be productive if changes require 4-hour tests every time they hit build or check in the code? To find a one-time issue that won't reoccur now that it's been fixed? Really? Small, fast tests often find nothing and are worthless because they don't test anything of interest. Big complicated tests are time consuming to write, still often find nothing, and slow down processes.

albertopv 5y ago
I worked on a project with no tests. We had to refactor a specific functionality changing database entities, augmenting algorithms and adding behaviour. Hundreds of possible cases, no way to be tested manually. I refactor all the code to be "testable", I wrote hundreds of tests. I wouldn't have done it without tests. In the same project, a single algorithm had a dozen of parameters, dozens of possible scenarios. An issue occured in prod. Luckily algorithm had over 60 tests, we added a red test, fixed, run all green and we were very confident of the release, same confidence not possible without tests.
- chris11 5y ago
  +1. That's one of the first things I look for if I need to refactor something. I want tests to exist to verify refactoring doesn't change functionality.
cweagans 5y ago
4 hour tests are a nonstarter, but ponder this: how can you guarantee that bug will never happen again? You can be pretty sure. You can reason about the code that you’re now familiar with and infer (based on your current understanding) that something like that won’t happen again. But you can’t guarantee it. Some bugs don’t really deserve tests that run all the time - that I’ll grant you. I frequently have tests for stuff like that as a form of documentation for myself down the road. A year from now, will you remember the specific steps you took to fix your case sensitivity bug? I wouldn’t. But my VCS will. Future me will be very grateful if I write a “verify this annoying case sensitivity thing isn’t a problem” script somewhere so that if I’m seeing weird behavior in the future, I can be reasonably sure that it’s a different problem (or that my fix got reverted or overwritten). I also don’t find a lot of value in unit tests or having 100% code coverage. But that doesn’t invalidate automated testing altogether. I cannot tell you how often tests - even just simple end to end, happy path tests - have saved my ass. I’m willing to bet that you’re already testing your software manually, right? Like if you change (for the sake of example) a form, you’re probably going to submit that form a few times with different inputs. Somebody else down the road is going to do that too. Why wouldn’t you write down the different inputs you gave so that the next person has them available? Maybe that person doesn’t have the same context that you do. Maybe they just aren’t as thorough as you. Either way though: if you’ve already done the work to figure out how to fully exercise the form, why would you make someone else do that work again? The next logical step after writing that stuff down is to automate it if for no other reason than to avoid having smart, expensive people sitting around doing data entry when a computer can do it for them. That seems extremely uncontroversial to me and I have a hard time understanding why someone wouldn’t be on board with that.