11 ms·

No. Tests are like any other code. They incur technical debt and bugs at the same rate as other code. They also introduce friction to the development process. A

No. Tests are like any other code. They incur technical debt and bugs at the same rate as other code. They also introduce friction to the development process. As your test suite grows, your dev process often begins to slow down unless you apply additional work to grease the wheels, which is yet another often unmeasured cost of testing in this fashion. So, in short, I view tests as a super useful, but over-applied tool. I want my tests to deliver high enough value to warrant their ongoing maintenance and costs. That means I don't write nearly as many tests as I used to (in my own projects), and far fewer than my peers. Where I work, tests are practically mandated for everything, and a full CI run takes hours, even when distributed across 20 machines. Anecdotally, I've worked for companies that test super heavily, and I've worked for companies that had no automated tests at all. (They tested manually before releases.) The ratio of production issues across all of my jobs is roughly flat. This issue tends to trigger people. It's like religion or global warming or any other hot-button issue. It would be interesting to try to come up with some statistical analysis of the costs / benefit of automated tests.

mipmap04 7y ago
I used to be a TDD zealot. In recent years, I've taken a much more selective approach to test coverage. I typically focus my testing on pieces of code that contain business logic whereas I used to test everything. I've also found automated UI testing is not worth the squeeze and I've had better luck just looking at impacted objects and manually testing those. I'd be interested to hear if anyone has automated UI testing tools in place that are easier to write test cases for than to just do the manual testing.
- Trisell 7y ago
  I’ve had really good luck with react UI testing using Jest and React Testing Library and throwing in UI screenshot testing. At my previous job we eliminated selenium from our testing suite. We found that good UI unit and integration testing caught 99% of the bugs and the bugs that did make it through most likely wouldn’t have been caught by selenium so the added time wasn’t worth it.
  - SkyPuncher 7y ago
    IMO, snapshots, like Jest/React, are one of my favorite ways to test UIs. They require minimal effort and address the major points of testing a UI. Actual pixel "perfect" UI testing should still be done from time to time in a real browser. It's nearly impossible to properly capture some of the layout differences/bug that can come simply from a new browser version.
    - Trisell 7y ago
      We ran screenshots and looked for a percentage same. Because 100% is almost impossible. Where it saved us over snapshots was we were using MUI and they introduced a breaking UI change that our snapshots didn’t catch the change. And the screenshot did because the menu was no longer hidden. But the code hadn’t changed.
- andrethegiant 7y ago
  I'd recommend taking a look at Cypress. https://www.cypress.io/ https://www.cypress.io/
roland35 7y ago
This is a good point. Just like with anything in engineering understanding the process is more important than just following some steps because you think you should. When developing software there are many steps before testing even occurs to catch problems, and the earlier you catch problems the better. Coding standards, having requirements, and peer reviews all are important too. I find tests useful for having a "checklist" of things to do before releasing a new build. In robotics automated tests are especially helpful since there is a lot of code which only happens in certain physical conditions which are hard to recreate manually (i.e. in a low battery condition the robot should do this behavior). But just having the checklist is more important than how you execute it.
- hackerm0nkey 7y ago
  > This is a good point. Just like with anything in engineering understanding the process is more important than just following some steps because you think you should. Music to my ears. We've been all at some point or another being followers to the cargo-cult. But taking a step back and evaluating things from first principles greatly help getting one's self out of being stuck in a rut. At the end of the day, there are no silver bullets. Everything has trade off, just pick whichever is the least worst :)
agentultra 7y ago
Thankfully we don't have to. Hillel Wayne links to a few of the studies that have been done on TDD [0]. It doesn't have a conclusive effect on error rates in software. While I do write tests before I write code, more so in a dynamic language without a strong, static type system; it appears that there isn't any correlation to a reduced number of bugs. But I still do it. And I think that's because that while I may prevent a few obvious errors with the practice the benefit I get is from the informal, automated specifications of the units of a module that I'm building. I also get the benefit of a continuous refactor cycle. Less experienced developers wonder how I write clean, simple code: I refactor, a lot, all the time, because I have tests checking my code. If it's functional correctness we're after TDD is only one, small, piece of the puzzle. A strong, formal specification will go a lot further to ensuring correctness than fifty more unit tests. [0] https://www.hillelwayne.com/talks/what-we-know-we-dont-know/ https://www.hillelwayne.com/talks/what-we-know-we-dont-know/
- Cthulhu_ 7y ago
  I haven't written tests as part of my regular workflow in... years now, but one of my earlier mind blown / level up as a developer was when I wrote tests, which made it required that my code actually had to be clean and conform to a load of best practices that weren't in my mind's eye yet. The other day I was doing a part of Go Blueprints which touched upon TDD; one thing it highlighted was that if you use TDD you should only write until your test is green, no more - this (supposedly) avoids you writing too much code. It has some merits. I'll need to do a lot more TDD and co to actually see it though.
- omarhaneef 7y ago
  Don't you think that if tests are effective, they may let you go faster? To put it another way, it is not surprising that error rates are similar since, presumably, you keep coding till you produce an error, fix it and then repeat. The question is how much error free code you wrote between the errors. If testing -- or any technique for that matter, like relaxing, jogging, discussing, planning -- reduces errors, it will manifest as more functionality per error.
  - 7y ago
dmilicevic 7y ago
very good point.
jswizzy 7y ago
From what I understand about TDD it's more about getting the design right then testing for bugs.
- hackerm0nkey 7y ago
  TDD is also a design tool. I agree for newly written code, yes. Writing a test before fixing a bug reveals the bug and solidify a proof that the fix addresses it adequately. So in this context, it's less about the design (since it's already there) and more about fixing and proving that the fix works.
- eyelidlessness 7y ago
  TDD is about getting design right, but it's also about ensuring that the design's functionality is defined, demonstrated and protected from regression.
ElatedOwl 7y ago
Spot on with cost, as with everything you have to be pragmatic. Tests are great for: * High risk items (large consequence when it goes wrong) * Documentation * Weird unintuitive things We had a C# project recently that needs to detect changes between a DTOs properties. At implementation, all the comparisons would be done over value types, but if someone added a reference type that didn't properly implement equality this would silently fail (likely for months). Good case for adding a test that ensures the change detection works for all properties.
swat535 7y ago
Maybe you can help me understand this. Since you don't write as many tests, that means you're not actually testing all your code branches because tests incur technical debt after all. So does this mean you test every single branch manually? just don't bother with it at all? Do you just have a few integration tests and they break and you spend a good chunk of time figuring out which logical branch broke? What happens if you make a typo, comment out a piece of code and forget to uncomment , etc? I'd love to write less tests but don't know how to do it?
- LanceH 7y ago
  I write tests when I know the answers before to some complicated code. A good example of this is writing some code to tell if a point is inside a triangle or not. Setting up the test case is easy with three points for the triangle, a point or two inside, points outside, etc... Writing the code is simple enough, but possibly prone to flipping a sign for a slope, or maybe translating the equation wrong. If I'm writing a web app, I just don't bother to write them before hand. I mostly code until I get what I want and then write the test. I think of this as "nailing it in place". I want to make sure through the test that this code continues to produce what I expect. I'll write as many tests as I feel are necessary but probably not explore every branch.
- sdenton4 7y ago
  Yeah, as soon as code has more than two "real" branches, I don't trust myself to manually test them all. One of them will be broken quickly if I keep hacking in a particular branch. (This is, secretly, also an argument for writing code in sufficient generality to avoid this phenomenon in the first place.) I also never trust a test that passes the first time I run it. I am both terrible at writing correct code, and completely normal in that regard.
  - allannienhuis 7y ago
    I believe he's using the term branch different than you : 'alternate code path'
    - 7y ago
ebiester 7y ago
I take the perspective that tests - and particularly unit tests - are a living specification for the software that you know cannot be out of date. If you write tests in a way that you understand the business purpose of the system, you are providing a significant part of the documentation while also providing a regression suite that is automated. There are other ways to handle this. You can have a design document that is continually updated. Some teams can do this well. You can use literate programming a la Knuth. (Many code bases have abbreviated forms of this.) You can assume that the small number of developers who have been around for these decisions will never leave the company and forgo all of it. (I do not recommend this.) So, what is your preferred alternative to unit tests as a specification? (And if your set of unit tests don't provide clarity to the source code, that may be a source of the frustration.)
- deleted 7y ago
  [deleted]
voxl 7y ago
It's weird you bring up global warming when there is scientific consensus that it is not only happening but man made. Doesn't seem like a good analogy for something you want more data on.
- jfengel 7y ago
  I believe they're using it as an example of a thing that people have very strong, immovable opinions about, making it difficult to discuss. It just so happens that in the case of climate change one side is just objectively wrong, and its opinions are factually incorrect... but that just underscores the OP's use that it's impossible to discuss in public with them. They hold strong opinions in spite of overwhelming evidence against them, indicating strong feelings that will dominate any conversation you try to hold.
  - lukifer 7y ago
    I accept the reality of global warming (I'm almost a single-issue voter on the subject), but I look at it from a Bayesian point of view: if we compared two universes, one where AGW was real, and one where the scientific consensus was mistaken (not exactly unprecedented), popular opinion would probably vary by no more than 1% between the two. Humans tend to start with moral intuitions and tribal affiliations, and then cherry-pick data to support them; that irrational force is at work even in the cases where tribal values-signaling happens to align perfectly with reality.
    - RobertRoberts 7y ago
      Have you ever wondered why it became partisan to begin with? Lots of other issues are not partisan like "smoking causes cancer", everyone generally agrees that it does, even many years ago before the "science proved it". There is some science that is so obviously manipulated by power and money everyone knows something is wrong, but can't do anything about it.
      - aleister_777 7y ago
lukifer 7y ago
I'm wary of "testing theater" (in the spirit of "security theater"); but I've come to think of testing as similar to two-factor authentication: it doesn't guarantee correctness, but it does reduce the likelihood of bugs and regressions, especially during refactors. I think the other benefit of testability has nothing to do with the tests themselves, but rather the discipline of writing testable code: in general, writing code that is easy to test will tend to be higher-quality and easier to reason about. The thing I'm not fully sold on is mocking, which ends up being a huge timesink, and may or may not improve reliability since you're testing against a fake system and not the real thing. I vastly prefer a combination of small functional/unit tests, and E2E integration tests in a real environment (cypress/etc); the uncanny valley in between has a poor ROI IMO.
Bootwizard 7y ago
I've honestly thought I was crazy for thinking that required tests are ridiculous. Thank you for confirming that I'm right in my beliefs. At least 50% of my last job was writing tests, and the snail's pace of their dev process was the main reason I left.
aazaa 7y ago
> Anecdotally, I've worked for companies that test super heavily, and I've worked for companies that had no automated tests at all. How did refactoring work at that latter company?
- christophilus 7y ago
  Those companies were mostly C#, with a bit of Ruby and JavaScript. The C# type checker generally made refactoring a non issue. In the case of JavaScript and Ruby, it was not much harder. Mostly find and replace. Sometimes a bit more work, depending on the complexity of the change. In general, the code bases were pretty well designed and modular enough that refactoring was pretty simple. I tend to be a repl-driven developer, and I also use the product as I code. If you've ever watched Jonathan Blow programming on YouTube, that's similar to my process. I find the majority of issues as I go. The bugs that slip through tend to be the surprising edge cases where I think I'm just modifying a single vertical, but there's some bleeding over into other verticals which I'm not using while developing. These sorts of bugs can usually caught by a handful of basic integration tests.

mipmap04 7y ago

I used to be a TDD zealot. In recent years, I've taken a much more selective approach to test coverage. I typically focus my testing on pieces of code that contain business logic whereas I used to test everything. I've also found automated UI testing is not worth the squeeze and I've had better luck just looking at impacted objects and manually testing those. I'd be interested to hear if anyone has automated UI testing tools in place that are easier to write test cases for than to just do the manual testing.

Trisell 7y ago
I’ve had really good luck with react UI testing using Jest and React Testing Library and throwing in UI screenshot testing. At my previous job we eliminated selenium from our testing suite. We found that good UI unit and integration testing caught 99% of the bugs and the bugs that did make it through most likely wouldn’t have been caught by selenium so the added time wasn’t worth it.
- SkyPuncher 7y ago
  IMO, snapshots, like Jest/React, are one of my favorite ways to test UIs. They require minimal effort and address the major points of testing a UI. Actual pixel "perfect" UI testing should still be done from time to time in a real browser. It's nearly impossible to properly capture some of the layout differences/bug that can come simply from a new browser version.
  - Trisell 7y ago
    We ran screenshots and looked for a percentage same. Because 100% is almost impossible. Where it saved us over snapshots was we were using MUI and they introduced a breaking UI change that our snapshots didn’t catch the change. And the screenshot did because the menu was no longer hidden. But the code hadn’t changed.
andrethegiant 7y ago
I'd recommend taking a look at Cypress. https://www.cypress.io/ https://www.cypress.io/

roland35 7y ago

This is a good point. Just like with anything in engineering understanding the process is more important than just following some steps because you think you should. When developing software there are many steps before testing even occurs to catch problems, and the earlier you catch problems the better. Coding standards, having requirements, and peer reviews all are important too. I find tests useful for having a "checklist" of things to do before releasing a new build. In robotics automated tests are especially helpful since there is a lot of code which only happens in certain physical conditions which are hard to recreate manually (i.e. in a low battery condition the robot should do this behavior). But just having the checklist is more important than how you execute it.

hackerm0nkey 7y ago
> This is a good point. Just like with anything in engineering understanding the process is more important than just following some steps because you think you should. Music to my ears. We've been all at some point or another being followers to the cargo-cult. But taking a step back and evaluating things from first principles greatly help getting one's self out of being stuck in a rut. At the end of the day, there are no silver bullets. Everything has trade off, just pick whichever is the least worst :)

agentultra 7y ago

Thankfully we don't have to. Hillel Wayne links to a few of the studies that have been done on TDD [0]. It doesn't have a conclusive effect on error rates in software. While I do write tests before I write code, more so in a dynamic language without a strong, static type system; it appears that there isn't any correlation to a reduced number of bugs. But I still do it. And I think that's because that while I may prevent a few obvious errors with the practice the benefit I get is from the informal, automated specifications of the units of a module that I'm building. I also get the benefit of a continuous refactor cycle. Less experienced developers wonder how I write clean, simple code: I refactor, a lot, all the time, because I have tests checking my code. If it's functional correctness we're after TDD is only one, small, piece of the puzzle. A strong, formal specification will go a lot further to ensuring correctness than fifty more unit tests. [0] https://www.hillelwayne.com/talks/what-we-know-we-dont-know/ https://www.hillelwayne.com/talks/what-we-know-we-dont-know/

Cthulhu_ 7y ago
I haven't written tests as part of my regular workflow in... years now, but one of my earlier mind blown / level up as a developer was when I wrote tests, which made it required that my code actually had to be clean and conform to a load of best practices that weren't in my mind's eye yet. The other day I was doing a part of Go Blueprints which touched upon TDD; one thing it highlighted was that if you use TDD you should only write until your test is green, no more - this (supposedly) avoids you writing too much code. It has some merits. I'll need to do a lot more TDD and co to actually see it though.
omarhaneef 7y ago
Don't you think that if tests are effective, they may let you go faster? To put it another way, it is not surprising that error rates are similar since, presumably, you keep coding till you produce an error, fix it and then repeat. The question is how much error free code you wrote between the errors. If testing -- or any technique for that matter, like relaxing, jogging, discussing, planning -- reduces errors, it will manifest as more functionality per error.
- 7y ago

dmilicevic 7y ago

very good point.

jswizzy 7y ago

From what I understand about TDD it's more about getting the design right then testing for bugs.

hackerm0nkey 7y ago
TDD is also a design tool. I agree for newly written code, yes. Writing a test before fixing a bug reveals the bug and solidify a proof that the fix addresses it adequately. So in this context, it's less about the design (since it's already there) and more about fixing and proving that the fix works.
eyelidlessness 7y ago
TDD is about getting design right, but it's also about ensuring that the design's functionality is defined, demonstrated and protected from regression.

ElatedOwl 7y ago

Spot on with cost, as with everything you have to be pragmatic. Tests are great for: * High risk items (large consequence when it goes wrong) * Documentation * Weird unintuitive things We had a C# project recently that needs to detect changes between a DTOs properties. At implementation, all the comparisons would be done over value types, but if someone added a reference type that didn't properly implement equality this would silently fail (likely for months). Good case for adding a test that ensures the change detection works for all properties.

swat535 7y ago

Maybe you can help me understand this. Since you don't write as many tests, that means you're not actually testing all your code branches because tests incur technical debt after all. So does this mean you test every single branch manually? just don't bother with it at all? Do you just have a few integration tests and they break and you spend a good chunk of time figuring out which logical branch broke? What happens if you make a typo, comment out a piece of code and forget to uncomment , etc? I'd love to write less tests but don't know how to do it?

LanceH 7y ago
I write tests when I know the answers before to some complicated code. A good example of this is writing some code to tell if a point is inside a triangle or not. Setting up the test case is easy with three points for the triangle, a point or two inside, points outside, etc... Writing the code is simple enough, but possibly prone to flipping a sign for a slope, or maybe translating the equation wrong. If I'm writing a web app, I just don't bother to write them before hand. I mostly code until I get what I want and then write the test. I think of this as "nailing it in place". I want to make sure through the test that this code continues to produce what I expect. I'll write as many tests as I feel are necessary but probably not explore every branch.
sdenton4 7y ago
Yeah, as soon as code has more than two "real" branches, I don't trust myself to manually test them all. One of them will be broken quickly if I keep hacking in a particular branch. (This is, secretly, also an argument for writing code in sufficient generality to avoid this phenomenon in the first place.) I also never trust a test that passes the first time I run it. I am both terrible at writing correct code, and completely normal in that regard.
- allannienhuis 7y ago
  I believe he's using the term branch different than you : 'alternate code path'
  - 7y ago

ebiester 7y ago

I take the perspective that tests - and particularly unit tests - are a living specification for the software that you know cannot be out of date. If you write tests in a way that you understand the business purpose of the system, you are providing a significant part of the documentation while also providing a regression suite that is automated. There are other ways to handle this. You can have a design document that is continually updated. Some teams can do this well. You can use literate programming a la Knuth. (Many code bases have abbreviated forms of this.) You can assume that the small number of developers who have been around for these decisions will never leave the company and forgo all of it. (I do not recommend this.) So, what is your preferred alternative to unit tests as a specification? (And if your set of unit tests don't provide clarity to the source code, that may be a source of the frustration.)

deleted 7y ago
[deleted]

voxl 7y ago

It's weird you bring up global warming when there is scientific consensus that it is not only happening but man made. Doesn't seem like a good analogy for something you want more data on.

jfengel 7y ago
I believe they're using it as an example of a thing that people have very strong, immovable opinions about, making it difficult to discuss. It just so happens that in the case of climate change one side is just objectively wrong, and its opinions are factually incorrect... but that just underscores the OP's use that it's impossible to discuss in public with them. They hold strong opinions in spite of overwhelming evidence against them, indicating strong feelings that will dominate any conversation you try to hold.
- lukifer 7y ago
  I accept the reality of global warming (I'm almost a single-issue voter on the subject), but I look at it from a Bayesian point of view: if we compared two universes, one where AGW was real, and one where the scientific consensus was mistaken (not exactly unprecedented), popular opinion would probably vary by no more than 1% between the two. Humans tend to start with moral intuitions and tribal affiliations, and then cherry-pick data to support them; that irrational force is at work even in the cases where tribal values-signaling happens to align perfectly with reality.
  - RobertRoberts 7y ago
    Have you ever wondered why it became partisan to begin with? Lots of other issues are not partisan like "smoking causes cancer", everyone generally agrees that it does, even many years ago before the "science proved it". There is some science that is so obviously manipulated by power and money everyone knows something is wrong, but can't do anything about it.
    - aleister_777 7y ago

lukifer 7y ago

I'm wary of "testing theater" (in the spirit of "security theater"); but I've come to think of testing as similar to two-factor authentication: it doesn't guarantee correctness, but it does reduce the likelihood of bugs and regressions, especially during refactors. I think the other benefit of testability has nothing to do with the tests themselves, but rather the discipline of writing testable code: in general, writing code that is easy to test will tend to be higher-quality and easier to reason about. The thing I'm not fully sold on is mocking, which ends up being a huge timesink, and may or may not improve reliability since you're testing against a fake system and not the real thing. I vastly prefer a combination of small functional/unit tests, and E2E integration tests in a real environment (cypress/etc); the uncanny valley in between has a poor ROI IMO.

Bootwizard 7y ago

I've honestly thought I was crazy for thinking that required tests are ridiculous. Thank you for confirming that I'm right in my beliefs. At least 50% of my last job was writing tests, and the snail's pace of their dev process was the main reason I left.

aazaa 7y ago

> Anecdotally, I've worked for companies that test super heavily, and I've worked for companies that had no automated tests at all. How did refactoring work at that latter company?

christophilus 7y ago
Those companies were mostly C#, with a bit of Ruby and JavaScript. The C# type checker generally made refactoring a non issue. In the case of JavaScript and Ruby, it was not much harder. Mostly find and replace. Sometimes a bit more work, depending on the complexity of the change. In general, the code bases were pretty well designed and modular enough that refactoring was pretty simple. I tend to be a repl-driven developer, and I also use the product as I code. If you've ever watched Jonathan Blow programming on YouTube, that's similar to my process. I find the majority of issues as I go. The bugs that slip through tend to be the surprising edge cases where I think I'm just modifying a single vertical, but there's some bleeding over into other verticals which I'm not using while developing. These sorts of bugs can usually caught by a handful of basic integration tests.