4 ms·

I don't agree with the implicit premise that if the cheaper consultants had been required to maintain the tests and provide coverage for what they wrote, the co

by joeld42 11y ago

I don't agree with the implicit premise that if the cheaper consultants had been required to maintain the tests and provide coverage for what they wrote, the code quality would have stayed high. In my experience, it would have turned into just as much of a mess, except would have taken three times as much work to maintain all the tests in the process. Maybe the tests would have had some benefits, but would they have enough to justify all the extra time to write/maintain them? Certainly not. In my experience, good programmers write good code, tests or not, and bad programmers write bad code, tests or not. I'm not against testing altogether, it makes sense in some contexts (like very widely used libraries). It's also useful for beginners who aren't sure "where to start" with a project. But even in cases where people swear that their tests are invaluable because it's tricky code that breaks often, that's a sign to me that that code needs to be made less fragile, not more well tested. TDD is a cargo cult.

joesmo 11y ago
I couldn't agree more. Almost every other form of testing, automated or manual, is more effective at catching bugs than unit testing in my experience. I rarely find unit tests catch any actual bugs (two I can remember in two years in two major codebases) and they are actually a liability rather than an asset because of that in many cases due to the time it takes to create and maintain them. Often, tests are so far removed from their actual run context with mocks and stubs that they're only testing the functionality of the mock/stub framework itself. OOP code is hardly ever unit test friendly by definition and a lot of bugs are at integration points and due to app state that won't show up in unit tests. There are few places, in other words, where they are useful outside known algorithms with expected outputs in OOP. For one bug prevented per application per year, it's easy to see why spending weeks a year on preventing said bug might not be worth it. I'd rather use them sparingly and test in other ways that are more effective at catching bugs.
- twistedpair 11y ago
  > I rarely find unit tests catch any actual bugs What kind of code are you writing? Sure, a Spring service that just integrates a bunch of other services is hard to test due to mocks galore, but any dense logic blocks should be moved to standalone, testable classes. For my math libraries, Fibonacci compression algos, etc, I find the hundreds of unit tests invaluable. Add to that, in tools like SBT, "sbt ~testOnly com.MyTest" will rerun tests on every change, giving me feedback in under a second, rather than the whole dev/test cycle of save/restart_server/click_on_UI. Write unit tests for the tricky bits that will benefit from them.
  - misja111 11y ago
    Yes, unit tests are ideal for stuff like math libraries. A math function has a specification which will stay fixed for centuries, which you can easily test with a unit test and in the mean while you can change the implementation to optimize performance. But this is not what coding looks like for the average IT job, which is more about routing data flows between a database and a web service endpoint or building web gui's. That code can also be unit tested and it can certainly have some benefit to do that while writing new code, because as other have said, it promotes a good coding style with small blocks of testable code, can serve as documentation etc. But for catching regressions, unit tests have much less value there. These unit tests will typically test only a few lines of business logic where database and web service calls have been mocked away. And chances are that the code will only need to be changed again when some requirement has changed. Which means that your unit test needs to be changed as well, or even completely rewritten. If you're not careful, maintaining your unit test suite can take you more time than updating the business logic. Which doesn't mean that you shouldn't write any tests at all, but for catching regressions, integration and system tests will have much more value.
    - AznHisoka 11y ago
      I'm a big fan of tests but have to agree with you. The average IT job is just CRUD or all the logic is in a stored procedure.
      - herge 11y ago
        > all the logic is in a stored procedure If you don't have unit tests for your stored procedures, how do you know any changes to them actually work? Run it on production data?
  - joesmo 11y ago
    These are known algorithms you're talking about, the one place where unit tests actually consistently make sense. I'm not writing a math library, however. I'm writing APIs and web GUIs. We have an immense amount of business logic and we do unit test a lot of it, but as I mentioned above, those unit tests hardly ever catch any bugs and they take a lot of time to maintain. Every time there's a change in the logic, the tests have to be updated. 999/1000 times, when a unit test fails after a chance it's because it needs to be updated, not because there is a bug. Keeping up with those updates is simply not worth the time to catch one bug per year. Neither is writing new unit tests in these areas. Writing them for well-defined algorithms with a limited (or at least definable) set of known inputs and outputs, as I mentioned in my original reply, is still useful. Other types of testing that our developers could do to make sure the code is high quality are much more useful of course and therefore preferred because they actually help make the codebase better rather than creating the illusion of doing so as unit tests often do.
- litmus 11y ago
  This is also true in my experience when writing unit tests that are forced as part of the process. I've seen situations where unit tests were prioritized over glaring critical bugs. Sure in an ideal world you would do both but life is a trade-off and there are some people who when faced with a failing project cling to conventional wisdom fanatically instead facing their fear and thinking clearly, because at the end of the day its easier writing unit tests instead of fixing the critical bugs that you haven't been able to solve for the past year. Sometimes you need to throw the baby out with the bathwater in order to realize that the baby was never in the bathtub, that it had left the bathtub a long time ago.
- LoSboccacc 11y ago
  I agree most of the tests one writes before integration only capture the most trivial stuff that seldom breaks. Now I write tests after a bug or some weird condition arise as not to fall in that trap twice. Capturing bugs as test before solving them also ensure the fix is proper and doesn't regress other stuff. The real issue is to make sure if anything happens, tests can replicate that
lifeisstillgood 11y ago
> good programmers write good code, tests or not, and bad programmers write bad code, tests or not. I would suggest TDD or not. Vast swathes of unit tests that are unrelated to the API/contract or not But yes, that's the essence.
- collyw 11y ago
  Testing the framework seems to be a common unwritten theme in unit testing tutorials I have seen
hardwaresofton 11y ago
So I agree with the sentiment (tests are not a panacea, author says the same), but the first sentence is wrong. A lot of ink gets spilled about TDD on the internet... Hoping not to rehash all that stuff, I think I can put forward the point simply: Tests are one of the best (only?) ways to prevent breaking changes to a codebase. If you want to ensure some functionality, a regression test (whether that's unit, integration, or end to end) is the way to do it. Of course, you can't stop someone from coming along and changing the test and invalidating it, but if written correctly a test will offer a 100% guarantee that the codebase fulfills certain requirements. It doesn't matter how much of a mess is made, what matters is that the system remains running at the end of the day (it just gets harder and harder to change, and runs worse and worse). TDD may be a cargo cult, but don't throw the baby out with the bath water -- tests are a useful tool in a software developer's repertoire. This fact should become more and more apparent for any developer who has written lots of software. A good developer, in anticipation of the fact that they will eventually write a bug/commit some regression in functionality, will guard themselves.
- jxramos 11y ago
  and besides its much cheaper along many dimensions to find a bug early through testing than as possibly far late as out in the field.
- LoSboccacc 11y ago
  Prepare to spend half time writing test or double time fixing bugs - the mythical man month
jonesb6 11y ago
I want the "does not leak environmental variables in production" test to stay in my codebase, thank you.
ergothus 11y ago
I have to to disagree. Tests _promote_ good coding behavior. Yes, people can circumvent them. Yes, people can write terrible tests. Yes, people could theoretically write great code without tests (but most won't). But if you're doing TDD with the intention of doing it right, you'll find code smells early, you'll tend to write things in a more change-tolerant way the first time, and you'll get some extra confidence in your code. There's a reason switching to TDD from not using tests takes time and effort (general estimates say 6 months before you reach previous productivity levels) - this is the time in which you learning to code better. After which, your productivity tends to exceed your previous values. I went through this a couple of years ago when TDD became something that moved from "it sounds cool but I don't have time for that" to something I could really do. It was hard to learn, but in the end I found that instead of writing code to pass tests, I wrote code that I liked, and the tests promoted those skills/behaviors. Much of the pain people have from writing tests (in my limited experience) stems from: * They are trying to write their tests after writing the code, and while their code may be "okay", it's not as free of side effects and external state as it could be == hard to test. * Poor general education as to HOW to write good tests. I recall a few weeks where I tried to reconcile advice from person who said "don't have tests for simple existence" and another that said "write tests as you code". Eventually I figured out that the red/green/refactor cycle means you CAN and SHOULD delete redundant tests, but there's no reason not to write them as you start. * Tests can be hard for exploratory code. Again, this experience. Once you find the proper balance, tests are not only NOT a cost, but a BENEFIT.
- joeld42 11y ago
  "After which, your productivity tends to exceed your previous values." Really, this is the only bit that I disagree with. Tests do promote good code behavior, and they can be useful for certain cases (another poster mentioned math libraries, which are a good case for them.) And they are good for learning how to code and learning to identify "code smells". But once a programmer has learned those skills, maybe they can stop wasting time writing tests. In my experience, the real productivity gains from unit testing are less than the productivity drag from writing and maintaining the tests. Not to mention the upfront cost of bringing a messy codebase under test. Tests also often require contorting your codebase in weird ways to make it testable. Sometimes this is an improvement and makes things more modular, sometimes it's just adding layers and indirections for no real reason other than "testability". If your codebase is breaking all the time, you've got problems. But maybe those problems are sloppy programmers, lack of code review and badly designed codebase. Unit testing is one way to address those problems, but discipline, experience, code reviews, firing bad coders, mentoring and not being sloppy are other ways that are often overlooked. Every team and organization varies, and if unit testing works for you, great. I just feel like it's becoming dogma, rather than simply another tool or strategy.

by joeld42 11y ago

joesmo 11y ago

I couldn't agree more. Almost every other form of testing, automated or manual, is more effective at catching bugs than unit testing in my experience. I rarely find unit tests catch any actual bugs (two I can remember in two years in two major codebases) and they are actually a liability rather than an asset because of that in many cases due to the time it takes to create and maintain them. Often, tests are so far removed from their actual run context with mocks and stubs that they're only testing the functionality of the mock/stub framework itself. OOP code is hardly ever unit test friendly by definition and a lot of bugs are at integration points and due to app state that won't show up in unit tests. There are few places, in other words, where they are useful outside known algorithms with expected outputs in OOP. For one bug prevented per application per year, it's easy to see why spending weeks a year on preventing said bug might not be worth it. I'd rather use them sparingly and test in other ways that are more effective at catching bugs.

twistedpair 11y ago
> I rarely find unit tests catch any actual bugs What kind of code are you writing? Sure, a Spring service that just integrates a bunch of other services is hard to test due to mocks galore, but any dense logic blocks should be moved to standalone, testable classes. For my math libraries, Fibonacci compression algos, etc, I find the hundreds of unit tests invaluable. Add to that, in tools like SBT, "sbt ~testOnly com.MyTest" will rerun tests on every change, giving me feedback in under a second, rather than the whole dev/test cycle of save/restart_server/click_on_UI. Write unit tests for the tricky bits that will benefit from them.
- misja111 11y ago
  Yes, unit tests are ideal for stuff like math libraries. A math function has a specification which will stay fixed for centuries, which you can easily test with a unit test and in the mean while you can change the implementation to optimize performance. But this is not what coding looks like for the average IT job, which is more about routing data flows between a database and a web service endpoint or building web gui's. That code can also be unit tested and it can certainly have some benefit to do that while writing new code, because as other have said, it promotes a good coding style with small blocks of testable code, can serve as documentation etc. But for catching regressions, unit tests have much less value there. These unit tests will typically test only a few lines of business logic where database and web service calls have been mocked away. And chances are that the code will only need to be changed again when some requirement has changed. Which means that your unit test needs to be changed as well, or even completely rewritten. If you're not careful, maintaining your unit test suite can take you more time than updating the business logic. Which doesn't mean that you shouldn't write any tests at all, but for catching regressions, integration and system tests will have much more value.
  - AznHisoka 11y ago
    I'm a big fan of tests but have to agree with you. The average IT job is just CRUD or all the logic is in a stored procedure.
    - herge 11y ago
      > all the logic is in a stored procedure If you don't have unit tests for your stored procedures, how do you know any changes to them actually work? Run it on production data?
- joesmo 11y ago
  These are known algorithms you're talking about, the one place where unit tests actually consistently make sense. I'm not writing a math library, however. I'm writing APIs and web GUIs. We have an immense amount of business logic and we do unit test a lot of it, but as I mentioned above, those unit tests hardly ever catch any bugs and they take a lot of time to maintain. Every time there's a change in the logic, the tests have to be updated. 999/1000 times, when a unit test fails after a chance it's because it needs to be updated, not because there is a bug. Keeping up with those updates is simply not worth the time to catch one bug per year. Neither is writing new unit tests in these areas. Writing them for well-defined algorithms with a limited (or at least definable) set of known inputs and outputs, as I mentioned in my original reply, is still useful. Other types of testing that our developers could do to make sure the code is high quality are much more useful of course and therefore preferred because they actually help make the codebase better rather than creating the illusion of doing so as unit tests often do.
litmus 11y ago
This is also true in my experience when writing unit tests that are forced as part of the process. I've seen situations where unit tests were prioritized over glaring critical bugs. Sure in an ideal world you would do both but life is a trade-off and there are some people who when faced with a failing project cling to conventional wisdom fanatically instead facing their fear and thinking clearly, because at the end of the day its easier writing unit tests instead of fixing the critical bugs that you haven't been able to solve for the past year. Sometimes you need to throw the baby out with the bathwater in order to realize that the baby was never in the bathtub, that it had left the bathtub a long time ago.
LoSboccacc 11y ago
I agree most of the tests one writes before integration only capture the most trivial stuff that seldom breaks. Now I write tests after a bug or some weird condition arise as not to fall in that trap twice. Capturing bugs as test before solving them also ensure the fix is proper and doesn't regress other stuff. The real issue is to make sure if anything happens, tests can replicate that

lifeisstillgood 11y ago

> good programmers write good code, tests or not, and bad programmers write bad code, tests or not. I would suggest TDD or not. Vast swathes of unit tests that are unrelated to the API/contract or not But yes, that's the essence.

collyw 11y ago
Testing the framework seems to be a common unwritten theme in unit testing tutorials I have seen

hardwaresofton 11y ago

So I agree with the sentiment (tests are not a panacea, author says the same), but the first sentence is wrong. A lot of ink gets spilled about TDD on the internet... Hoping not to rehash all that stuff, I think I can put forward the point simply: Tests are one of the best (only?) ways to prevent breaking changes to a codebase. If you want to ensure some functionality, a regression test (whether that's unit, integration, or end to end) is the way to do it. Of course, you can't stop someone from coming along and changing the test and invalidating it, but if written correctly a test will offer a 100% guarantee that the codebase fulfills certain requirements. It doesn't matter how much of a mess is made, what matters is that the system remains running at the end of the day (it just gets harder and harder to change, and runs worse and worse). TDD may be a cargo cult, but don't throw the baby out with the bath water -- tests are a useful tool in a software developer's repertoire. This fact should become more and more apparent for any developer who has written lots of software. A good developer, in anticipation of the fact that they will eventually write a bug/commit some regression in functionality, will guard themselves.

jxramos 11y ago
and besides its much cheaper along many dimensions to find a bug early through testing than as possibly far late as out in the field.
LoSboccacc 11y ago
Prepare to spend half time writing test or double time fixing bugs - the mythical man month

jonesb6 11y ago

I want the "does not leak environmental variables in production" test to stay in my codebase, thank you.

ergothus 11y ago

I have to to disagree. Tests _promote_ good coding behavior. Yes, people can circumvent them. Yes, people can write terrible tests. Yes, people could theoretically write great code without tests (but most won't). But if you're doing TDD with the intention of doing it right, you'll find code smells early, you'll tend to write things in a more change-tolerant way the first time, and you'll get some extra confidence in your code. There's a reason switching to TDD from not using tests takes time and effort (general estimates say 6 months before you reach previous productivity levels) - this is the time in which you learning to code better. After which, your productivity tends to exceed your previous values. I went through this a couple of years ago when TDD became something that moved from "it sounds cool but I don't have time for that" to something I could really do. It was hard to learn, but in the end I found that instead of writing code to pass tests, I wrote code that I liked, and the tests promoted those skills/behaviors. Much of the pain people have from writing tests (in my limited experience) stems from: * They are trying to write their tests after writing the code, and while their code may be "okay", it's not as free of side effects and external state as it could be == hard to test. * Poor general education as to HOW to write good tests. I recall a few weeks where I tried to reconcile advice from person who said "don't have tests for simple existence" and another that said "write tests as you code". Eventually I figured out that the red/green/refactor cycle means you CAN and SHOULD delete redundant tests, but there's no reason not to write them as you start. * Tests can be hard for exploratory code. Again, this experience. Once you find the proper balance, tests are not only NOT a cost, but a BENEFIT.

joeld42 11y ago
"After which, your productivity tends to exceed your previous values." Really, this is the only bit that I disagree with. Tests do promote good code behavior, and they can be useful for certain cases (another poster mentioned math libraries, which are a good case for them.) And they are good for learning how to code and learning to identify "code smells". But once a programmer has learned those skills, maybe they can stop wasting time writing tests. In my experience, the real productivity gains from unit testing are less than the productivity drag from writing and maintaining the tests. Not to mention the upfront cost of bringing a messy codebase under test. Tests also often require contorting your codebase in weird ways to make it testable. Sometimes this is an improvement and makes things more modular, sometimes it's just adding layers and indirections for no real reason other than "testability". If your codebase is breaking all the time, you've got problems. But maybe those problems are sloppy programmers, lack of code review and badly designed codebase. Unit testing is one way to address those problems, but discipline, experience, code reviews, firing bad coders, mentoring and not being sloppy are other ways that are often overlooked. Every team and organization varies, and if unit testing works for you, great. I just feel like it's becoming dogma, rather than simply another tool or strategy.