6 ms·

When you first hear about unit tests it seems like the greatest idea ever. Automation of a process you already do manually makes that process repeated-able and

When you first hear about unit tests it seems like the greatest idea ever. Automation of a process you already do manually makes that process repeated-able and quick. When I was a junior programmer I was all over it. Over the years I have come to reject some things that are seemingly fundamental to software engineering in the industry. One of them is unit testing. I don't believe that unit testing is harmful in itself, but excessive use of it is a sign of weakness in another area. - if your code relies on unit testing for correctness than your coding methodology (or language) is error prone. A good programming methodology where the domain is well understood usually does not need a single unit test. - If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object. This logic sort of applies integration tests which is basically testing the same thing as unit tests but touching functions that have IO. However it is a little different. If your program does Not need unit tests but needs integration tests it means things like: - your type checker does not extend across systems - the programming methodologies become more error prone as you move to other systems and you have no control over it. - you lack understanding of the foreign system. The thing with integration tests is that all of the above is largely inevitable and integration testing really becomes the only way to correlate (not verify) the entire system with correctness when you lack the integrated approach of a single programming language. Therefore because of this I am not against integration testing. I recently green fielded a small micro-service at my company and I did not write a single unit test. It had one logic error and that was largely due to a flaw in the type checker. I can imagine many engineers being aghast at the whole concept. So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong.

MattPalmer1086 7y ago
Speaking as someone who has fielded many reliable bits of software at least a decade or more before unit testing became a thing, I can't agree unit testing is a sign of bad engineering. Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!
- crimsonalucard 7y ago
  It's a sign that your programming methodology is so error prone that you need unit tests to catch errors. Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness. Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed. Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
  - bobm_kite9 7y ago
    Can you point us to some code that demonstrates how you do this?
    - crimsonalucard 7y ago
      Haskell code. I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.
war1025 7y ago
> If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all. If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.
- crimsonalucard 7y ago
  Well, "something" is being tested as your code is verifying that an input produces an output. But your philosophy is correct and we are in agreement on that. The thing I was referring to is the abstraction of dependency injection itself. It leads to code that is over complicated as the abstraction itself is not only pointless but adds unnecessary layers of complexity. There are ways to target the philosophy of what dependency injection for unit testing is trying to accomplish without introducing the side effect of unnecessary complexity that comes with dependency injection. It's as simple as just separating IO functions from core logical functions. Don't bundle these two things together into an object or a single function, that's it. (Also don't use objects period, but that's another topic.)
  - l_t 7y ago
    I like dependency injection even when I'm not writing tests (which I usually skip for side-projects.) I find it natural and convenient to think about "What dependencies on other parts of the system does this code have?" Expressing those dependencies explicitly feels like it reduces complexity, not adds it. But, I'm not talking about pulling in a big fancy DI framework, just making dependencies explicit in your function/class parameters. I will say that DI is sometimes used as a tool in overly-abstracted systems. An example that comes to mind is the ASP.NET MVC framework -- with DI and inheritance, literally any behavior can be overridden in fairly opaque ways. Trying to suss out the concrete behavior details is like swimming in quicksand. (Or it used to be that way, haven't touched ASP.NET in a while.) As an aside -- I'm curious about your programming language of choice. I think DI is a lot more useful in some PLs than others. For example, I find JS code often uses imports to create complex graphs of implicit dependencies, and DI can help tame that complexity. But for other PLs like Python or Clojure, I basically don't use DI at all.
bbrks 7y ago
The real value in unit tests comes when you need to refactor a codebase. Especially if you're not the original author. Unit tests are a codified API contract. It's easy to write something small and correct from scratch (like a greenfield microservice). It's far harder to maintain that over several years and several changes of hands.
- crimsonalucard 7y ago
  A type checker should track an aspect of this thoroughly. If you find yourself relying unit tests as a contract to prevent future changes something is wrong with your coding methodology. Too many dependencies in your code, not enough modularity. If your coding methodology was good, the type checker is enough to verify these contracts. A language without a type checker will have more need of unit tests.
  - mchaynes 7y ago
    For very simple programs, sure. Type checkers do not verify behavior. Unit tests verify behavior
    - continuational 7y ago
      Unit tests do not verify behavior, except for a few cases (out of a virtually infinite set of cases) that the programmer happened to think about. Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
      - bluGill 7y ago
        That is a very different behavior though. If my multiply function returns ADDs the input and returns an int it passes the type checker but is still wrong.
        morelisp 7y ago
adamredwoods 7y ago
My ideas of writing tests has changed over the years: - It's a must-have for business logic, because errors can lead to actual dollar costs (and looked at by the SVPs). Any maintenance change should require a unit test for this type of logic. This way our code confidence in deployment of a new feature or change goes up. - Catching regressions. We've had errors we've missed in the past, and as our team changes and team members come and go, the want to make sure we are not making the same mistakes. Thus, automated testing on regression tests helps our confidence in code stability goes up. - System failures. This is more obvious as an integration, but it's good to have these tests in place because once again they can be traced to actual dollar amounts. ALso helps confidence level of deployments by junior/new employees. - Maintenance of legacy code over YEARS. I think most people skip unit tests because they are writing new code, which gets tested over and over during development, but adding a new feature to massive legacy code is still an aspect of our jobs. I don't remember how my code worked last month let alone 3 years ago, so unit tests help not only in confidence levels, but in speed of development because I wrap mocks around any services that I don't want to wait for or should not be triggering for this rapid-iteration development. Each circumstance is different, but I think unit testing goes beyond just knowing the domain well.
ivanhoe 7y ago
Unit testing is not just about catching errors while you build the code, it's (and much more) about being able to catch errors at any time. Now, tomorrow, in 2 weeks, by someone else trying to refactor something in 10 years from now. Refactoring without tests is a nightmare and people will avoid touching anything because of it. With time that results in piles of old crap layered one over the another, hack over the hack, as no one ever wanted to take the responsibility of untangling it and refactoring the core problems properly. Many things you can do without tests, but tests make it more approachable and less scary to try.
sgtnoodle 7y ago
A lot of the software I write would be considered "safety critical" in that a crashed program could lead to injury to people and damage to property. Unit testing is by far the most valuable form of testing during development, in that it catches 90% of bugs before the code touches hardware. Also, after development, unit tests serve as regression tests that catch unintended changes made by others. Mocking, whether or not through dependency injection, is absolutely necessary to enable testing when targeting bare-metal platforms, and very helpful to speed up testing when targeting an operating system. Dependency injection in particular usually constrains code to be written in a way that, although perhaps seemingly more complex at the surface, is a lot more maintainable long term. Integration testing is also a very good idea. The thing with integration tests, though, is that they get bigger and more complicated and slower over time. Relying solely on integration testing will likely become a development bottleneck if your system grows in complexity. If you can run an integration test suite deterministically and efficiently on a development system, then the software under test probably isn't that complicated, relatively speaking. Also of note is the tendency for mocks and dependency injection to help out when integration testing as well; being able to completely abstract away hardware allows for faster than real-time deterministic simulation at a system level. If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though.
- crimsonalucard 7y ago
  >If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though. Maybe it's you who hasn't taken the red pill. Safety critical software would be even more safe if you used formal methods. Formal methods verify logic to a degree of 100% while integration tests dig out unintended side effects or leakage. Let me put it another way. In Computer science there's a field called formal methods. With formal methods it's possible to do things like dependent typing. These techniques and technologies can verify your program to 100% correctness. This is not unit testing to correlate correctness to a certain degree but actual 100% verification. Such a technique has pitfalls in the sense that it is brutally hard. I am saying that there are programming methodologies that can be used to go above standard typing and below dependent typing where the program is not verified to 100% correctness but the correlation of correctness is so high that you can forego unit tests. Some languages like Haskell and Rust already sort of enforce these programming methodologies / technologies on you such that most unit tests become redundant. You need a lot less unit tests for a haskell program than you do for javascript. Therefore a program that relies to heavily on unit tests is a sign that the programming methodology is flawed. Something is wrong.
Seenso 7y ago
> So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong. I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices. You just have to be pragmatic about how you use these practices. Teams definitely get too dogmatic and narrow minded about mock-heavy unit testing, and some people do manage to write code without writing any unit tests, however those are both extremes that are best avoided or of only limited applicability. The benefit of unit tests is that certain things are more economical to test that way. You literally can test everything though integration tests of varying levels (or manual testing!), but it'll take more effort and run more slowly. If you want to thoroughly test a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings).
- couchand 7y ago
  > If you want to thoroughly a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings). Or even write the test cases separately, then import them into a unit test runner that can check them all very fast as well as an integration test runner that can check them all very slowly.
- crimsonalucard 7y ago
  >I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices. I use documentation for communication and documentation. Unit tests may do this but this is a side effect. The purpose of unit tests is to test your own logic and find mistakes you've made. I'm saying there are ways to code where your intuition and type checker alone are enough to verify correctness without the need of a single unit test but with the same correctness as if you did verify your program with unit tests. You don't even know what this "way" is as I haven't even described it, so you can't even argue for or against it. Dogmatic is a word for those who spout their point without understanding the reasoning behind the other point.
bernawil 7y ago
- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object. what other ways?
- crimsonalucard 7y ago
  Programming is about composition of primitives. You compose primitives together to reduce the number of modules and be better able to reason about complexity. During composition you also want to maintain flexibility, you want modules to be able to be swapped out and replaced. Dependency Injection is one form of composition. One perspective to look at it, is that objects are like tiny scoped programs with variables and methods. To do a dependency injection is to put a program within a program. Is this a good way to compose primitives? It's like parsing a python program inside a java program inside javascript. What are other ways to manage complexity and compose primitives without dependencies everywhere? What is a programming primitive? Is the most fundamental primitive of computing a function? Should we be composing bricks to form buildings or compose buildings to form bigger buildings?

MattPalmer1086 7y ago

Speaking as someone who has fielded many reliable bits of software at least a decade or more before unit testing became a thing, I can't agree unit testing is a sign of bad engineering. Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!

crimsonalucard 7y ago
It's a sign that your programming methodology is so error prone that you need unit tests to catch errors. Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness. Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed. Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
- bobm_kite9 7y ago
  Can you point us to some code that demonstrates how you do this?
  - crimsonalucard 7y ago
    Haskell code. I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.

war1025 7y ago

> If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all. If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.

crimsonalucard 7y ago
Well, "something" is being tested as your code is verifying that an input produces an output. But your philosophy is correct and we are in agreement on that. The thing I was referring to is the abstraction of dependency injection itself. It leads to code that is over complicated as the abstraction itself is not only pointless but adds unnecessary layers of complexity. There are ways to target the philosophy of what dependency injection for unit testing is trying to accomplish without introducing the side effect of unnecessary complexity that comes with dependency injection. It's as simple as just separating IO functions from core logical functions. Don't bundle these two things together into an object or a single function, that's it. (Also don't use objects period, but that's another topic.)
- l_t 7y ago
  I like dependency injection even when I'm not writing tests (which I usually skip for side-projects.) I find it natural and convenient to think about "What dependencies on other parts of the system does this code have?" Expressing those dependencies explicitly feels like it reduces complexity, not adds it. But, I'm not talking about pulling in a big fancy DI framework, just making dependencies explicit in your function/class parameters. I will say that DI is sometimes used as a tool in overly-abstracted systems. An example that comes to mind is the ASP.NET MVC framework -- with DI and inheritance, literally any behavior can be overridden in fairly opaque ways. Trying to suss out the concrete behavior details is like swimming in quicksand. (Or it used to be that way, haven't touched ASP.NET in a while.) As an aside -- I'm curious about your programming language of choice. I think DI is a lot more useful in some PLs than others. For example, I find JS code often uses imports to create complex graphs of implicit dependencies, and DI can help tame that complexity. But for other PLs like Python or Clojure, I basically don't use DI at all.

bbrks 7y ago

The real value in unit tests comes when you need to refactor a codebase. Especially if you're not the original author. Unit tests are a codified API contract. It's easy to write something small and correct from scratch (like a greenfield microservice). It's far harder to maintain that over several years and several changes of hands.

crimsonalucard 7y ago
A type checker should track an aspect of this thoroughly. If you find yourself relying unit tests as a contract to prevent future changes something is wrong with your coding methodology. Too many dependencies in your code, not enough modularity. If your coding methodology was good, the type checker is enough to verify these contracts. A language without a type checker will have more need of unit tests.
- mchaynes 7y ago
  For very simple programs, sure. Type checkers do not verify behavior. Unit tests verify behavior
  - continuational 7y ago
    Unit tests do not verify behavior, except for a few cases (out of a virtually infinite set of cases) that the programmer happened to think about. Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
    - bluGill 7y ago
      That is a very different behavior though. If my multiply function returns ADDs the input and returns an int it passes the type checker but is still wrong.
      - morelisp 7y ago

adamredwoods 7y ago

My ideas of writing tests has changed over the years: - It's a must-have for business logic, because errors can lead to actual dollar costs (and looked at by the SVPs). Any maintenance change should require a unit test for this type of logic. This way our code confidence in deployment of a new feature or change goes up. - Catching regressions. We've had errors we've missed in the past, and as our team changes and team members come and go, the want to make sure we are not making the same mistakes. Thus, automated testing on regression tests helps our confidence in code stability goes up. - System failures. This is more obvious as an integration, but it's good to have these tests in place because once again they can be traced to actual dollar amounts. ALso helps confidence level of deployments by junior/new employees. - Maintenance of legacy code over YEARS. I think most people skip unit tests because they are writing new code, which gets tested over and over during development, but adding a new feature to massive legacy code is still an aspect of our jobs. I don't remember how my code worked last month let alone 3 years ago, so unit tests help not only in confidence levels, but in speed of development because I wrap mocks around any services that I don't want to wait for or should not be triggering for this rapid-iteration development. Each circumstance is different, but I think unit testing goes beyond just knowing the domain well.

ivanhoe 7y ago

Unit testing is not just about catching errors while you build the code, it's (and much more) about being able to catch errors at any time. Now, tomorrow, in 2 weeks, by someone else trying to refactor something in 10 years from now. Refactoring without tests is a nightmare and people will avoid touching anything because of it. With time that results in piles of old crap layered one over the another, hack over the hack, as no one ever wanted to take the responsibility of untangling it and refactoring the core problems properly. Many things you can do without tests, but tests make it more approachable and less scary to try.

sgtnoodle 7y ago

A lot of the software I write would be considered "safety critical" in that a crashed program could lead to injury to people and damage to property. Unit testing is by far the most valuable form of testing during development, in that it catches 90% of bugs before the code touches hardware. Also, after development, unit tests serve as regression tests that catch unintended changes made by others. Mocking, whether or not through dependency injection, is absolutely necessary to enable testing when targeting bare-metal platforms, and very helpful to speed up testing when targeting an operating system. Dependency injection in particular usually constrains code to be written in a way that, although perhaps seemingly more complex at the surface, is a lot more maintainable long term. Integration testing is also a very good idea. The thing with integration tests, though, is that they get bigger and more complicated and slower over time. Relying solely on integration testing will likely become a development bottleneck if your system grows in complexity. If you can run an integration test suite deterministically and efficiently on a development system, then the software under test probably isn't that complicated, relatively speaking. Also of note is the tendency for mocks and dependency injection to help out when integration testing as well; being able to completely abstract away hardware allows for faster than real-time deterministic simulation at a system level. If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though.

crimsonalucard 7y ago
>If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though. Maybe it's you who hasn't taken the red pill. Safety critical software would be even more safe if you used formal methods. Formal methods verify logic to a degree of 100% while integration tests dig out unintended side effects or leakage. Let me put it another way. In Computer science there's a field called formal methods. With formal methods it's possible to do things like dependent typing. These techniques and technologies can verify your program to 100% correctness. This is not unit testing to correlate correctness to a certain degree but actual 100% verification. Such a technique has pitfalls in the sense that it is brutally hard. I am saying that there are programming methodologies that can be used to go above standard typing and below dependent typing where the program is not verified to 100% correctness but the correlation of correctness is so high that you can forego unit tests. Some languages like Haskell and Rust already sort of enforce these programming methodologies / technologies on you such that most unit tests become redundant. You need a lot less unit tests for a haskell program than you do for javascript. Therefore a program that relies to heavily on unit tests is a sign that the programming methodology is flawed. Something is wrong.

Seenso 7y ago

> So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong. I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices. You just have to be pragmatic about how you use these practices. Teams definitely get too dogmatic and narrow minded about mock-heavy unit testing, and some people do manage to write code without writing any unit tests, however those are both extremes that are best avoided or of only limited applicability. The benefit of unit tests is that certain things are more economical to test that way. You literally can test everything though integration tests of varying levels (or manual testing!), but it'll take more effort and run more slowly. If you want to thoroughly test a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings).

couchand 7y ago
> If you want to thoroughly a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings). Or even write the test cases separately, then import them into a unit test runner that can check them all very fast as well as an integration test runner that can check them all very slowly.
crimsonalucard 7y ago
>I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices. I use documentation for communication and documentation. Unit tests may do this but this is a side effect. The purpose of unit tests is to test your own logic and find mistakes you've made. I'm saying there are ways to code where your intuition and type checker alone are enough to verify correctness without the need of a single unit test but with the same correctness as if you did verify your program with unit tests. You don't even know what this "way" is as I haven't even described it, so you can't even argue for or against it. Dogmatic is a word for those who spout their point without understanding the reasoning behind the other point.

bernawil 7y ago

- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object. what other ways?

crimsonalucard 7y ago
Programming is about composition of primitives. You compose primitives together to reduce the number of modules and be better able to reason about complexity. During composition you also want to maintain flexibility, you want modules to be able to be swapped out and replaced. Dependency Injection is one form of composition. One perspective to look at it, is that objects are like tiny scoped programs with variables and methods. To do a dependency injection is to put a program within a program. Is this a good way to compose primitives? It's like parsing a python program inside a java program inside javascript. What are other ways to manage complexity and compose primitives without dependencies everywhere? What is a programming primitive? Is the most fundamental primitive of computing a function? Should we be composing bricks to form buildings or compose buildings to form bigger buildings?