5 ms·

// Tests waste a lot of time Back when I was a developer, I had a manager who was super religious about unit tests (this seemed incongruous to me, I was a seni

// Tests waste a lot of time Back when I was a developer, I had a manager who was super religious about unit tests (this seemed incongruous to me, I was a senior lead at the time and it was my vibe that such tests weren't what the project really needed the most at the time) The basic "problem" I have with tests is that - at least maybe in my experience - things tend to go wrong in cases that you haven't thought about so your tests don't cover them anyway. Like, what's the point of asserting that add(2,2) will yield 4 if you've already been thinking about that. And then, if you've already thought more complicated cases (eg negative numbers, cases that overflow) then your code already very likely handles them. And if you hadn't thought about them, your tests won't cover them either. I get it, it's cool to be able to run a test suite especially if I refactored something and be like "yup, add(2,2) is still 4!) but not sure how much actual value vs cost/annoyance I've seen from it.

pixl97 4y ago
I'm not really a developer here, more of the wedge between the customer and developer when things go wrong, but don't these tests cover the continuous updating of the application as specifications change? Especially as multiple people are working on the code and may have different ideas of what's going on. May not catch everything, but can trigger a warning in many cases.
- xyzelement 4y ago
  Ah - I can see those kinds of tests as useful. I guess they are something like "integration tests" or something like that which tests the user-visible outputs of the system. My comment was about very low-level unit testing, where it's literally every function of code must be covered, isolated from other parts of code.
- crazygringo 4y ago
  Exactly this. Also in codebases where the authors seemed opposed to writing good comments, tests are extremely useful in documenting expected behavior in edge cases, how an optional argument is supposed to be dealt with when missing, etc. Not to mention that people are often a lot less rigorous with "manual testing" than they should be. They don't think of scenarios, they skip scenarios, because nobody's checking on them. While when you write tests, you're creating a permanent record where you can be held accountable if you miss something.
  - pixl97 4y ago
    >They don't think of scenarios, they skip scenarios, Bane of my life. Me: "Hey, this big customer we're not going to tell to change their behaviors and pays us a lot of money is using this product in this particular way. We should put in a functional test case to ensure that we don't mess this up" Dev/QA: "Eh, no" This generally leads to a support call half a year later with the customer panicking, support groaning, and the developers panicking because they 'could have never predicted the customer using the application in this manner'.
  - kraftman 4y ago
    unit tests are comments that are forced to stay up to date and attached to the code they are for.
  - hbrn 4y ago
    > in codebases where the authors seemed opposed to writing good comments, tests are extremely useful in documenting expected behavior in edge cases In other words, tests are useful in teams with awful engineering culture. I tend to agree, in my experience they are almost synonymous. If you treat engineering culture as weather (i.e. something you have no control over), then tests are a good tool to address your problems. > you're creating a permanent record where you can be held accountable if you miss something Bureaucracy is another word for it. You know who excels at it? Government. And you know what government is really bad at? Running business and innovating.
PathOfEclipse 4y ago
I attribute some of the greatest successes of my career in part to having good unit test coverage. I have seen no other pragmatic way to solve the problem that, in the worst case, a minor software update can be O(N) expensive to make, where N is the size or complexity of your program. That little update has the habit of breaking things in code paths you'd never expect. With a good suite of unit tests, you can validate that your program likely still functions, and, far more importantly, you can do the validation quickly, which results fast feedback loops and iteration times. Fast feedback loops are essential to high developer productivity. When developers are being slow, in my experience it usually is related to the system they are working on having slow feedback cycles. > The basic "problem" I have with tests is that - at least maybe in my experience - things tend to go wrong in cases that you haven't thought about so your tests don't cover them anyway. You are right in that unit testing can't make up for bad engineering. One advantage of investing time in it, though, is it gives you potentially more opportunity to think through those cases. And, as a consolation, prize, you write unit tests for bugs that make it through and now you have a great regression suite. Furthermore, a major issue that often comes when unit tests are omitted or not taken seriously, is that these tests can't easily be added after the fact, A system has to be designed with unit testing in mind, or else you end up with a bunch of untestable code. I believe that many legacy codebases remain without tests because the act of refactoring them enough to support any testing at all would be both time-consuming and high risk in terms of potential breakage.
- trgn 4y ago
  I agree 100%, but just want to add my2c. Unit tests are in general never wasted, in the sense that they trap regressions before release. I don't think they necessarily are good to validate work though, or at least, they do so poorly. It's easy to write lots of tests which don't cover the areas your program will bork on in production. Which I think is point of GP. Value of tests compound over time. The longer the software is in production and maintained, the more work the unit tests have been doing. It could be a paltry 20% code coverage set of tests, but nonetheless they're fighting the good fight. The value of a test suite is expressed in the integral over time. Few tests can do a lot of work; on the flip side, a lot of tests are useless when they're continuously thrown away due to pivots in business logic.
sitkack 4y ago
The system using the libraries should exercise those libraries sufficiently that integration tests catch those kinds of errors. If you are allowed only one test, it should be an integration test that runs in under XX seconds so you can maintain flow and correctness. If a system is under constant churn, test at the interface boundaries, unit tests are further burden during refactoring. Unit tests ensure that the internals function in a certain manner, external users should be concerned with internals. Unit tests are extremely useful in the core of a system like an evaluator, a rules engine, timezone, geospatial, computational geometry, etc. Your type system should capture many of the invariants in your system. I know what you are trying to convey, but the example `add(2,2)` encourages conversational confusion. If you are verifying hardware, that is a perfectly valid test, for a high level codebase, it is most likely a worthless test.
- sitkack 4y ago
  *edit, fixing typo* > external users shouldn't be concerned with internals
jdblair 4y ago
> And then, if you've already thought more complicated cases (eg negative numbers, cases that overflow) then your code already very likely handles them. And if you hadn't thought about them, your tests won't cover them either. Good tests check boundary conditions and edge cases. They check errors conditions (like overflow), not just the happy path. And yes, they can be a pain to write, and they affect the way your code is written. On the other hand, a test suite with good coverage can give you tremendous confidence to make sweeping changes. I have a side project, an art project, and part of that is a custom on-the-wire networking protocol. I have tests that check all the edge cases I could think of. Did I miss something? Very probably. On the other hand, I've written the protocol 3 times now: once in python, once in C, and once in C++ (the C++ case was more like a giant re-factor of the C code to make it more usable), and I'll probably write it again in rust. Each time I had confidence I got it right because my tests pass.
justin_oaks 4y ago
> what's the point of asserting that add(2,2) will yield 4 if you've already been thinking about that There have been plenty of times when I have run a test like add(2,2) and I don't get 4. Usually it's a typo or some logic error rather than my "not thinking about it". Just like what was mentioned in the article, the alternative is manual testing. One benefit of testing is that then you start thinking about edge cases you didn't think about before. My common thought process: "What other tests should I add in here? What are the boundary conditions that are possible? Oh, running this function with an empty list even though it generally should have something in it. Huh, I didn't even account for that in my code!"
marcosdumay 4y ago
> what's the point of asserting that add(2,2) will yield 4 if you've already been thinking about that You are probably thinking about unit tests here, and yes, if it exercise a single point that you placed there by design, that test can only guarantee that somebody won't mess with it on the future. On the other hand, I'm writing an interpreter in a side project, and the single most useful test there is literally and assert that eval("1 1 +") equals eval("2"). That is because it's not a unit test, and it touches a lot of different points. But for an outside observer, it looks exactly the same as a useless unit test.
siva7 4y ago
Tests waste a lot of time. That's certainly true for some teams. On the other hand, those tests have saved me in case a useful documentation is lacking which is virtually true for almost every closed source software project. To be fair, sometimes these tests were also misguided by misunderstood requirements, providing a false "documentation".
ravenstine 4y ago
Tests can waste time because developers treat the test suite as its own application that needs to be managed. Stop. Doing. This. With some exceptions, tests should NOT be DRY. They should be dumb, explicit, repetitive, conventional, and as independent as possible. This allows tests to be more quickly understood by any given developer and changed without breaking other tests for stupid reasons unrelated to the actual application. So much of my time has been wasted at every job I've had because developers believed that the tests should be really clever and use a bunch of magic and shared logic to "save time", which in reality becomes difficult to manage and understand. Most tests are crap for that reason alone.
- kapep 4y ago
  On the other hand I've seen test suites where each test consisted of 98% boilerplate that got copy&pasted every single time. Depending on the tests setup, they would mix and match hundredths of lines from other tests, which sometimes introduced subtle, hard to spot errors. There wasn't much magic and it was nightmare to read and maintain.
dwheeler 4y ago
Testing is important, because there are many ways things can go wrong. Automated testing is critically important, because "manual testing" is often another phrase for "testing we plan to do but won't do often if ever". Unit testing may or may not be the best approach. Unit tests are fast, and make it easy to answer very specific answers, but they're tied to the specific software design. Since computers have become much faster, I generally recommend that people use more integration testing, because they reveal when the components fail to work together (for whatever reason), and their added CPU effort is typically no big deal. That said, I agree that testing is only useful if it's plausible it would detect a problem. But developers are really good at asserting something "can't happen" and then does :-).
pjdesno 4y ago
Being able to automatically run your old tests to find regressions is super-super useful. I would argue that every bug you fix should turn into a test, since you're probably going to encounter it again.
dkarl 4y ago
> things tend to go wrong in cases that you haven't thought about so your tests don't cover them anyway I think this is a symptom of writing all the tests at the same time or immediately after the implementation. Adding test cases during implementation can be a good thing. You want to test cases that hit implementation edge cases, like when the length of the input is equal to, or one less or greater than, an internal buffer length or batch size. While you're coding the challenging parts of the implementation, these cases naturally pop out at you. However, while you're focusing on the hard parts, the "easy" parts fade into the background. You're much more likely to cover them if you make a quick list of test cases before you get immersed in the implementation. My experience is that when you (I) write test cases before implementation, I tend to cover more types of invalid inputs, as well as other conceptual corner cases, such as "degenerate" cases: zero or empty inputs, cases that have zero or empty outputs, and cases where an output could be computed in a straightforward way, but shouldn't be, because it wouldn't be valid. For example, one thing that has been a huge pet peeve for me my entire career is programmers treating empty inputs as invalid when there's a straightforward and correct way to handle them. After twenty years in the industry trying to get people to do this right, I've started to realize: people get this right if they talk about it before they start the implementation! If you sort an empty list, the result is an empty list, no problem. If you search for an alphanumeric character in an empty string, the result is that it's not found. But if they write the implementation before considering those cases, that's when they end up thinking it's okay to throw exceptions for them, because in their mind the implementation ("start with the first letter") becomes the specification.
ozim 4y ago
You know that when you have testable code because you started writing it in such way - you might not have to write all failure modes right away into your test suit. Then when things go wrong in a way you haven't thought and your test did not cover for it - you can easily add test to cover that case. One of the workflows in testable code base is "find a bug" -> "write tests to cover the bug" -> "fix the bug" -> "tests passing" -> "your test suit now covers something that was actual bug and should not be ever again".
Reefersleep 4y ago
Turn it upside down. Do the bare minimum of thinking and writing when setting up the initial test case, really just have it be scaffolding to run a single sunny scenario test case. Then let users (testers, other developers, automated systems or actual users - whatever hits your code) do any further discovery for you. Whenever they cause a bug, that's a new test case (or a series of new cases) to add to your initial single case.

pixl97 4y ago

I'm not really a developer here, more of the wedge between the customer and developer when things go wrong, but don't these tests cover the continuous updating of the application as specifications change? Especially as multiple people are working on the code and may have different ideas of what's going on. May not catch everything, but can trigger a warning in many cases.

xyzelement 4y ago
Ah - I can see those kinds of tests as useful. I guess they are something like "integration tests" or something like that which tests the user-visible outputs of the system. My comment was about very low-level unit testing, where it's literally every function of code must be covered, isolated from other parts of code.
crazygringo 4y ago
Exactly this. Also in codebases where the authors seemed opposed to writing good comments, tests are extremely useful in documenting expected behavior in edge cases, how an optional argument is supposed to be dealt with when missing, etc. Not to mention that people are often a lot less rigorous with "manual testing" than they should be. They don't think of scenarios, they skip scenarios, because nobody's checking on them. While when you write tests, you're creating a permanent record where you can be held accountable if you miss something.
- pixl97 4y ago
  >They don't think of scenarios, they skip scenarios, Bane of my life. Me: "Hey, this big customer we're not going to tell to change their behaviors and pays us a lot of money is using this product in this particular way. We should put in a functional test case to ensure that we don't mess this up" Dev/QA: "Eh, no" This generally leads to a support call half a year later with the customer panicking, support groaning, and the developers panicking because they 'could have never predicted the customer using the application in this manner'.
- kraftman 4y ago
  unit tests are comments that are forced to stay up to date and attached to the code they are for.
- hbrn 4y ago
  > in codebases where the authors seemed opposed to writing good comments, tests are extremely useful in documenting expected behavior in edge cases In other words, tests are useful in teams with awful engineering culture. I tend to agree, in my experience they are almost synonymous. If you treat engineering culture as weather (i.e. something you have no control over), then tests are a good tool to address your problems. > you're creating a permanent record where you can be held accountable if you miss something Bureaucracy is another word for it. You know who excels at it? Government. And you know what government is really bad at? Running business and innovating.

PathOfEclipse 4y ago

I attribute some of the greatest successes of my career in part to having good unit test coverage. I have seen no other pragmatic way to solve the problem that, in the worst case, a minor software update can be O(N) expensive to make, where N is the size or complexity of your program. That little update has the habit of breaking things in code paths you'd never expect. With a good suite of unit tests, you can validate that your program likely still functions, and, far more importantly, you can do the validation quickly, which results fast feedback loops and iteration times. Fast feedback loops are essential to high developer productivity. When developers are being slow, in my experience it usually is related to the system they are working on having slow feedback cycles. > The basic "problem" I have with tests is that - at least maybe in my experience - things tend to go wrong in cases that you haven't thought about so your tests don't cover them anyway. You are right in that unit testing can't make up for bad engineering. One advantage of investing time in it, though, is it gives you potentially more opportunity to think through those cases. And, as a consolation, prize, you write unit tests for bugs that make it through and now you have a great regression suite. Furthermore, a major issue that often comes when unit tests are omitted or not taken seriously, is that these tests can't easily be added after the fact, A system has to be designed with unit testing in mind, or else you end up with a bunch of untestable code. I believe that many legacy codebases remain without tests because the act of refactoring them enough to support any testing at all would be both time-consuming and high risk in terms of potential breakage.

trgn 4y ago
I agree 100%, but just want to add my2c. Unit tests are in general never wasted, in the sense that they trap regressions before release. I don't think they necessarily are good to validate work though, or at least, they do so poorly. It's easy to write lots of tests which don't cover the areas your program will bork on in production. Which I think is point of GP. Value of tests compound over time. The longer the software is in production and maintained, the more work the unit tests have been doing. It could be a paltry 20% code coverage set of tests, but nonetheless they're fighting the good fight. The value of a test suite is expressed in the integral over time. Few tests can do a lot of work; on the flip side, a lot of tests are useless when they're continuously thrown away due to pivots in business logic.

sitkack 4y ago

The system using the libraries should exercise those libraries sufficiently that integration tests catch those kinds of errors. If you are allowed only one test, it should be an integration test that runs in under XX seconds so you can maintain flow and correctness. If a system is under constant churn, test at the interface boundaries, unit tests are further burden during refactoring. Unit tests ensure that the internals function in a certain manner, external users should be concerned with internals. Unit tests are extremely useful in the core of a system like an evaluator, a rules engine, timezone, geospatial, computational geometry, etc. Your type system should capture many of the invariants in your system. I know what you are trying to convey, but the example `add(2,2)` encourages conversational confusion. If you are verifying hardware, that is a perfectly valid test, for a high level codebase, it is most likely a worthless test.

sitkack 4y ago
*edit, fixing typo* > external users shouldn't be concerned with internals

jdblair 4y ago

> And then, if you've already thought more complicated cases (eg negative numbers, cases that overflow) then your code already very likely handles them. And if you hadn't thought about them, your tests won't cover them either. Good tests check boundary conditions and edge cases. They check errors conditions (like overflow), not just the happy path. And yes, they can be a pain to write, and they affect the way your code is written. On the other hand, a test suite with good coverage can give you tremendous confidence to make sweeping changes. I have a side project, an art project, and part of that is a custom on-the-wire networking protocol. I have tests that check all the edge cases I could think of. Did I miss something? Very probably. On the other hand, I've written the protocol 3 times now: once in python, once in C, and once in C++ (the C++ case was more like a giant re-factor of the C code to make it more usable), and I'll probably write it again in rust. Each time I had confidence I got it right because my tests pass.

justin_oaks 4y ago

> what's the point of asserting that add(2,2) will yield 4 if you've already been thinking about that There have been plenty of times when I have run a test like add(2,2) and I don't get 4. Usually it's a typo or some logic error rather than my "not thinking about it". Just like what was mentioned in the article, the alternative is manual testing. One benefit of testing is that then you start thinking about edge cases you didn't think about before. My common thought process: "What other tests should I add in here? What are the boundary conditions that are possible? Oh, running this function with an empty list even though it generally should have something in it. Huh, I didn't even account for that in my code!"

marcosdumay 4y ago

> what's the point of asserting that add(2,2) will yield 4 if you've already been thinking about that You are probably thinking about unit tests here, and yes, if it exercise a single point that you placed there by design, that test can only guarantee that somebody won't mess with it on the future. On the other hand, I'm writing an interpreter in a side project, and the single most useful test there is literally and assert that eval("1 1 +") equals eval("2"). That is because it's not a unit test, and it touches a lot of different points. But for an outside observer, it looks exactly the same as a useless unit test.

siva7 4y ago

Tests waste a lot of time. That's certainly true for some teams. On the other hand, those tests have saved me in case a useful documentation is lacking which is virtually true for almost every closed source software project. To be fair, sometimes these tests were also misguided by misunderstood requirements, providing a false "documentation".

ravenstine 4y ago

Tests can waste time because developers treat the test suite as its own application that needs to be managed. Stop. Doing. This. With some exceptions, tests should NOT be DRY. They should be dumb, explicit, repetitive, conventional, and as independent as possible. This allows tests to be more quickly understood by any given developer and changed without breaking other tests for stupid reasons unrelated to the actual application. So much of my time has been wasted at every job I've had because developers believed that the tests should be really clever and use a bunch of magic and shared logic to "save time", which in reality becomes difficult to manage and understand. Most tests are crap for that reason alone.

kapep 4y ago
On the other hand I've seen test suites where each test consisted of 98% boilerplate that got copy&pasted every single time. Depending on the tests setup, they would mix and match hundredths of lines from other tests, which sometimes introduced subtle, hard to spot errors. There wasn't much magic and it was nightmare to read and maintain.

dwheeler 4y ago

Testing is important, because there are many ways things can go wrong. Automated testing is critically important, because "manual testing" is often another phrase for "testing we plan to do but won't do often if ever". Unit testing may or may not be the best approach. Unit tests are fast, and make it easy to answer very specific answers, but they're tied to the specific software design. Since computers have become much faster, I generally recommend that people use more integration testing, because they reveal when the components fail to work together (for whatever reason), and their added CPU effort is typically no big deal. That said, I agree that testing is only useful if it's plausible it would detect a problem. But developers are really good at asserting something "can't happen" and then does :-).

pjdesno 4y ago

Being able to automatically run your old tests to find regressions is super-super useful. I would argue that every bug you fix should turn into a test, since you're probably going to encounter it again.

dkarl 4y ago

> things tend to go wrong in cases that you haven't thought about so your tests don't cover them anyway I think this is a symptom of writing all the tests at the same time or immediately after the implementation. Adding test cases during implementation can be a good thing. You want to test cases that hit implementation edge cases, like when the length of the input is equal to, or one less or greater than, an internal buffer length or batch size. While you're coding the challenging parts of the implementation, these cases naturally pop out at you. However, while you're focusing on the hard parts, the "easy" parts fade into the background. You're much more likely to cover them if you make a quick list of test cases before you get immersed in the implementation. My experience is that when you (I) write test cases before implementation, I tend to cover more types of invalid inputs, as well as other conceptual corner cases, such as "degenerate" cases: zero or empty inputs, cases that have zero or empty outputs, and cases where an output could be computed in a straightforward way, but shouldn't be, because it wouldn't be valid. For example, one thing that has been a huge pet peeve for me my entire career is programmers treating empty inputs as invalid when there's a straightforward and correct way to handle them. After twenty years in the industry trying to get people to do this right, I've started to realize: people get this right if they talk about it before they start the implementation! If you sort an empty list, the result is an empty list, no problem. If you search for an alphanumeric character in an empty string, the result is that it's not found. But if they write the implementation before considering those cases, that's when they end up thinking it's okay to throw exceptions for them, because in their mind the implementation ("start with the first letter") becomes the specification.

ozim 4y ago

You know that when you have testable code because you started writing it in such way - you might not have to write all failure modes right away into your test suit. Then when things go wrong in a way you haven't thought and your test did not cover for it - you can easily add test to cover that case. One of the workflows in testable code base is "find a bug" -> "write tests to cover the bug" -> "fix the bug" -> "tests passing" -> "your test suit now covers something that was actual bug and should not be ever again".

Reefersleep 4y ago

Turn it upside down. Do the bare minimum of thinking and writing when setting up the initial test case, really just have it be scaffolding to run a single sunny scenario test case. Then let users (testers, other developers, automated systems or actual users - whatever hits your code) do any further discovery for you. Whenever they cause a bug, that's a new test case (or a series of new cases) to add to your initial single case.