4 ms·

Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a tes

Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.

kortilla 2y ago
It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
- SatvikBeri 2y ago
  It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
  - tetha 2y ago
    This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
- stouset 2y ago
  Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
  - taeric 2y ago
    I mean, anything can be done poorly. But consider that the team that builds the vehicles you drive is not the same as the team that tested said vehicle for safety concerns. That is, you can have a solid testing team for pretty much anything. You have to empower them, though. Largely, a lot of what you get to empower them with is ability to put constraints onto the engineering team. This could be crappy constraints that are far sweeping on the product. There is no need that it has to be that, though. The only constraint you have to have is stability of product interface. Which.... yeah, our industry doesn't do that so well. (Note, not stability as in "doesn't crash." Stability as in, same inputs that worked last year work today.)
  - RandomThoughts3 2y ago
    When I was doing safety critical software at the beginning of my career, we had a whole team in charge of building simulators replicating the API provided by the pieces of equipment we were integrating and their full behaviour. These simulators were themselves fully tested using both mock network traffic and recording of the real piece of equipment. This was used for integration testing and then repurposed for training. Every testing environment I have seen since have looked like a joke but to be fair the testing budget of this place was bigger than what some companies spend on a whole product.
- stingraycharles 2y ago
  Yeah. I work for a database company, if we didn’t have copious amounts of integration, regression and stress tests we would absolutely be having a very bad time. And of course we still have manual exploratory QA, it’s difficult to replace that safety net.
  - stouset 2y ago
    To be clear, tests when done well are incredible. Very little in this industry is done well. Tests are one of those things.
WalterBright 2y ago
> As a result code ends up being dumb, short, and obvious. I did a presentation on that last year: https://dconf.org/2023/#walterb https://dconf.org/2023/#walterb
cjfd 2y ago
All this speak about whether we prefer to have no tests to bad tests is pretty pointless. It is a bit like asking whether you would prefer to be beheaded or hung. The correct answer is neither. If one wants some semblance of quality one needs BOTH tests and good quality tests. Let me sum up some attributes of good tests. (1) They do not test at a level that is too low. If you test individual classes that do not have much logic in them, it probably is pointless. You are probably testing exciting facts like 'is the container type in my favorite language still capable of storing elements'. Mostly, it is best to test the interplay between a small number of classes on such a level that the attributes are such that the customer could recognize them as something that they value. (2) Don't test at a level that is too high. Integration tests often take a lot of effort to set up, they are brittle, and they are slow. Have a small number of them but not too many. I.e., the concept of the testing pyramid. In this thread one can read various assertions that sound like nonsense to me. (1) Only write functional code: nice if you can get away with it but some code has the explicit purpose to manipulate things in the real worlds or is explicitly there to maintain some state. One might also watch https://www.youtube.com/watch?v=j71n33A0CkI&t=314s https://www.youtube.com/watch?v=j71n33A0CkI&t=314s . The video is right. There actually is not much difference. (2) Don't mock. Well, if your program is big enough to contain many classes and/or functions so that it would be too big according to criteria (1) and (2) above, it becomes impractical to test all of them together so you will need mocks. What can just be passed in as function arguments can be passed in as function arguments but what comes out as behavior may need to be mocked. (3) Create test doubles instead of mocks. Use whatever is most convenient. Mocks record series of function calls. Test doubles maintain some state and one can see how that changes. Both can be good or bad depending on the situation. It seems pointless to have a preference separate from what you are trying to do
moonlessday 2y ago
what would you recommend to read to be better at writing tests?
pfdietz 2y ago
> a waste of build times There's an assumption here that tests need to be run for each build. This rules out more laborious testing that could be exposing bugs not found by time-limited tests. For compiler testing, one can set up property-based tests that run for unlimited lengths of time. These can be set up to be always running in the background. Literally billions of tests can be constructed and run.
hyperman1 2y ago
I understand this very much, but I have one big difference in experience: Once your coverage reaches about 60%, tests expose crashes simply by running the code. Even if there are no assertions. I've tamed a few untested code bases by writing huge integration test-ish unit tests, and comparing the output with pre-recorded answers. After that, you start adding real unit tests for fresh code, knowing the existing stuff is quite safe.

kortilla 2y ago

It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater

SatvikBeri 2y ago
It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
- tetha 2y ago
  This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
stouset 2y ago
Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
- taeric 2y ago
  I mean, anything can be done poorly. But consider that the team that builds the vehicles you drive is not the same as the team that tested said vehicle for safety concerns. That is, you can have a solid testing team for pretty much anything. You have to empower them, though. Largely, a lot of what you get to empower them with is ability to put constraints onto the engineering team. This could be crappy constraints that are far sweeping on the product. There is no need that it has to be that, though. The only constraint you have to have is stability of product interface. Which.... yeah, our industry doesn't do that so well. (Note, not stability as in "doesn't crash." Stability as in, same inputs that worked last year work today.)
- RandomThoughts3 2y ago
  When I was doing safety critical software at the beginning of my career, we had a whole team in charge of building simulators replicating the API provided by the pieces of equipment we were integrating and their full behaviour. These simulators were themselves fully tested using both mock network traffic and recording of the real piece of equipment. This was used for integration testing and then repurposed for training. Every testing environment I have seen since have looked like a joke but to be fair the testing budget of this place was bigger than what some companies spend on a whole product.
stingraycharles 2y ago
Yeah. I work for a database company, if we didn’t have copious amounts of integration, regression and stress tests we would absolutely be having a very bad time. And of course we still have manual exploratory QA, it’s difficult to replace that safety net.
- stouset 2y ago
  To be clear, tests when done well are incredible. Very little in this industry is done well. Tests are one of those things.

WalterBright 2y ago

> As a result code ends up being dumb, short, and obvious. I did a presentation on that last year: https://dconf.org/2023/#walterb https://dconf.org/2023/#walterb

cjfd 2y ago

All this speak about whether we prefer to have no tests to bad tests is pretty pointless. It is a bit like asking whether you would prefer to be beheaded or hung. The correct answer is neither. If one wants some semblance of quality one needs BOTH tests and good quality tests. Let me sum up some attributes of good tests. (1) They do not test at a level that is too low. If you test individual classes that do not have much logic in them, it probably is pointless. You are probably testing exciting facts like 'is the container type in my favorite language still capable of storing elements'. Mostly, it is best to test the interplay between a small number of classes on such a level that the attributes are such that the customer could recognize them as something that they value. (2) Don't test at a level that is too high. Integration tests often take a lot of effort to set up, they are brittle, and they are slow. Have a small number of them but not too many. I.e., the concept of the testing pyramid. In this thread one can read various assertions that sound like nonsense to me. (1) Only write functional code: nice if you can get away with it but some code has the explicit purpose to manipulate things in the real worlds or is explicitly there to maintain some state. One might also watch https://www.youtube.com/watch?v=j71n33A0CkI&t=314s https://www.youtube.com/watch?v=j71n33A0CkI&t=314s . The video is right. There actually is not much difference. (2) Don't mock. Well, if your program is big enough to contain many classes and/or functions so that it would be too big according to criteria (1) and (2) above, it becomes impractical to test all of them together so you will need mocks. What can just be passed in as function arguments can be passed in as function arguments but what comes out as behavior may need to be mocked. (3) Create test doubles instead of mocks. Use whatever is most convenient. Mocks record series of function calls. Test doubles maintain some state and one can see how that changes. Both can be good or bad depending on the situation. It seems pointless to have a preference separate from what you are trying to do

moonlessday 2y ago

what would you recommend to read to be better at writing tests?

pfdietz 2y ago

> a waste of build times There's an assumption here that tests need to be run for each build. This rules out more laborious testing that could be exposing bugs not found by time-limited tests. For compiler testing, one can set up property-based tests that run for unlimited lengths of time. These can be set up to be always running in the background. Literally billions of tests can be constructed and run.

hyperman1 2y ago

I understand this very much, but I have one big difference in experience: Once your coverage reaches about 60%, tests expose crashes simply by running the code. Even if there are no assertions. I've tamed a few untested code bases by writing huge integration test-ish unit tests, and comparing the output with pre-recorded answers. After that, you start adding real unit tests for fresh code, knowing the existing stuff is quite safe.