4 ms·

I feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, bu

by gleenn 2y ago

I feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, but that should be a vast minority of the tests being written. Test driven development and Agile and all that had some people upset and whiny 20 years ago, but if I see code without tests, I assume it's broken. The only way anyone knows anything works both now AND when the next 10, 100, 1000 commits roll in, your manually-tested edge cases have long been forgotten and I doubt you even got it right at all. Let's keep normalizing writing tests and thinking, up front, how to ensure our code works. That's the best time to think about all those pesky edge cases, and it's the only practical way to keep code working. Don't hoist this on to some poor chum because you are lazy or don't know how. (Spelling)

stouset 2y ago
Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.
- kortilla 2y ago
  It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
  - SatvikBeri 2y ago
    It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
    - tetha 2y ago
      This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
  - stouset 2y ago
    Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
    - taeric 2y ago
      I mean, anything can be done poorly. But consider that the team that builds the vehicles you drive is not the same as the team that tested said vehicle for safety concerns. That is, you can have a solid testing team for pretty much anything. You have to empower them, though. Largely, a lot of what you get to empower them with is ability to put constraints onto the engineering team. This could be crappy constraints that are far sweeping on the product. There is no need that it has to be that, though. The only constraint you have to have is stability of product interface. Which.... yeah, our industry doesn't do that so well. (Note, not stability as in "doesn't crash." Stability as in, same inputs that worked last year work today.)
- WalterBright 2y ago
  > As a result code ends up being dumb, short, and obvious. I did a presentation on that last year: https://dconf.org/2023/#walterb https://dconf.org/2023/#walterb
- cjfd 2y ago
  All this speak about whether we prefer to have no tests to bad tests is pretty pointless. It is a bit like asking whether you would prefer to be beheaded or hung. The correct answer is neither. If one wants some semblance of quality one needs BOTH tests and good quality tests. Let me sum up some attributes of good tests. (1) They do not test at a level that is too low. If you test individual classes that do not have much logic in them, it probably is pointless. You are probably testing exciting facts like 'is the container type in my favorite language still capable of storing elements'. Mostly, it is best to test the interplay between a small number of classes on such a level that the attributes are such that the customer could recognize them as something that they value. (2) Don't test at a level that is too high. Integration tests often take a lot of effort to set up, they are brittle, and they are slow. Have a small number of them but not too many. I.e., the concept of the testing pyramid. In this thread one can read various assertions that sound like nonsense to me. (1) Only write functional code: nice if you can get away with it but some code has the explicit purpose to manipulate things in the real worlds or is explicitly there to maintain some state. One might also watch https://www.youtube.com/watch?v=j71n33A0CkI&t=314s https://www.youtube.com/watch?v=j71n33A0CkI&t=314s . The video is right. There actually is not much difference. (2) Don't mock. Well, if your program is big enough to contain many classes and/or functions so that it would be too big according to criteria (1) and (2) above, it becomes impractical to test all of them together so you will need mocks. What can just be passed in as function arguments can be passed in as function arguments but what comes out as behavior may need to be mocked. (3) Create test doubles instead of mocks. Use whatever is most convenient. Mocks record series of function calls. Test doubles maintain some state and one can see how that changes. Both can be good or bad depending on the situation. It seems pointless to have a preference separate from what you are trying to do
- moonlessday 2y ago
  what would you recommend to read to be better at writing tests?
- pfdietz 2y ago
  > a waste of build times There's an assumption here that tests need to be run for each build. This rules out more laborious testing that could be exposing bugs not found by time-limited tests. For compiler testing, one can set up property-based tests that run for unlimited lengths of time. These can be set up to be always running in the background. Literally billions of tests can be constructed and run.
- hyperman1 2y ago
  I understand this very much, but I have one big difference in experience: Once your coverage reaches about 60%, tests expose crashes simply by running the code. Even if there are no assertions. I've tamed a few untested code bases by writing huge integration test-ish unit tests, and comparing the output with pre-recorded answers. After that, you start adding real unit tests for fresh code, knowing the existing stuff is quite safe.
BobbyTables2 2y ago
I fully agree. For most of my career, I hated the idea of writing (unit) tests. They were almost always a waste of time. At one point, I ended up on a team that had virtually no QA and only did automated end-to-end tests on a complex product. After working through a lot of tech debt on the testing (took a few months), it was solid and caught many defects introduced by junior/new developers (easily missed in code review). I now have the fortune to be in a different group with a similar mindset. I’m motivated to write tests for my own code just to prove to myself that it works properly in odd error cases too. My team prefers testing instead of “run fast and break things.” (And with multiple decades of experience, I humbly can state I cause significantly fewer bugs than in the past.) This also makes me write code in an easily testable manner. Trying to test already written code is usually a lengthy and painful exercise in frustration. It’s also too easy to write error handlers that are NEVER actually exercised once, even manually — and surprise, they may not even work! People who never write tests do this quite often.

by gleenn 2y ago

stouset 2y ago

Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems. Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly. These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point. The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.

kortilla 2y ago
It sounds like you’re specifically talking about unit tests and people that suck at writing those. I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do. Don’t throw the baby out with the bathwater
- SatvikBeri 2y ago
  It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?" At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.
  - tetha 2y ago
    This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right. Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such. That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.
- stouset 2y ago
  Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable. I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.
  - taeric 2y ago
    I mean, anything can be done poorly. But consider that the team that builds the vehicles you drive is not the same as the team that tested said vehicle for safety concerns. That is, you can have a solid testing team for pretty much anything. You have to empower them, though. Largely, a lot of what you get to empower them with is ability to put constraints onto the engineering team. This could be crappy constraints that are far sweeping on the product. There is no need that it has to be that, though. The only constraint you have to have is stability of product interface. Which.... yeah, our industry doesn't do that so well. (Note, not stability as in "doesn't crash." Stability as in, same inputs that worked last year work today.)
WalterBright 2y ago
> As a result code ends up being dumb, short, and obvious. I did a presentation on that last year: https://dconf.org/2023/#walterb https://dconf.org/2023/#walterb
cjfd 2y ago
All this speak about whether we prefer to have no tests to bad tests is pretty pointless. It is a bit like asking whether you would prefer to be beheaded or hung. The correct answer is neither. If one wants some semblance of quality one needs BOTH tests and good quality tests. Let me sum up some attributes of good tests. (1) They do not test at a level that is too low. If you test individual classes that do not have much logic in them, it probably is pointless. You are probably testing exciting facts like 'is the container type in my favorite language still capable of storing elements'. Mostly, it is best to test the interplay between a small number of classes on such a level that the attributes are such that the customer could recognize them as something that they value. (2) Don't test at a level that is too high. Integration tests often take a lot of effort to set up, they are brittle, and they are slow. Have a small number of them but not too many. I.e., the concept of the testing pyramid. In this thread one can read various assertions that sound like nonsense to me. (1) Only write functional code: nice if you can get away with it but some code has the explicit purpose to manipulate things in the real worlds or is explicitly there to maintain some state. One might also watch https://www.youtube.com/watch?v=j71n33A0CkI&t=314s https://www.youtube.com/watch?v=j71n33A0CkI&t=314s . The video is right. There actually is not much difference. (2) Don't mock. Well, if your program is big enough to contain many classes and/or functions so that it would be too big according to criteria (1) and (2) above, it becomes impractical to test all of them together so you will need mocks. What can just be passed in as function arguments can be passed in as function arguments but what comes out as behavior may need to be mocked. (3) Create test doubles instead of mocks. Use whatever is most convenient. Mocks record series of function calls. Test doubles maintain some state and one can see how that changes. Both can be good or bad depending on the situation. It seems pointless to have a preference separate from what you are trying to do
moonlessday 2y ago
what would you recommend to read to be better at writing tests?
pfdietz 2y ago
> a waste of build times There's an assumption here that tests need to be run for each build. This rules out more laborious testing that could be exposing bugs not found by time-limited tests. For compiler testing, one can set up property-based tests that run for unlimited lengths of time. These can be set up to be always running in the background. Literally billions of tests can be constructed and run.
hyperman1 2y ago
I understand this very much, but I have one big difference in experience: Once your coverage reaches about 60%, tests expose crashes simply by running the code. Even if there are no assertions. I've tamed a few untested code bases by writing huge integration test-ish unit tests, and comparing the output with pre-recorded answers. After that, you start adding real unit tests for fresh code, knowing the existing stuff is quite safe.

BobbyTables2 2y ago

I fully agree. For most of my career, I hated the idea of writing (unit) tests. They were almost always a waste of time. At one point, I ended up on a team that had virtually no QA and only did automated end-to-end tests on a complex product. After working through a lot of tech debt on the testing (took a few months), it was solid and caught many defects introduced by junior/new developers (easily missed in code review). I now have the fortune to be in a different group with a similar mindset. I’m motivated to write tests for my own code just to prove to myself that it works properly in odd error cases too. My team prefers testing instead of “run fast and break things.” (And with multiple decades of experience, I humbly can state I cause significantly fewer bugs than in the past.) This also makes me write code in an easily testable manner. Trying to test already written code is usually a lengthy and painful exercise in frustration. It’s also too easy to write error handlers that are NEVER actually exercised once, even manually — and surprise, they may not even work! People who never write tests do this quite often.