6 ms·

>I get paid for code that works, not for tests A blog post could be written about just this statement and how it contributes to a low trust workplace where tho

>I get paid for code that works, not for tests A blog post could be written about just this statement and how it contributes to a low trust workplace where those who cut corners are favored by stakeholders and everyone else is left scrambling to clean up the messes left in their wake. If you're writing code for yourself, sure, be targeted and conservative with your tests. But when you're working with others, for goodness sake, put the safety nets in place for the next poor soul that has to work on your code.

makeitdouble 3y ago
That quote is totally true though. Ultimately, tests are there to make sure code works, not for tests' sake. The rest of the sentence you're quoting being "so my philosophy is to test as little as possible to reach a given level of confidence" Overall the approach in the OP looks to me like a decently balanced take, trying to aim for enough tests without excess.
- godelski 3y ago
  It's impossible to prove that code works. But tests are a strong indicator and at least put bounds on where the program does work. If you're paid to write code that works, you're paid to write tests. This is fairly standard in every other engineering field.
  - makeitdouble 3y ago
    This is the kind of shortcut that gets easily forgotten after a while IMHO. Why you write tests is important, and for instance coverage numbers are not that. Most automated coverage assessments still won't guarantee you're testing all the critical patterns (you just need enough to touch all the paths) and a low number doesn't always mean it's not enough. I understand the use as an heuristic's, but as it gets widely adopted it also becomes more and more useless. I mean, today we see people eyeing at LLMs to boost their coverage numbers automatically, and that trend of writing low effort tests has been going all for a while IMO.
    - godelski 3y ago
      > I understand the use as an heuristic's, but as it gets widely adopted it also becomes more and more useless. I too am a big fan of Goodhart's Law. It seems many took this as advice and not a warning.
    - ffsm8 3y ago
      It's like that common misconception about the testing pyramid. The reason it's smaller at the top isn't because you should have numerically more tests at the bottom then at the top. It just shows that if you're doing a higher level test, you're also testing the layers below this. As an unrealistic example: if you'd have 1 IT and 1 UT, you'd still have double coverage at the bottom. You're probably still gonna create more UT then ITs though, as they're easier to write... so this is probably more academic pedantry then anything insightful
  - saurik 3y ago
    You can at least try to prove the code works, and you can get a lot further than people seem to bother. Hell: even just using a language with types--which many people don't do--is a form of invariant that proves away a lot of potential bugs you would otherwise have to test. And like, we absolutely have proof assistants and model checkers and more advanced languages with dependent types (even C++ can do a lot more than many languages due to how it can template over values, which includes stuff like the sizes of buffers on which it can do math at compile time, but we should be spending more time coding in the like of Coq/Idris/Lean)... let's not normalize a world in which people entirely give up on formal correctness. The problem with tests is that people use them as a crutch to not have to even just prove to themselves in their head -- much less to someone else or in a way that can be checked by a machine -- why their code does or doesn't work... they just throw together a ton of tests and if they hammer the code and when the tests stop failing they proudly announce "I guess it works now" and move on. My challenge: try to code for a while with the mentality that every every time you stop typing to test it or run and it doesn't work (including "my tests failed"), that is a serious problem that should be avoided. Instead, try your best to make it so the first time you get around to testing your code it works because you are that confident in how it works.
- MaxBarraclough 3y ago
  > tests are there to make sure code works, not for tests' sake. It works is awfully imprecise. Are you talking about perfect code? Some specific use-case? Some project-specific level of quality? Ordinary (non-fuzzing) tests are generally there to help protect against regressions when changing existing functionality, or to offer baseline quality assurance for new functionality. They can also be helpful in, say, determining whether the code behaves as expected when using a different compiler. They aren't normally how you discover a long-standing bug. There's always more to software quality than just testing.
- keybored 3y ago
  Such a reply is at best bad faith. Why would tests not be about ensuring working software? Why is the other side doing-it-for-itself while you care about the real goals? We could continue down this line: it’s not about working software; it’s about software that is useful for the user. Because it’s better to deliver slightly wrong results sometime as long as it is what the user expects and wants. But wait. Why did I assume that you want working-software-for-itself at the expense of what the user wants? Just a bad faith assumption, again. And we continue: it’s not about software that is useful for the user; it’s about software that is useful for the business.
  - makeitdouble 3y ago
    Sadly, writing tests to have tests is totally a thing. A tech lead can come in a new team and ask for 80% or more of code coverage and no one will bat an eye. Will the software work better for it ? that's up to debate and it will depend on the quality of the tests. I get your point of view, it feels completely absurd. The same way writing JIRA tickets just to have tickets, writing dumb comments because the CI will yet at you for not having comments frel absurd. And it's a reality in more places that I wish. > And we continue: it’s not about software that is useful for the user; it’s about software that is useful for the business. If we look at the number of failed businesses, that's a lesson that needs to be relearned again and again.
    - keybored 3y ago
      > Sadly, writing tests to have tests is totally a thing. A tech lead can come in a new team and ask for 80% or more of code coverage and no one will bat an eye. Will the software work better for it ? that's up to debate and it will depend on the quality of the tests. This is true but not the bad faith part. The bad faith part is assuming—based on nothing—that the other interlocutor is doing thing because of cargo-cult/silly reasons.
    - epgui 3y ago
      I completely take your point and agree with you. I’d just like to add a nuance/complication: sometimes it makes perfect sense for a lead to come and insist on the test coverage metric hitting X%. Not because it’s a good idea on its own or in general, but probably because of very context-specific reasons. It’s not uncommon for engineers to not think about how their code tests at all (which is related to the interfaces they design), so something as arbitrary as this can be a forcing function to bring a baseline of awareness and competence to a team. In that sort of scenario, ideally the norms change as the team problems evolve, and so that arbitrary test coverage goal would ideally become irrelevant over time as a team improves and matures. Think back to your first weeks / months of programming: this sort of reductionist and over-simplistic “rule” was probably very formative in your early years, even if you had to unlearn them over time. Real teams have people with a spectrum of skills, and sometimes you have to set crazy-ish rules to level-up the baseline.
godelski 3y ago
I think another blog could be similarly written on "don't fix it if it ain't broken." Fixing things before they are broken is substantially cheaper and takes far less time. But it is far easier to push off. Maintenance and fixing things BEFORE they are broken is key. Of course, not everything needs to be fixed. But many sayings are often taken too literally.
mkl95 3y ago
At my current company we have pretty high test coverage. The test suite also happens to be full of mocks plus some cargoculted antipatterns that make many tests useless. The engineers who actually test their stuff can be counted with one hand.
- stouset 3y ago
  I see this so often. Mocks and stubs get used all over the place because nobody understands what it means to write code that’s easily testable. They’re great when used correctly (e.g., remote services or inherently stateful APIs like time). But they almost never are. You end up with tests that ensure one and only one thing: the code is written the way it’s currently written. Tests should do two things: find unexpected out-of-spec behavior, and prevent regressions during the course of editing and refactoring. These overly-mocked tests by definition can’t do the first one and they actively inhibit the second. They have negative value insofar as they constantly trigger failure while making completely benign edits.
  - codethatwerks 3y ago
    [flagged]
  - keybored 3y ago
    It feels like - First write the needed change - Now assert that we just wrote the needed change
- rhdunn 3y ago
  This is why I tend to lean toward the approach of only using mocks, stubs, etc. when absolutely needed. That tends to be at the places where the code is interacting with external components like databases or web services than with components that are within the current layer, such as connecting controllers and services. It's also why I don't like the philosophy of unit testing being about only testing a specific class. -- You end up in situations where you convince yourself you have to mock the helper classes it uses, so end up with disconnected pointless tests. Instead, each test should be testing as much real code as possible.
th3byrdm4n 3y ago
This assumes useful tests. There's a false equivalency drawn between writing tests means you wrote good code. In reality, those who can write good tests can also write good code. My rules of thumb: "Be a goldfish" - Forget everything you know about your project, is it complicated, non-intuitive? Tests + clear documentation. But don't test for stupid stuff. AI's already generate+test the stupid stuff for us anyway ... why are we writing it
- kmac_ 3y ago
  Exactly this. Most of the tests reimplement the implementation using mocks. Such tests are useless, as they always prove the code is correct. Worse, such tests make refactoring much slower. On a low level, only black-box interface tests make sense, and on a high level, use scenario testing. The implementation has to be tested indirectly, otherwise, it leaks.
codethatwerks 3y ago
“that works” is doing a lot of the heavy lifting. I take it to mean some point in the quality space where the number of bugs (noisy crashes or silent inaccuracies) is almost zero. “that works” can also mean demo to pointy haired boss and it works one time on a developers laptop. But that really means “that seems to work”. There are different concerns at play: Are you writing redundant tests? Could tests be made redundant by say making better tests, sophisticated type systems or better architecture. “Make illegal state impossible” kind of things. For example in C# the private keyword is an assurance of a constraint. Your compiler will check the illegal state cannot be made by an external class corrupting it, reducing the scope of code that needs to be checked. Modularizing code well can also help. Making it easier for developers to understand how things fit together, further improving quality. “10000 tests + shit architecture < 1000 tests and good architecture” will often be true for the real definition of “that works”.
keybored 3y ago
See also “let’s be pragmatic”. “Pragmatic” can be used to short-circuit any debate when the interlocutor arbitrarily decides that you care too much about whatever subject. Like, do you think that adding tests in this PR will give us concrete, tangible results? Well no... but in the long term it will make the application more robust and review more streamlined and— and now the other guy has rhetorically won because he has a snippy “pragmatic” argument while you replied with a whole paragraph. Let’s be pragmatic. I care about working code. I care about the bottom-line of the business. On and on and on.

makeitdouble 3y ago

That quote is totally true though. Ultimately, tests are there to make sure code works, not for tests' sake. The rest of the sentence you're quoting being "so my philosophy is to test as little as possible to reach a given level of confidence" Overall the approach in the OP looks to me like a decently balanced take, trying to aim for enough tests without excess.

godelski 3y ago
It's impossible to prove that code works. But tests are a strong indicator and at least put bounds on where the program does work. If you're paid to write code that works, you're paid to write tests. This is fairly standard in every other engineering field.
- makeitdouble 3y ago
  This is the kind of shortcut that gets easily forgotten after a while IMHO. Why you write tests is important, and for instance coverage numbers are not that. Most automated coverage assessments still won't guarantee you're testing all the critical patterns (you just need enough to touch all the paths) and a low number doesn't always mean it's not enough. I understand the use as an heuristic's, but as it gets widely adopted it also becomes more and more useless. I mean, today we see people eyeing at LLMs to boost their coverage numbers automatically, and that trend of writing low effort tests has been going all for a while IMO.
  - godelski 3y ago
    > I understand the use as an heuristic's, but as it gets widely adopted it also becomes more and more useless. I too am a big fan of Goodhart's Law. It seems many took this as advice and not a warning.
  - ffsm8 3y ago
    It's like that common misconception about the testing pyramid. The reason it's smaller at the top isn't because you should have numerically more tests at the bottom then at the top. It just shows that if you're doing a higher level test, you're also testing the layers below this. As an unrealistic example: if you'd have 1 IT and 1 UT, you'd still have double coverage at the bottom. You're probably still gonna create more UT then ITs though, as they're easier to write... so this is probably more academic pedantry then anything insightful
- saurik 3y ago
  You can at least try to prove the code works, and you can get a lot further than people seem to bother. Hell: even just using a language with types--which many people don't do--is a form of invariant that proves away a lot of potential bugs you would otherwise have to test. And like, we absolutely have proof assistants and model checkers and more advanced languages with dependent types (even C++ can do a lot more than many languages due to how it can template over values, which includes stuff like the sizes of buffers on which it can do math at compile time, but we should be spending more time coding in the like of Coq/Idris/Lean)... let's not normalize a world in which people entirely give up on formal correctness. The problem with tests is that people use them as a crutch to not have to even just prove to themselves in their head -- much less to someone else or in a way that can be checked by a machine -- why their code does or doesn't work... they just throw together a ton of tests and if they hammer the code and when the tests stop failing they proudly announce "I guess it works now" and move on. My challenge: try to code for a while with the mentality that every every time you stop typing to test it or run and it doesn't work (including "my tests failed"), that is a serious problem that should be avoided. Instead, try your best to make it so the first time you get around to testing your code it works because you are that confident in how it works.
MaxBarraclough 3y ago
> tests are there to make sure code works, not for tests' sake. It works is awfully imprecise. Are you talking about perfect code? Some specific use-case? Some project-specific level of quality? Ordinary (non-fuzzing) tests are generally there to help protect against regressions when changing existing functionality, or to offer baseline quality assurance for new functionality. They can also be helpful in, say, determining whether the code behaves as expected when using a different compiler. They aren't normally how you discover a long-standing bug. There's always more to software quality than just testing.
keybored 3y ago
Such a reply is at best bad faith. Why would tests not be about ensuring working software? Why is the other side doing-it-for-itself while you care about the real goals? We could continue down this line: it’s not about working software; it’s about software that is useful for the user. Because it’s better to deliver slightly wrong results sometime as long as it is what the user expects and wants. But wait. Why did I assume that you want working-software-for-itself at the expense of what the user wants? Just a bad faith assumption, again. And we continue: it’s not about software that is useful for the user; it’s about software that is useful for the business.
- makeitdouble 3y ago
  Sadly, writing tests to have tests is totally a thing. A tech lead can come in a new team and ask for 80% or more of code coverage and no one will bat an eye. Will the software work better for it ? that's up to debate and it will depend on the quality of the tests. I get your point of view, it feels completely absurd. The same way writing JIRA tickets just to have tickets, writing dumb comments because the CI will yet at you for not having comments frel absurd. And it's a reality in more places that I wish. > And we continue: it’s not about software that is useful for the user; it’s about software that is useful for the business. If we look at the number of failed businesses, that's a lesson that needs to be relearned again and again.
  - keybored 3y ago
    > Sadly, writing tests to have tests is totally a thing. A tech lead can come in a new team and ask for 80% or more of code coverage and no one will bat an eye. Will the software work better for it ? that's up to debate and it will depend on the quality of the tests. This is true but not the bad faith part. The bad faith part is assuming—based on nothing—that the other interlocutor is doing thing because of cargo-cult/silly reasons.
  - epgui 3y ago
    I completely take your point and agree with you. I’d just like to add a nuance/complication: sometimes it makes perfect sense for a lead to come and insist on the test coverage metric hitting X%. Not because it’s a good idea on its own or in general, but probably because of very context-specific reasons. It’s not uncommon for engineers to not think about how their code tests at all (which is related to the interfaces they design), so something as arbitrary as this can be a forcing function to bring a baseline of awareness and competence to a team. In that sort of scenario, ideally the norms change as the team problems evolve, and so that arbitrary test coverage goal would ideally become irrelevant over time as a team improves and matures. Think back to your first weeks / months of programming: this sort of reductionist and over-simplistic “rule” was probably very formative in your early years, even if you had to unlearn them over time. Real teams have people with a spectrum of skills, and sometimes you have to set crazy-ish rules to level-up the baseline.

godelski 3y ago

I think another blog could be similarly written on "don't fix it if it ain't broken." Fixing things before they are broken is substantially cheaper and takes far less time. But it is far easier to push off. Maintenance and fixing things BEFORE they are broken is key. Of course, not everything needs to be fixed. But many sayings are often taken too literally.

mkl95 3y ago

At my current company we have pretty high test coverage. The test suite also happens to be full of mocks plus some cargoculted antipatterns that make many tests useless. The engineers who actually test their stuff can be counted with one hand.

stouset 3y ago
I see this so often. Mocks and stubs get used all over the place because nobody understands what it means to write code that’s easily testable. They’re great when used correctly (e.g., remote services or inherently stateful APIs like time). But they almost never are. You end up with tests that ensure one and only one thing: the code is written the way it’s currently written. Tests should do two things: find unexpected out-of-spec behavior, and prevent regressions during the course of editing and refactoring. These overly-mocked tests by definition can’t do the first one and they actively inhibit the second. They have negative value insofar as they constantly trigger failure while making completely benign edits.
- codethatwerks 3y ago
  [flagged]
- keybored 3y ago
  It feels like - First write the needed change - Now assert that we just wrote the needed change
rhdunn 3y ago
This is why I tend to lean toward the approach of only using mocks, stubs, etc. when absolutely needed. That tends to be at the places where the code is interacting with external components like databases or web services than with components that are within the current layer, such as connecting controllers and services. It's also why I don't like the philosophy of unit testing being about only testing a specific class. -- You end up in situations where you convince yourself you have to mock the helper classes it uses, so end up with disconnected pointless tests. Instead, each test should be testing as much real code as possible.

th3byrdm4n 3y ago

This assumes useful tests. There's a false equivalency drawn between writing tests means you wrote good code. In reality, those who can write good tests can also write good code. My rules of thumb: "Be a goldfish" - Forget everything you know about your project, is it complicated, non-intuitive? Tests + clear documentation. But don't test for stupid stuff. AI's already generate+test the stupid stuff for us anyway ... why are we writing it

kmac_ 3y ago
Exactly this. Most of the tests reimplement the implementation using mocks. Such tests are useless, as they always prove the code is correct. Worse, such tests make refactoring much slower. On a low level, only black-box interface tests make sense, and on a high level, use scenario testing. The implementation has to be tested indirectly, otherwise, it leaks.

codethatwerks 3y ago

“that works” is doing a lot of the heavy lifting. I take it to mean some point in the quality space where the number of bugs (noisy crashes or silent inaccuracies) is almost zero. “that works” can also mean demo to pointy haired boss and it works one time on a developers laptop. But that really means “that seems to work”. There are different concerns at play: Are you writing redundant tests? Could tests be made redundant by say making better tests, sophisticated type systems or better architecture. “Make illegal state impossible” kind of things. For example in C# the private keyword is an assurance of a constraint. Your compiler will check the illegal state cannot be made by an external class corrupting it, reducing the scope of code that needs to be checked. Modularizing code well can also help. Making it easier for developers to understand how things fit together, further improving quality. “10000 tests + shit architecture < 1000 tests and good architecture” will often be true for the real definition of “that works”.

keybored 3y ago

See also “let’s be pragmatic”. “Pragmatic” can be used to short-circuit any debate when the interlocutor arbitrarily decides that you care too much about whatever subject. Like, do you think that adding tests in this PR will give us concrete, tangible results? Well no... but in the long term it will make the application more robust and review more streamlined and— and now the other guy has rhetorically won because he has a snippy “pragmatic” argument while you replied with a whole paragraph. Let’s be pragmatic. I care about working code. I care about the bottom-line of the business. On and on and on.