3 ms·

From the article: > The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay o

From the article: > The argument for isolating the units from each other is that it is easier to spot a potential bug. (...) In my opinion, this does not pay out because of the huge amount of false positive test cases you get and the time you need to fix them. Also, if you know the code base a little you should have an idea where the problem is. If not, this is your chance to get to know the code base a little better. This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail. If you somehow decide to change your code so that a few invariants break, these are not "false positives". This is your tests working as expected and warning you that you must pay attention to what you are doing so that you do to not introduce regressions. It's also completely mind-boggling and absurd to argue that "knowing the code" is any argument to avoid tracking invariants. The whole point of automated test suites is that you do not want the app to fail because you missed any detail or corner case or failure mode. Knowing the code does not prevent bugs or errors or regressions. I'm perplexed by the way we have people write long articles on unit tests when they don't really seem to understand what they are supposed to achieve.

theamk 2y ago
Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on. Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests. Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive. Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives. I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.
chowells 2y ago
It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage. All they check is that the code is still structured identically to the original implementation. This is the reality that people encounter, so of course it's what they talk about. Why would they talk about some ivory tower idea instead?
- motorest 2y ago
  > It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage. At most, you can only arrive at that conclusion by observing your personal reality. If your teams crank out crappy code that's error-prone and untestable with their crappy tests, and can't manage to fix either their code or the tests, then naturally they end up living in the reality they created for themselves. Back in the real world, some projects renowned by their stability and robustness go out of their way to single out their test-drive approach to software design and development as the fundamental reason their software is stable and robust. What do you think is the difference? > All they check is that the code is still structured identically to the original implementation. Unit tests don't test structure. Their whole point is that they do not reflect structe Unit tests only cover the behavior that's expected from a specific unit of code, and they only check for the invariants relevant to that specific unit of code. Perhaps this is a telltale sign you should rethink what you're doing.
- hitchstory 2y ago
  A shit test written before writing the code is still a shit test. Mimetic tests arent be any better written after the code either. If I had to choose between 1) always writing specification-linked tests that make as few architectural assumptions as possible and 2) TDD, sure, I'd pick 1 every time. 1 and 2 is still better though.
ants_everywhere 2y ago
People don't like to write tests so any argument saying you need fewer of them goes down real smooth. The details don't matter so much. But when software actually needs to be made at scale and with high quality you almost always have lots of tests. Statistically, the average developer is not writing software anymore where corner cases matter. They're writing various permutations on standard web apps. Most web apps are pretty buggy, even ones from major corporations. So stakes are relatively low for having failures. People writing file systems or crypto routines are going to have an easier time understanding the value of tests.
Kamq 2y ago
> This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. I mean, yeah. Most devs don't understand the point. It turns into a check-boxing exercise. And most devs are vaguely smart and can figure out how to do a check-boxing exercise with minimal effort while evading the point of doing it. Relying on people who don't care enough to write good code to care enough to write good tests has a certain irony to it. > To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail. I'll push back on this. I'm a veteran of a codebase where the tests broke all the time with tons of false positives. This was mostly because of over-mocking, and an abundance of assert X called with Y and Z tests. Even pure functions were mocked. Every time I re-wrote a function implementation, even if it was a pure function and produced the same results, it would break tens of tests. Bonus points if the implementation of the thing that was mocked out had changed since the last time someone reviewed the tests and the way it was calling the function didn't do what the test assumed anymore.
Izkata 2y ago
> To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes.
- motorest 2y ago
  > They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes. The point is that it's absurd to try to characterize the regressions you're introducing as "false positive"/"false negative". Again, this is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior. If you change behavior but fail to change how that behavior is checked, that is not a false positive or false negative. That represents a few problems you introduced: failing to update how that behavior is checked, and failing implement new behavior without checking if it works. Both reflect problems created by a developer failing to perform the basics of their work, and worse: blaming working tests for the mess he's creating. Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
  - Izkata 2y ago
    > Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior. If your tests don't fail before you update them, that's one false positive. Usually introduced by mocks that hide part of the behavior. If your tests were passing but a bug is discovered in the code that should have been caught by the given tests, that's another false positive. Also possibly introduced by mocks, but hopefully is more like just a missing assert. If your tests start failing with a correct code change and just straight need to be updated, that's a false negative (this is your example, halfway through the work). In either false positive case if it was caused by mocks, those are bad tests and should be rewritten and not just updated. > Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens? Lights at an intersection aren't tests, they're part of the working system. Tests in this example would be something more like a central reporting system that's supposed to show what's going on at the intersection. The system (the intersection and lights) can be working totally fine, but the reporting system is showing it messed up (false negative) or messed-up lights can be causing crashes but the reporting system shows it's working fine (false positive). In both cases it should be fixed, but the fix is independent of what - if anything - should be done with the intersection and lights themselves.

theamk 2y ago

Maybe in the ideal case, but in practice, a lot of unit tests track much more than invariants other components depend on. Example: a function makes two RPC calls to query two independent statuses, which are of course mocked for the unit tests. Due to the the test library used, unit tests expect query A first and query B second. The function is refactored, and it now queries B first, so unit tests start to fail. This is 100% false positive - nothing in the real world cares if A or B are queried first, those actually go to systems which don't even know each other. And yet, the change author now has to fix dozens of failed tests. Another example: there is an event-based system, and unit tests hardcode number of main loop executions ("emit event A; poll; poll; poll; ensure event B arrives"). A change which adds one more poll cycle will have zero effect on the real app, but causes many tests to break. It's a false positive. Yet another example: there are random numbers involved, so test driver makes sure to seed RNG to known value at start of each test. A single call to random() is added somewhere early in the code and boom! each test explodes. Another hundreds of false positives. I could go on an on... Should those tests be rewritten to be more robust and only check things that really matter? Yep. Does real-life code has fragile unit tests like this? Yep, almost every single codebase I have seen.

chowells 2y ago

It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage. All they check is that the code is still structured identically to the original implementation. This is the reality that people encounter, so of course it's what they talk about. Why would they talk about some ivory tower idea instead?

motorest 2y ago
> It mostly comes from observing the reality of how tests are used rather than idealism. In a lot of "test-first" codebases, the majority of tests are garbage. At most, you can only arrive at that conclusion by observing your personal reality. If your teams crank out crappy code that's error-prone and untestable with their crappy tests, and can't manage to fix either their code or the tests, then naturally they end up living in the reality they created for themselves. Back in the real world, some projects renowned by their stability and robustness go out of their way to single out their test-drive approach to software design and development as the fundamental reason their software is stable and robust. What do you think is the difference? > All they check is that the code is still structured identically to the original implementation. Unit tests don't test structure. Their whole point is that they do not reflect structe Unit tests only cover the behavior that's expected from a specific unit of code, and they only check for the invariants relevant to that specific unit of code. Perhaps this is a telltale sign you should rethink what you're doing.
hitchstory 2y ago
A shit test written before writing the code is still a shit test. Mimetic tests arent be any better written after the code either. If I had to choose between 1) always writing specification-linked tests that make as few architectural assumptions as possible and 2) TDD, sure, I'd pick 1 every time. 1 and 2 is still better though.

ants_everywhere 2y ago

People don't like to write tests so any argument saying you need fewer of them goes down real smooth. The details don't matter so much. But when software actually needs to be made at scale and with high quality you almost always have lots of tests. Statistically, the average developer is not writing software anymore where corner cases matter. They're writing various permutations on standard web apps. Most web apps are pretty buggy, even ones from major corporations. So stakes are relatively low for having failures. People writing file systems or crypto routines are going to have an easier time understanding the value of tests.

Kamq 2y ago

> This is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. I mean, yeah. Most devs don't understand the point. It turns into a check-boxing exercise. And most devs are vaguely smart and can figure out how to do a check-boxing exercise with minimal effort while evading the point of doing it. Relying on people who don't care enough to write good code to care enough to write good tests has a certain irony to it. > To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. The whole point of having these tests is to have a way to automatically check for them each and every single time we touch the code, so that the tests warn us that a change we are doing will cause the application to fail. I'll push back on this. I'm a veteran of a codebase where the tests broke all the time with tons of false positives. This was mostly because of over-mocking, and an abundance of assert X called with Y and Z tests. Even pure functions were mocked. Every time I re-wrote a function implementation, even if it was a pure function and produced the same results, it would break tens of tests. Bonus points if the implementation of the thing that was mocked out had changed since the last time someone reviewed the tests and the way it was calling the function didn't do what the test assumed anymore.

Izkata 2y ago

> To start off, there is no such thing as a false positive test. Your tests track invariants, specially those which other components depend on. They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes.

motorest 2y ago
> They were just talking about test doubles / mocks above that. Those cause all sorts of risks with false positives and false negatives, for example by mocking out a function the code under test relies on, but the mock not being changed when the contract between those two pieces of code changes. The point is that it's absurd to try to characterize the regressions you're introducing as "false positive"/"false negative". Again, this is at best specious reasoning, and to me reflects that the blogger completely misses the point of having tests. Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior. If you change behavior but fail to change how that behavior is checked, that is not a false positive or false negative. That represents a few problems you introduced: failing to update how that behavior is checked, and failing implement new behavior without checking if it works. Both reflect problems created by a developer failing to perform the basics of their work, and worse: blaming working tests for the mess he's creating. Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens?
- Izkata 2y ago
  > Your code has behavior. Your tests verify invariants associated with this behavior. If you purposely change that behavior, it stands to reason that your task also involves updating how your code checks for that behavior. If your tests don't fail before you update them, that's one false positive. Usually introduced by mocks that hide part of the behavior. If your tests were passing but a bug is discovered in the code that should have been caught by the given tests, that's another false positive. Also possibly introduced by mocks, but hopefully is more like just a missing assert. If your tests start failing with a correct code change and just straight need to be updated, that's a false negative (this is your example, halfway through the work). In either false positive case if it was caused by mocks, those are bad tests and should be rewritten and not just updated. > Think about the issue: if a highway administration decides to redesign an intersection but fails to move/update its stoplights, does this means the old stoplights cause accidents with their false positive greens? Lights at an intersection aren't tests, they're part of the working system. Tests in this example would be something more like a central reporting system that's supposed to show what's going on at the intersection. The system (the intersection and lights) can be working totally fine, but the reporting system is showing it messed up (false negative) or messed-up lights can be causing crashes but the reporting system shows it's working fine (false positive). In both cases it should be fixed, but the fix is independent of what - if anything - should be done with the intersection and lights themselves.