5 ms·

>Tests have nothing to do with code quality. I didn't downvote your comment but I vehemently disagree. Mission-critical code such as NASA flight guidance, avi

by jasode 7y ago

>Tests have nothing to do with code quality. I didn't downvote your comment but I vehemently disagree. Mission-critical code such as NASA flight guidance, avionics, and low-level libraries like SQLite depend on a suite of tests to maintain software quality. (I wrote a previous comment on this.[0]) We also want the new software that commands self-driving cars to have thousands of tests that cover as many scenarios as possible. I don't have inside knowledge of either Waymo or Tesla but it seems like common sense to assume those software programmers rely on a massive suite of unit tests to stress test their cars' decision algorithms. One can't write software with that level complexity that has life-&-death consequences without relying on numerous tests at all layers of the stack. Yes, the cars will still have bugs and will sometimes make the wrong decision but their software would be worse without the tests. High quality software relies on both lower-level unit tests and higher-level integration tests. Or put another way, both "black box" and "white box" testing strategies are used. [0] https://news.ycombinator.com/item?id=15592392 https://news.ycombinator.com/item?id=15592392

commandlinefan 7y ago
I think what he means is that just because you have tests (and even if you have high code coverage) doesn't meant that your code is high quality. They're correlated, but I've actually seen code with high test coverage... whose tests never made any assertions.
- jasode 7y ago
  >tests [...] doesn't meant that your code is high quality. They're correlated, Yes, if they're correlated, that contradicts the absolutist statement of "Tests have nothing to do with code quality." Trying to improve code correctness is directly affecting code quality.
oldmanhorton 7y ago
Isn't this disagreement basically the same point made by Martin about different kinds of quality? SQLites tests don't say the code is architected well and reusable and modular and blah blah blah, it says that it works. When people talk about the quality of NASA code or SQLite, that feels more like external quality than internal quality.
- jasode 7y ago
  >Isn't this disagreement basically the same point made by Martin about different kinds of quality? M Fowler's comment about "tests" was also made in the context of internal quality. He mentions "cruft" as the buildup of bad internal code that the customer can't see: >[...] the best teams both create much less cruft but also remove enough of the cruft they do create that they can continue to add features quickly. They spend time creating _automated tests_ so that they can surface problems quickly and spend less time removing bugs.
- SQLite 7y ago
  The 100% MC/DC testing in SQLite does not force the code to be well-architected, but it does help us to improve the architecture. (1) The 100% branch test coverage requirement forces us to remove unreachable code, or else convert that code into assert() statements, thereby helping to remove cruft. (2) High test coverage gives us freedom to refactor the code aggressively without fear of breaking things. So, if your developers are passionate about long term maintainability, then having 100% MC/DC testing is a big big win. But if your developers are not interested in maintainability, then forcing a 100% MC/DC requirement on them does not help and could make things worse.
streetcat1 7y ago
So self driving systems are based on machine learning, and thus does not have regular (deterministic) unit tests. They will mainly be tested on past data, but the end results are always probabilistic. I.e. no one (not even musk himself) know how the car would behave when it see something that it was not trained on.
- copperx 7y ago
  I've always wondered whether there's a bunch of conditional statements constraining the output of the probabilistic model. That would seem like the logical thing to do, but I'm not that familiar with ML to know whether such thing is needed or not.
a-priori 7y ago
Skilled carpenters use hammers. That doesn't mean a hammer can't cause a lot of damage if used incorrectly.
cryptica 7y ago
I think that unit tests make sense for safety-critical systems but still in those cases, my point would be that it's better to add them near the end of the project once the code has settled.

by jasode 7y ago

commandlinefan 7y ago

I think what he means is that just because you have tests (and even if you have high code coverage) doesn't meant that your code is high quality. They're correlated, but I've actually seen code with high test coverage... whose tests never made any assertions.

jasode 7y ago
>tests [...] doesn't meant that your code is high quality. They're correlated, Yes, if they're correlated, that contradicts the absolutist statement of "Tests have nothing to do with code quality." Trying to improve code correctness is directly affecting code quality.

oldmanhorton 7y ago

Isn't this disagreement basically the same point made by Martin about different kinds of quality? SQLites tests don't say the code is architected well and reusable and modular and blah blah blah, it says that it works. When people talk about the quality of NASA code or SQLite, that feels more like external quality than internal quality.

jasode 7y ago
>Isn't this disagreement basically the same point made by Martin about different kinds of quality? M Fowler's comment about "tests" was also made in the context of internal quality. He mentions "cruft" as the buildup of bad internal code that the customer can't see: >[...] the best teams both create much less cruft but also remove enough of the cruft they do create that they can continue to add features quickly. They spend time creating _automated tests_ so that they can surface problems quickly and spend less time removing bugs.
SQLite 7y ago
The 100% MC/DC testing in SQLite does not force the code to be well-architected, but it does help us to improve the architecture. (1) The 100% branch test coverage requirement forces us to remove unreachable code, or else convert that code into assert() statements, thereby helping to remove cruft. (2) High test coverage gives us freedom to refactor the code aggressively without fear of breaking things. So, if your developers are passionate about long term maintainability, then having 100% MC/DC testing is a big big win. But if your developers are not interested in maintainability, then forcing a 100% MC/DC requirement on them does not help and could make things worse.

streetcat1 7y ago

So self driving systems are based on machine learning, and thus does not have regular (deterministic) unit tests. They will mainly be tested on past data, but the end results are always probabilistic. I.e. no one (not even musk himself) know how the car would behave when it see something that it was not trained on.

copperx 7y ago
I've always wondered whether there's a bunch of conditional statements constraining the output of the probabilistic model. That would seem like the logical thing to do, but I'm not that familiar with ML to know whether such thing is needed or not.

a-priori 7y ago

Skilled carpenters use hammers. That doesn't mean a hammer can't cause a lot of damage if used incorrectly.

cryptica 7y ago

I think that unit tests make sense for safety-critical systems but still in those cases, my point would be that it's better to add them near the end of the project once the code has settled.