5 ms·

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just because it forces you to write a failing test first, and then write the code to make it pass, and you do this over and over as you build up the production code. In this way you are actually proving that the code you are adding is making tests pass. That doesn't mean your tests are great - you are probably still mostly testing the happy path. But then you have a solid foundation to layer mutation/fuzz testing on. I can't tell you how often I see people write tests after writing the production code - and I can go and delete most of the production code and the tests still pass. Writing quality tests for existing code is much harder than writing tests for code that doesn't exist - it is counterintuitive if you haven't done it both ways.

taeric 5y ago
I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code. That is, a mutation suit doesn't test your code, per se. It tests your test suit.
- sverhagen 5y ago
  I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?
  - taeric 5y ago
    No, I found the same. Team disabled them on every release build, as they felt it was too slow. I may bring them back in the release servers, but it is honestly trying to argue for these when nobody else cares.
    - masklinn 5y ago
      It needs someone who cares which us a thankless job but it’s useful as smote tests run nightly / weekly. It tends to be better for broad concerns which trigger somewhat rarely, tend not to be otherwise caught, but can be quite concerning. Crater is a good example of that, it’s infeasible to build & test every public crate on every CI check, but it’s a useful sanity check when release arrives or when messing with the more subtle aspects of the compiler.
thaumasiotes 5y ago
> But once it is "covered" the coverage report is useless. Well, yes, any report that is guaranteed to say the same thing every time is useless.
kragen 5y ago
I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better: 1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests for other parts of the system invoked it, it's not used at all. (This is less likely if you use a lot of test stubs.) 2. Maybe if you didn't think about the scenario where the uncovered code gets invoked when you were writing the test suite, there are other things you also didn't think about—and maybe some of them aren't covered in the implementation either. Write down the missed case and put the implementation away for a while—hopefully long enough to forget how it's implemented. Once you've forgotten, refer to your notes and write a suite of tests that covers the missed case as well as anything similar. — ⁂ — I agree that test-first improves your tests in the way you describe: your test suite is guaranteed to have nearly complete coverage. Also, you have some evidence that the test itself works rather than vacuously passing. But I think there are two other benefits of test-first programming that are commonly undersold. First, it makes programming more fun, because you have immediate feedback when you make a test pass. Second, sometimes you write a test for code you haven't written yet, and you can see by looking at the test that your design sucks: you need seven objects and six method calls to do something simple, and one of the method calls has a boolean parameter, making the code incomprehensible. This feedback allows you to improve your design, possibly several times, before writing the implementation. This allows for faster design iteration than refactoring the implementation toward a better design does. I think this is what jeffbee is saying in https://news.ycombinator.com/item?id=28677978 https://news.ycombinator.com/item?id=28677978.
- vitus 5y ago
  I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough. Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker. ... * Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000 times in the span of 30s.
SV_BubbleTime 5y ago
> Writing quality tests for existing code is much harder than writing tests for code that doesn't exist I’ve been trying this and the biggest place it’s helped me is parsing. Working out all the bad formats, types, unexpected stuff before I write the code was worth it. It’s still probably low quality compared to what it will eventually be, but a pretty good way to start.
SilverRed 5y ago
My favorite peer review trick is reverting the code change and showing that the test still passes. No tests will cover everything but its not hard to apply some simple tricks that raise the bar of actual coverage.
- deleted 5y ago
  [deleted]
optymizer 5y ago
I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.
- jacobsenscott 5y ago
  I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product. Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined problem. It is somewhat unfathomable to me that you can have an idea if what production code to write, but no idea what test to write. Certainly you can "expect this page has a button" if you are about to add a button and write the test first. Or "expect this method to add a record to the database", etc. Certainly you are about to write code that does something, so just write a test to expect that thing to happen. Especially in the case where someone is changing their mind all the time, you are dead in the water without tests to support your changes. You need to manually re-test everything every time they change their mind. Testing after the fact is not that useful for catching regressions because it is unlikely enough tests will be written due to pressure to release. Also a much greater percentage of the tests written after the production code tend to be vacuous tests.
  - brabel 5y ago
    > the result is usually an untested, and usually nearly untestable No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well but it will absolutely slow you down. If you end up not writing tests later, when the implementation has been chosen, it's just because you don't care about the code quality... I find it hard to believe TDD will fix you in that case.

taeric 5y ago

I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code. That is, a mutation suit doesn't test your code, per se. It tests your test suit.

sverhagen 5y ago
I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?
- taeric 5y ago
  No, I found the same. Team disabled them on every release build, as they felt it was too slow. I may bring them back in the release servers, but it is honestly trying to argue for these when nobody else cares.
  - masklinn 5y ago
    It needs someone who cares which us a thankless job but it’s useful as smote tests run nightly / weekly. It tends to be better for broad concerns which trigger somewhat rarely, tend not to be otherwise caught, but can be quite concerning. Crater is a good example of that, it’s infeasible to build & test every public crate on every CI check, but it’s a useful sanity check when release arrives or when messing with the more subtle aspects of the compiler.

thaumasiotes 5y ago

> But once it is "covered" the coverage report is useless. Well, yes, any report that is guaranteed to say the same thing every time is useless.

kragen 5y ago

I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better: 1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests for other parts of the system invoked it, it's not used at all. (This is less likely if you use a lot of test stubs.) 2. Maybe if you didn't think about the scenario where the uncovered code gets invoked when you were writing the test suite, there are other things you also didn't think about—and maybe some of them aren't covered in the implementation either. Write down the missed case and put the implementation away for a while—hopefully long enough to forget how it's implemented. Once you've forgotten, refer to your notes and write a suite of tests that covers the missed case as well as anything similar. — ⁂ — I agree that test-first improves your tests in the way you describe: your test suite is guaranteed to have nearly complete coverage. Also, you have some evidence that the test itself works rather than vacuously passing. But I think there are two other benefits of test-first programming that are commonly undersold. First, it makes programming more fun, because you have immediate feedback when you make a test pass. Second, sometimes you write a test for code you haven't written yet, and you can see by looking at the test that your design sucks: you need seven objects and six method calls to do something simple, and one of the method calls has a boolean parameter, making the code incomprehensible. This feedback allows you to improve your design, possibly several times, before writing the implementation. This allows for faster design iteration than refactoring the implementation toward a better design does. I think this is what jeffbee is saying in https://news.ycombinator.com/item?id=28677978 https://news.ycombinator.com/item?id=28677978.

vitus 5y ago
I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough. Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker. ... * Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000 times in the span of 30s.

SV_BubbleTime 5y ago

> Writing quality tests for existing code is much harder than writing tests for code that doesn't exist I’ve been trying this and the biggest place it’s helped me is parsing. Working out all the bad formats, types, unexpected stuff before I write the code was worth it. It’s still probably low quality compared to what it will eventually be, but a pretty good way to start.

SilverRed 5y ago

My favorite peer review trick is reverting the code change and showing that the test still passes. No tests will cover everything but its not hard to apply some simple tricks that raise the bar of actual coverage.

deleted 5y ago
[deleted]

optymizer 5y ago

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

jacobsenscott 5y ago
I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product. Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined problem. It is somewhat unfathomable to me that you can have an idea if what production code to write, but no idea what test to write. Certainly you can "expect this page has a button" if you are about to add a button and write the test first. Or "expect this method to add a record to the database", etc. Certainly you are about to write code that does something, so just write a test to expect that thing to happen. Especially in the case where someone is changing their mind all the time, you are dead in the water without tests to support your changes. You need to manually re-test everything every time they change their mind. Testing after the fact is not that useful for catching regressions because it is unlikely enough tests will be written due to pressure to release. Also a much greater percentage of the tests written after the production code tend to be vacuous tests.
- brabel 5y ago
  > the result is usually an untested, and usually nearly untestable No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well but it will absolutely slow you down. If you end up not writing tests later, when the implementation has been chosen, it's just because you don't care about the code quality... I find it hard to believe TDD will fix you in that case.