4 ms·

Good testing is hard to do - coverage is not a categorical good. You can easily write too many tests that calcify programs and basically just creates a change-

Good testing is hard to do - coverage is not a categorical good. You can easily write too many tests that calcify programs and basically just creates a change-detector program. Oh it looks like you changed something, oh no - all the tests are broken, but it's okay we can now ask the LLM to regenerate them! 100% Coverage! Amazing! What progress!

webdood90 3y ago
> ... basically just creates a change-detector program interesting perspective - why do you think this is a bad thing? to me, it's an opportunity to verify that the change is intended. without it, how do you know that the program does what it is supposed to do?
- whoisjuan 3y ago
  No op, but I don’t think test-driven development resounds with everyone who writes code. I don’t want to write tests for everything. I just want to write the ones that matter.
  - nyrikki 3y ago
    That is a common misconception about TDD. TDD is _about_ writing tests that matter, but most people think it is about writing all unit tests first. If you are following TDD anywhere close to the way it is described, you will only be writing tests that relate to domain functionality first. Note how it is described here, although it is turse. https://martinfowler.com/bliki/TestDrivenDevelopment.html https://martinfowler.com/bliki/TestDrivenDevelopment.html The coverage metric as a goal writing style doesn't work for TDD, sorry you were exposed to that. You are correct that model doesn't work.
    - randomdata 3y ago
      > The coverage metric as a goal writing style doesn't work for TDD Coverage is not a goal of TDD, but in practice you will have 100% coverage by following TDD as you would never have reason to write code that isn't covered by test. Ultimately, the purpose of coverage tools is to let you know what you might have forgotten to clean up during a refactor, to help you remove what you missed.
  - randomdata 3y ago
    TDD or not, why would you write tests for things that don't matter? More importantly, why are you writing any code for things that don't matter?
- siliconc0w 3y ago
  Without deliberate tests it can be very difficult and time consuming to parse out intended change from unwanted or incidental change.
- nyrikki 3y ago
  It tightly couples domain needs with implementation details. Thinking of it as a leaky abstraction helps me. I try hard to separate domain logic tests from implementation specific tests. Your code could be loosely coupled with high cohesion, but with lots of random tests like you get when code coverage is a performance metric, you have to add a lot of complexity that only relates to an implementation.
- elicksaur 3y ago
  How do you know that the tests accurately define what the code is supposed to do? Another way, if you know what the code is supposed to do, why write it down in two places?
  - bbojan 3y ago
    > Another way, if you know what the code is supposed to do, why write it down in two places? This would be like criticizing double-entry accounting by asking "if you know what the amount is, why write it down in two places?" We write the code down in two places because that gives us advantages that far outweigh the added effort: - Once written, your test will catch regressions forever - A test is often excellent documentation on what the code does - It's now much easier to refactor the code, making it more likely that it will be refactored when needed.
- IshKebab 3y ago
  It can be. E.g. consider GUI testing where people sometimes take automatic "golden" screenshots. The problem there is that if you change some minor thing that e.g. moves text by 1 pixel it will fail the test even though that's probably fine. People then get used to just blindly updating the golden images. It becomes basically "the output changed, do you want to continue anyway" which is not the most useful thing. You really want it to say "the output is wrong".
suzzer99 3y ago
Agreed. Good tests are an order of magnitude harder than good code.
- postalrat 3y ago
  Which is why they should be treated as a waste of time unless specific tests can be justified.
- brabel 3y ago
  I don't know where you're getting that from but it's simply wrong. Testing is quite easy IMO. I've been working at the same place for almost 10 years and I introduced our testing framework. We have hundreds of thousands of tests. New guys have some trouble to get started, but after a couple of months they're writing tests for our systems like a pro. Most tests are use-case based or written for checking error-handling. Use-case tests are easy to write: you don't even need to be a programmer (in fact, it's good if use-cases are defined by a Product Owner or Tester), though of course some cases are only known to the programmer as they're the only ones who dive into the details. The programmer should come up with all use-cases the PO missed, of course, and judge whether or not they need to test those too... sometimes it's ok to not test as the cost-benefit is low. Anyway, once you have this use-case based test mentality, it's very easy to write the tests (using a proper language to do it is important! Don't use just JUnit if you're doing Java as it will be really tedious to write and you will stop midway - I know, I've been there... I highly recommend using Spock, though other frameworks to make writing test pleasurable exist). This applies mostly for "integration tests". For unit tests, hopefully you don't find them difficult to write?! I find them quite easy to write since I know how to write testable code, which takes a while to learn but once you do, it's really easy. If you have examples of difficult to write tests, I would be curious to see it! Perhaps we can discuss how to make them easy.
  - viraptor 3y ago
    The positive ones (use cases) are usually pretty straightforward. It's once you get to failures like "how does the whole system recover if one packet, 3 steps into a transaction is corrupted". If the system is complex enough, that warrants a whole internal fault injection framework and can take a really long time to make it reliable and usable. To be fair most projects don't care beyond "rollback the database and maybe display a 5xx error or something". But some do. Anyway, use cases are fine. It's the failures / edge cases that cause pain.
pshc 3y ago
One gig I worked had web component tests where they committed a snapshot of the expected DOM and asserted that the component spat it out... so for every subsequent change the dev would naturally hit the re-generate button and commit it all. Plentiful deltas, questionable signal.
- gen220 3y ago
  Counterpoint, those tests are really useful when you're working on a shared sub-component/library and want to understand the scope of a change, or want to be confident that what should be a no-op change from the caller's point of view is indeed a no-op. But yea 9/10 times that a snapshot test fails, it's noise rather than signal.
hinkley 3y ago
I know for sure that code with no coverage has terrible tests. For everything else I have to read through five other people’s idea of good test. We are all terrible at writing tests. We just find our own ways to do it.
3abiton 3y ago
It's all about the long tail cases.

webdood90 3y ago

> ... basically just creates a change-detector program interesting perspective - why do you think this is a bad thing? to me, it's an opportunity to verify that the change is intended. without it, how do you know that the program does what it is supposed to do?

whoisjuan 3y ago
No op, but I don’t think test-driven development resounds with everyone who writes code. I don’t want to write tests for everything. I just want to write the ones that matter.
- nyrikki 3y ago
  That is a common misconception about TDD. TDD is _about_ writing tests that matter, but most people think it is about writing all unit tests first. If you are following TDD anywhere close to the way it is described, you will only be writing tests that relate to domain functionality first. Note how it is described here, although it is turse. https://martinfowler.com/bliki/TestDrivenDevelopment.html https://martinfowler.com/bliki/TestDrivenDevelopment.html The coverage metric as a goal writing style doesn't work for TDD, sorry you were exposed to that. You are correct that model doesn't work.
  - randomdata 3y ago
    > The coverage metric as a goal writing style doesn't work for TDD Coverage is not a goal of TDD, but in practice you will have 100% coverage by following TDD as you would never have reason to write code that isn't covered by test. Ultimately, the purpose of coverage tools is to let you know what you might have forgotten to clean up during a refactor, to help you remove what you missed.
- randomdata 3y ago
  TDD or not, why would you write tests for things that don't matter? More importantly, why are you writing any code for things that don't matter?
siliconc0w 3y ago
Without deliberate tests it can be very difficult and time consuming to parse out intended change from unwanted or incidental change.
nyrikki 3y ago
It tightly couples domain needs with implementation details. Thinking of it as a leaky abstraction helps me. I try hard to separate domain logic tests from implementation specific tests. Your code could be loosely coupled with high cohesion, but with lots of random tests like you get when code coverage is a performance metric, you have to add a lot of complexity that only relates to an implementation.
elicksaur 3y ago
How do you know that the tests accurately define what the code is supposed to do? Another way, if you know what the code is supposed to do, why write it down in two places?
- bbojan 3y ago
  > Another way, if you know what the code is supposed to do, why write it down in two places? This would be like criticizing double-entry accounting by asking "if you know what the amount is, why write it down in two places?" We write the code down in two places because that gives us advantages that far outweigh the added effort: - Once written, your test will catch regressions forever - A test is often excellent documentation on what the code does - It's now much easier to refactor the code, making it more likely that it will be refactored when needed.
IshKebab 3y ago
It can be. E.g. consider GUI testing where people sometimes take automatic "golden" screenshots. The problem there is that if you change some minor thing that e.g. moves text by 1 pixel it will fail the test even though that's probably fine. People then get used to just blindly updating the golden images. It becomes basically "the output changed, do you want to continue anyway" which is not the most useful thing. You really want it to say "the output is wrong".

suzzer99 3y ago

Agreed. Good tests are an order of magnitude harder than good code.

postalrat 3y ago
Which is why they should be treated as a waste of time unless specific tests can be justified.
brabel 3y ago
I don't know where you're getting that from but it's simply wrong. Testing is quite easy IMO. I've been working at the same place for almost 10 years and I introduced our testing framework. We have hundreds of thousands of tests. New guys have some trouble to get started, but after a couple of months they're writing tests for our systems like a pro. Most tests are use-case based or written for checking error-handling. Use-case tests are easy to write: you don't even need to be a programmer (in fact, it's good if use-cases are defined by a Product Owner or Tester), though of course some cases are only known to the programmer as they're the only ones who dive into the details. The programmer should come up with all use-cases the PO missed, of course, and judge whether or not they need to test those too... sometimes it's ok to not test as the cost-benefit is low. Anyway, once you have this use-case based test mentality, it's very easy to write the tests (using a proper language to do it is important! Don't use just JUnit if you're doing Java as it will be really tedious to write and you will stop midway - I know, I've been there... I highly recommend using Spock, though other frameworks to make writing test pleasurable exist). This applies mostly for "integration tests". For unit tests, hopefully you don't find them difficult to write?! I find them quite easy to write since I know how to write testable code, which takes a while to learn but once you do, it's really easy. If you have examples of difficult to write tests, I would be curious to see it! Perhaps we can discuss how to make them easy.
- viraptor 3y ago
  The positive ones (use cases) are usually pretty straightforward. It's once you get to failures like "how does the whole system recover if one packet, 3 steps into a transaction is corrupted". If the system is complex enough, that warrants a whole internal fault injection framework and can take a really long time to make it reliable and usable. To be fair most projects don't care beyond "rollback the database and maybe display a 5xx error or something". But some do. Anyway, use cases are fine. It's the failures / edge cases that cause pain.

pshc 3y ago

One gig I worked had web component tests where they committed a snapshot of the expected DOM and asserted that the component spat it out... so for every subsequent change the dev would naturally hit the re-generate button and commit it all. Plentiful deltas, questionable signal.

gen220 3y ago
Counterpoint, those tests are really useful when you're working on a shared sub-component/library and want to understand the scope of a change, or want to be confident that what should be a no-op change from the caller's point of view is indeed a no-op. But yea 9/10 times that a snapshot test fails, it's noise rather than signal.

hinkley 3y ago

I know for sure that code with no coverage has terrible tests. For everything else I have to read through five other people’s idea of good test. We are all terrible at writing tests. We just find our own ways to do it.

3abiton 3y ago

It's all about the long tail cases.