7 ms·

> one test per bug fixed this, and you should try to write the test before fixing the bug - you cannot trust a test that you haven't seen failing

draw_down 6y ago
I agree with the last part, but the ordering isn’t crucial. For example you can stash your changes in the implementation to check that the test fails when the change is not present and passes when it is. For this (among other reasons) it is nice to have a test runner that re-runs when it detects file changes. The ordering is one part of TDD that always bugged me- you have to write the tests first. But I often prefer to experiment and try a couple approaches before deciding on one. Having tests first would make a lot more overhead for that way of working.
- wh33zle 6y ago
  To be fair though, if your tests are at the right abstraction level, the specific approach you are choosing for the implementation shouldn't matter for the test. Writing the test first also forces you to think abouy what API do you actually want to expose. Once you got the API right, there is still room for experimenting.
  - draw_down 6y ago
    That’s only true when you have decided what the abstraction will be. That’s my point, a lot of times you don’t know yet!
    - wrs 6y ago
      In this approach you decide on the abstraction (i.e., the API) by writing example code (i.e., some tests) that uses the API. The tests are how you decide which abstraction seems to make the most sense. It sounds like you actually implement the abstraction to decide whether it seems like the right one, which is a lot more work.
      - draw_down 6y ago
        Yes, my position is that tests as client don't really tell you the truth about the abstraction because they don't represent a real usage of it. It is better to write tests for code when you know what it is and what it should do. Tests also introduce a drag on changing strategies- if the choice you made when you wrote them is now not necessarily the optimal one, you must now change your tests or convince yourself that you were actually right the first time. If people like to work this way then great, I'm just explaining why for me it feels bad and runs counter to my instincts.
        wh33zle 6y ago
        I think I understand what you mean. At the same time though, one crucial takeaway for me from Ian's talk is that my tests might be on a too small scale if they are not useful whilst I am changing the implementation strategy. For example, I found it useful to ditch concepts like the testing pyramid and focus on writing e2e tests for my HTTP API instead of trying to cover everything with module or function level tests. That makes it much less likely that they need to change during refactorings and hence provide more value. I generally think that "What is going to break this test?" is a really powerful question to ask to evaluate how good it is. Any answer apart from "a change in requirements" could be a hint that something is odd about the software design. But to ask this question, I need to write the test first or at least think about what kind of test I would write. At some point, writing the actual test might be obsolete as just thinking about it makes you realize a flaw in the design. Other interesting questions I like to ask myself are: "How much risk is this test eliminating?" and "How costly is it to write and maintain?"
        wrs 6y ago
        In reality I tend to do both: write example client code to think through the abstraction (some call this “README-driven development”) and then write tests once the implementation is under way. Though you can get the first as a side effect of the second, I find that good tests aren’t really good example code (too fragmented, focus on edge cases, etc.).
karmelapple 6y ago
Sometimes you’re in a hurry to fix a bug, so you write the functionality and ship it out fast after some manual testing, without an automated test reproducing the bug. That’s okay! Write the test afterwards in some branch, and after committing the tests, make an extra commit that undoes the bug fix. Let your CI run it and confirm it fails there, then bring back the bug fix on the next commit. That way, you can have flexibility to fix things fast, but still keep the regression test that’s proven to check for it around for the future.
- rad_gruchalski 6y ago
  > That’s okay! The effort put into manual tests can be committed directly to adding a test case. Furthermore, a fix should be accepted only when a full test suite with the new tests pass. Such manual testing, depending on what the issue was, may result in a regression in other parts of software. Just a thought...
- afarrell 6y ago
  > to fix things fast Note that if you see _another_ developer taking the time to write a test when you wouldn't, it doesn't necessarily mean they are wasting time. As I am debugging something, I need to write myself a very clear description of my hypothesis of the steps to reproduce the bug -- otherwise it is hard to see incremental progress. I work faster if I can have the machine execute those steps.
  - karmelapple 6y ago
    Fully agreed. I didn't intend to imply a developer writing a test is wasting time, either! Our team has been in situations where we're highly confident of the root cause, but we know creating a test to duplicate this might take hours, if not days. It might even be a fairly finicky scenario to try and setup. Rather than letting customers have to handle the negative consequence of our bug for hours, we'll make the change, run it through our existing test suite (to make sure we're not making yet more troubles for ourselves!), and then release it after another teammate reviews the change. But a test certainly will help with the confidence that the right thing was changed. I would definitely encourage writing a test if it's easy for your group to handle whatever negative consequences the bug existing in the wild is producing for as long as it takes to write a test.
parenthesis 6y ago
I often find that before I've diagnosed and fixed a bug, I don't know what the best test is. So I tend to: fix the bug; write the test; see the test fail on the old version of the code; see the test pass on the new version of the code. Something else I'll throw in on tests. Many times I've caught a bug I would otherwise be introducing because an existing test — which wasn't written to catch my new mistake — fails.

draw_down 6y ago

I agree with the last part, but the ordering isn’t crucial. For example you can stash your changes in the implementation to check that the test fails when the change is not present and passes when it is. For this (among other reasons) it is nice to have a test runner that re-runs when it detects file changes. The ordering is one part of TDD that always bugged me- you have to write the tests first. But I often prefer to experiment and try a couple approaches before deciding on one. Having tests first would make a lot more overhead for that way of working.

wh33zle 6y ago
To be fair though, if your tests are at the right abstraction level, the specific approach you are choosing for the implementation shouldn't matter for the test. Writing the test first also forces you to think abouy what API do you actually want to expose. Once you got the API right, there is still room for experimenting.
- draw_down 6y ago
  That’s only true when you have decided what the abstraction will be. That’s my point, a lot of times you don’t know yet!
  - wrs 6y ago
    In this approach you decide on the abstraction (i.e., the API) by writing example code (i.e., some tests) that uses the API. The tests are how you decide which abstraction seems to make the most sense. It sounds like you actually implement the abstraction to decide whether it seems like the right one, which is a lot more work.
    - draw_down 6y ago
      Yes, my position is that tests as client don't really tell you the truth about the abstraction because they don't represent a real usage of it. It is better to write tests for code when you know what it is and what it should do. Tests also introduce a drag on changing strategies- if the choice you made when you wrote them is now not necessarily the optimal one, you must now change your tests or convince yourself that you were actually right the first time. If people like to work this way then great, I'm just explaining why for me it feels bad and runs counter to my instincts.
      - wh33zle 6y ago
        I think I understand what you mean. At the same time though, one crucial takeaway for me from Ian's talk is that my tests might be on a too small scale if they are not useful whilst I am changing the implementation strategy. For example, I found it useful to ditch concepts like the testing pyramid and focus on writing e2e tests for my HTTP API instead of trying to cover everything with module or function level tests. That makes it much less likely that they need to change during refactorings and hence provide more value. I generally think that "What is going to break this test?" is a really powerful question to ask to evaluate how good it is. Any answer apart from "a change in requirements" could be a hint that something is odd about the software design. But to ask this question, I need to write the test first or at least think about what kind of test I would write. At some point, writing the actual test might be obsolete as just thinking about it makes you realize a flaw in the design. Other interesting questions I like to ask myself are: "How much risk is this test eliminating?" and "How costly is it to write and maintain?"
      - wrs 6y ago
        In reality I tend to do both: write example client code to think through the abstraction (some call this “README-driven development”) and then write tests once the implementation is under way. Though you can get the first as a side effect of the second, I find that good tests aren’t really good example code (too fragmented, focus on edge cases, etc.).

karmelapple 6y ago

Sometimes you’re in a hurry to fix a bug, so you write the functionality and ship it out fast after some manual testing, without an automated test reproducing the bug. That’s okay! Write the test afterwards in some branch, and after committing the tests, make an extra commit that undoes the bug fix. Let your CI run it and confirm it fails there, then bring back the bug fix on the next commit. That way, you can have flexibility to fix things fast, but still keep the regression test that’s proven to check for it around for the future.

rad_gruchalski 6y ago
> That’s okay! The effort put into manual tests can be committed directly to adding a test case. Furthermore, a fix should be accepted only when a full test suite with the new tests pass. Such manual testing, depending on what the issue was, may result in a regression in other parts of software. Just a thought...
afarrell 6y ago
> to fix things fast Note that if you see _another_ developer taking the time to write a test when you wouldn't, it doesn't necessarily mean they are wasting time. As I am debugging something, I need to write myself a very clear description of my hypothesis of the steps to reproduce the bug -- otherwise it is hard to see incremental progress. I work faster if I can have the machine execute those steps.
- karmelapple 6y ago
  Fully agreed. I didn't intend to imply a developer writing a test is wasting time, either! Our team has been in situations where we're highly confident of the root cause, but we know creating a test to duplicate this might take hours, if not days. It might even be a fairly finicky scenario to try and setup. Rather than letting customers have to handle the negative consequence of our bug for hours, we'll make the change, run it through our existing test suite (to make sure we're not making yet more troubles for ourselves!), and then release it after another teammate reviews the change. But a test certainly will help with the confidence that the right thing was changed. I would definitely encourage writing a test if it's easy for your group to handle whatever negative consequences the bug existing in the wild is producing for as long as it takes to write a test.

parenthesis 6y ago

I often find that before I've diagnosed and fixed a bug, I don't know what the best test is. So I tend to: fix the bug; write the test; see the test fail on the old version of the code; see the test pass on the new version of the code. Something else I'll throw in on tests. Many times I've caught a bug I would otherwise be introducing because an existing test — which wasn't written to catch my new mistake — fails.