6 ms·

I was ambivalent on unit tests until I discovered how much the mere act of writing them was finding bugs. I very vividly remember writing a test for a ~40 loc

by donatj 3y ago

I was ambivalent on unit tests until I discovered how much the mere act of writing them was finding bugs. I very vividly remember writing a test for a ~40 loc class of pure functions. I started out thinking the exercise was a waste of time. This class is simple, has no mutable state, and should have no reason to change. Why bother testing it? By the time I was done writing the test I had found three major bugs in that 40 loc, and it was a major aha moment. Truly enlightening.

linsomniac 3y ago
That reminds me of this time I wrote some code to add a method to Python string objects. The first reply to my issue on it in the bug tracker was "We shouldn't accept this, it's trivial to implement in your own code, see: XXXX". The second reply was "You have a bug in your implementation in the first reply." It took a couple years to be accepted.
- nerdponx 3y ago
  Sounds familiar. Was that str.removeprefix?
  - linsomniac 3y ago
    str.rsplit()
jamesu 3y ago
I bumped into so many corner case and dumb bugs on a recent python project that I'm even more of a unit testing enthusiast than before. Past a certain level of complexity they are definitely a net benefit.
- throwaway2037 3y ago
  You mentioned Python. I struggle with the weak(er) typing. It is a bottomless well of bugs. Did your unit tests find type issues or (business) logic / state issues?
  - jamesu 3y ago
    It was more business logic issues relating to conditional logic which needed to factor in a lot of edge cases. I don't think strong typing would have helped much in this case.
throwaway290 3y ago
Was this statically or dynamically typed language?
- eddd-ddde 3y ago
  I don't think it really matters. Major bugs are not "oh this can be null", major bugs are "this combination of preconditions yield a business logic edge case that wasn't accounted for". Static typing doesn't really help more than dynamic typing in these cases.
  - throwaway290 3y ago
    It matters enough that the question "does static typing dramatically reduce the benefits of unit testing" is an open question or at least seriously discussed in the industry. All other replies are about dynamic languages.
    - ska 3y ago
      I don't think it is really unresolved. Static languages with sufficiently rich and modifiable type systems avoid a fraction of cases where you may well want a unit test, but it's not the overwhelming majority. Merely static helps too but not all that much. So while there is a reduction, it's a stretch to call it "dramatic".
      - throwaway290 3y ago
        > I don't think it is really unresolved Well, you are currently participating in a thread that discusses this very question so there's that... and such threads are regular on HN. I meant just that. People discuss it. What you want to say is that you have a strong opinion about it, that's OK and still possible with open questions
        ska 3y ago
        People discuss lots of things that are pretty well solved; i wouldn't equate "open question" with "lots of discussion". I guess in this context I mean that the question of static vs. dynamic in unit testing turns out to not be that hard, but the questions like "what is a unit test" and "should we unit test at all" are much muddier. Because people are confused or argumentative about the latter, they tend to pull the former into discussions that don't really have much to do with static vs. dynamic.
virtue3 3y ago
according to some studies it's around 50 bugs per 1,000 LoC. So that puts it at about 1 bug per 20 LoC Some estimates are something as high as 75 bugs per 1,000 LoC but that many bugs don't make it out to customers because of QA / Developer actions. So yeah, right on the money.
IanCal 3y ago
I had this kind of thing when it came to property based testing. I built a property based testing library for ActionScript 3 (a fun journey in itself, with full test case reduction). I was testing my testing library, and tried one of the most basic tests: For any object A A == decode(encode(A)) And discovered the fun of floating point values not being perfectly representable as strings. The more significant one came from testing a UI library we'd built for TVs (so you has up, down, left, right as movement). We had in the spec that if you moved focus by pressing right, pressing left would take you back to the thing you were on before. The test looked something like For an arbitrary series of API calls generating the UI: For an arbitrary list of movements the user makes: If the focus changes, pressing the opposite direction moves your focus back where you came from Now, this was actually very easy to write as a test, but it's extremely powerful. It found a bug in an interesting corner case, so I fixed it. Fixing the bug broke an existing unit test. I checked and the unit test correctly tested something in the spec. The spec was inconsistent, but because we'd tested explicit examples it had never been spotted. I've been a convert since. I've never implemented property tests in an existing project without finding some bug. I don't recommend building your own property testing library unless you really want to, I highly recommend in python hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
- Tainnor 3y ago
  Floating point values are definitely perfectly representable as strings - worst case, you just output the binary as a string - but what you may refer to is that many exact fractions can't be represented exactly as floats and/or that floating-point arithmetic doesn't obey "normal" rules of arithmetic (e.g. addition isn't associative).
  - IanCal 3y ago
    I'll clarify, as yes it's possible, but not using the built in conversions from strings to floats. That is, the following is not true for all x: Number(String(x)) == x
    - 3y ago
joshjje 3y ago
I think they are definitely valuable. My issue is I would have to disassemble most of the large legacy code base to be able to effectively test things, and a large part of it is UI (Windows Forms). I know you can do it, but just takes so much time and effort. We do have some though.

by donatj 3y ago

linsomniac 3y ago

That reminds me of this time I wrote some code to add a method to Python string objects. The first reply to my issue on it in the bug tracker was "We shouldn't accept this, it's trivial to implement in your own code, see: XXXX". The second reply was "You have a bug in your implementation in the first reply." It took a couple years to be accepted.

nerdponx 3y ago
Sounds familiar. Was that str.removeprefix?
- linsomniac 3y ago
  str.rsplit()

jamesu 3y ago

I bumped into so many corner case and dumb bugs on a recent python project that I'm even more of a unit testing enthusiast than before. Past a certain level of complexity they are definitely a net benefit.

throwaway2037 3y ago
You mentioned Python. I struggle with the weak(er) typing. It is a bottomless well of bugs. Did your unit tests find type issues or (business) logic / state issues?
- jamesu 3y ago
  It was more business logic issues relating to conditional logic which needed to factor in a lot of edge cases. I don't think strong typing would have helped much in this case.

throwaway290 3y ago

Was this statically or dynamically typed language?

eddd-ddde 3y ago
I don't think it really matters. Major bugs are not "oh this can be null", major bugs are "this combination of preconditions yield a business logic edge case that wasn't accounted for". Static typing doesn't really help more than dynamic typing in these cases.
- throwaway290 3y ago
  It matters enough that the question "does static typing dramatically reduce the benefits of unit testing" is an open question or at least seriously discussed in the industry. All other replies are about dynamic languages.
  - ska 3y ago
    I don't think it is really unresolved. Static languages with sufficiently rich and modifiable type systems avoid a fraction of cases where you may well want a unit test, but it's not the overwhelming majority. Merely static helps too but not all that much. So while there is a reduction, it's a stretch to call it "dramatic".
    - throwaway290 3y ago
      > I don't think it is really unresolved Well, you are currently participating in a thread that discusses this very question so there's that... and such threads are regular on HN. I meant just that. People discuss it. What you want to say is that you have a strong opinion about it, that's OK and still possible with open questions
      - ska 3y ago
        People discuss lots of things that are pretty well solved; i wouldn't equate "open question" with "lots of discussion". I guess in this context I mean that the question of static vs. dynamic in unit testing turns out to not be that hard, but the questions like "what is a unit test" and "should we unit test at all" are much muddier. Because people are confused or argumentative about the latter, they tend to pull the former into discussions that don't really have much to do with static vs. dynamic.

virtue3 3y ago

according to some studies it's around 50 bugs per 1,000 LoC. So that puts it at about 1 bug per 20 LoC Some estimates are something as high as 75 bugs per 1,000 LoC but that many bugs don't make it out to customers because of QA / Developer actions. So yeah, right on the money.

IanCal 3y ago

I had this kind of thing when it came to property based testing. I built a property based testing library for ActionScript 3 (a fun journey in itself, with full test case reduction). I was testing my testing library, and tried one of the most basic tests: For any object A A == decode(encode(A)) And discovered the fun of floating point values not being perfectly representable as strings. The more significant one came from testing a UI library we'd built for TVs (so you has up, down, left, right as movement). We had in the spec that if you moved focus by pressing right, pressing left would take you back to the thing you were on before. The test looked something like For an arbitrary series of API calls generating the UI: For an arbitrary list of movements the user makes: If the focus changes, pressing the opposite direction moves your focus back where you came from Now, this was actually very easy to write as a test, but it's extremely powerful. It found a bug in an interesting corner case, so I fixed it. Fixing the bug broke an existing unit test. I checked and the unit test correctly tested something in the spec. The spec was inconsistent, but because we'd tested explicit examples it had never been spotted. I've been a convert since. I've never implemented property tests in an existing project without finding some bug. I don't recommend building your own property testing library unless you really want to, I highly recommend in python hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/

Tainnor 3y ago
Floating point values are definitely perfectly representable as strings - worst case, you just output the binary as a string - but what you may refer to is that many exact fractions can't be represented exactly as floats and/or that floating-point arithmetic doesn't obey "normal" rules of arithmetic (e.g. addition isn't associative).
- IanCal 3y ago
  I'll clarify, as yes it's possible, but not using the built in conversions from strings to floats. That is, the following is not true for all x: Number(String(x)) == x
  - 3y ago

joshjje 3y ago

I think they are definitely valuable. My issue is I would have to disassemble most of the large legacy code base to be able to effectively test things, and a large part of it is UI (Windows Forms). I know you can do it, but just takes so much time and effort. We do have some though.