6 ms·

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the m

by 13of40 4y ago

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in some code, you're only giving the poor bastard investigating it two puzzles to debug instead of one.

dgb23 4y ago
What about data driven tests where you lay out several variants, including edge cases, for function arguments? Seems pretty clear to me.
midasz 4y ago
I always use (if the scenario is simple enough, which most are): @Test public void myTestMethod_Scenario_ShouldReturnThis() {....
- geraldwhen 4y ago
  Jest makes this far more straightforward. It(“throws when the object belongs to another user”) It(“does a business thing when thing is in state BLAH”)
sirsinsalot 4y ago
To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention. I don't think it quite does it right, but it is of note.
- hgomersall 4y ago
  We have a policy of making each test a spec. That is, a test requires a plain text spec to be attached to it in its doc string. It's kind of like BDD but without all the weird DSLs.
amelius 4y ago
I suppose soon you could ask GPT "what is this code supposed to do?" (I would buy a Copilot subscription for this)
- nosianu 4y ago
  Which would replace all those humans producing perfectly valid sounding explanations that if you invest some research effort have no basis in (the usually far more complex, but also far more fascinating and infinitely deep) reality. So yes, I think AI can indeed replace lots of human-produced thoughts :-) I admit to have been guilty of this myself. I have a famous anecdote-example where I had a very well-paid contractor job and explained something about how my then department's software worked to someone from another department. I think I must have sounded very convincing, the person went off to change something in how they used our stuff. A few minutes later, after accidentally meeting and casually chatting with my boss for that job I realized everything I had said was total garbage. I quickly excused myself from my boss and hurried after the person to tell them to forget and ignore everything I had just explained to them because it was all wrong. I think this last step is not what happens in those cases because we don't usually realize that such a thing just happened. The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link?
  - Phemist 4y ago
    >The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link? Relevant are Sperry & Gazzaniga's split brain experiments. Participants of these experiments had had their corpus callosum (one of the major "information" pathways between our brain's two halves) cut. This was an operation performed to keep epileptic seizures in check. https://en.wikipedia.org/wiki/Split-brain https://en.wikipedia.org/wiki/Split-brain In these participants, specific brain "functions" such as speech were highly lateralized, meaning only one half of the brain was able to perform it to a satisfying degree. Note that these were already not neuro-typical people prior to the experiments (given the regular, debilitating epileptic seizures), so reaching general conclusions from these experiments is hard. Remember also that, like our brains, our bodies are also highly lateralized, such that the right-half of our brain controls the left-side of our body, and the left-half of the brain controls the right-side of our body. If you ever wanted proof against intelligent design, the way our brain connects to our eyes & body is one very strong argument.. Anyway, one experiment stands to mind where one half of the brain was instructed to perform some action (move the left arm, or something similar). Then the other half would be asked _why_ that arm was just moved. It would confabulate, on the spot, totally legit, but obviously bullshit, sounding reasoning. E.g. "I felt cold so I wanted to put on a coat", rather than "the experimenter instructed me to move it". So, rather than claiming "I don't know", it would just make up a plausible reasoning. It is really unimaginable to _not_ know why you moved your arm..
chriswarbo 4y ago
I agree. One nice feature of property-driven testing is that assumptions often end up causing test failures. For example (in ScalaTest): "Average of list" should "be within range" in { forAll() { (l: List[Float]) => { val avg = l.average assert(avg >= l.min && avg <= l.max) } } This test will fail, since it doesn't hold for e.g. empty lists. Requiring non-empty lists will still fail, if we have awkward values like NaNs, etc. The following version has a better chance of passing: "Average of list" should "be within range" in { forAll() { (raw: List[Float]) => { val l = raw.filter(n => !n.isNaN && !n.isInfinite) whenever (l.nonEmpty) { val avg = l.average assert(avg >= l.min && avg <= l.max) } } } Getting this test to pass required us to make those assumptions explicit. Of course, it doesn't spot everything; here's an article which explores this example in more depth (in Python) https://hypothesis.works/articles/calculating-the-mean https://hypothesis.works/articles/calculating-the-mean
debug-desperado 4y ago
So, "given when then" style tests (e.g. Spock) plus a descriptive test name. Or more than that?
ramesh31 4y ago
>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO. https://cucumber.io/docs/gherkin/reference/ https://cucumber.io/docs/gherkin/reference/
- invalidname 4y ago
  This sounds like a good theory but the practice of it is really hard. Pretty quickly you end up with tests that "say" one thing but have nuanced different behavior in the underlying implementation. Then try to debug a "document"... I like the idea. But having tried it at scale, it becomes a mess. Code I can understand. I can read English comments. I can't debug English.
  - deleted 4y ago
    [deleted]
  - brabel 4y ago
    We have it at scale, and no, it doesn't become a mess. We use Spock, which make "comments" a very expected thing, which helps us not let tests without comments pass a code review. Just use a tool that helps you and stop writing stupid tests whose impl code looks worse than the code being tested.
  - urthor 4y ago
    https://www.scalatest.org/at_a_glance/WordSpec https://www.scalatest.org/at_a_glance/WordSpec Seems pretty straightforward.
    - invalidname 4y ago
      Then why all the comments? I know what typical code does. This code looks simple but that's misleading when you're trying to understand a failure. You want consistency and clarity. You want readablity like code is readable not like a book is readable.
randomdata 4y ago
Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentation in an executable form (to be able to automatically validate that the documentation is true).
- hulitu 4y ago
  The test plan is the documentation. That people are cutting corners is unfortunate. A test must be reproduceable. If it is not, is not a test.
- vincnetas 4y ago
  So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable document where intentions and expectations are specified it's much easier to come to common (working) solution. Not all languages have capability to express complex intentions so code as documentation does not work most of the time.
  - alpaca128 4y ago
    Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice that's the kind of doc I tend to have the best experience with. For example the Rust stdlib docs are auto-generated but the language also supports notes and (automatically unit-tested) examples in docstrings which means the API docs are filled with explanations & examples and mentions what assumptions are made about inputs.
quadrifoliate 4y ago
At an old job, one manager would put in his employees' annual reports stuff like "Developer X wrote N automated tests, fixed M bugs, and filed P new bugs this quarter..." The obvious result of Goodhart's Law ensued, leading to test cases like you mention. Lesson to leaders: Please stop your bad managers from pulling stupid crap like this. It wastes a lot more time in the longer run.

by 13of40 4y ago

dgb23 4y ago

What about data driven tests where you lay out several variants, including edge cases, for function arguments? Seems pretty clear to me.

midasz 4y ago

I always use (if the scenario is simple enough, which most are): @Test public void myTestMethod_Scenario_ShouldReturnThis() {....

geraldwhen 4y ago
Jest makes this far more straightforward. It(“throws when the object belongs to another user”) It(“does a business thing when thing is in state BLAH”)

sirsinsalot 4y ago

To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention. I don't think it quite does it right, but it is of note.

hgomersall 4y ago
We have a policy of making each test a spec. That is, a test requires a plain text spec to be attached to it in its doc string. It's kind of like BDD but without all the weird DSLs.

amelius 4y ago

I suppose soon you could ask GPT "what is this code supposed to do?" (I would buy a Copilot subscription for this)

nosianu 4y ago
Which would replace all those humans producing perfectly valid sounding explanations that if you invest some research effort have no basis in (the usually far more complex, but also far more fascinating and infinitely deep) reality. So yes, I think AI can indeed replace lots of human-produced thoughts :-) I admit to have been guilty of this myself. I have a famous anecdote-example where I had a very well-paid contractor job and explained something about how my then department's software worked to someone from another department. I think I must have sounded very convincing, the person went off to change something in how they used our stuff. A few minutes later, after accidentally meeting and casually chatting with my boss for that job I realized everything I had said was total garbage. I quickly excused myself from my boss and hurried after the person to tell them to forget and ignore everything I had just explained to them because it was all wrong. I think this last step is not what happens in those cases because we don't usually realize that such a thing just happened. The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link?
- Phemist 4y ago
  >The brain, or parts of it, are great at producing "explanations". I think that it was part of the more established and reproducible results of psychology that our brain first decides and acts, and only then produces some (often bullshit) "reason" when/if our conscious self asks for one? Does anybody remember if this is true and has a link? Relevant are Sperry & Gazzaniga's split brain experiments. Participants of these experiments had had their corpus callosum (one of the major "information" pathways between our brain's two halves) cut. This was an operation performed to keep epileptic seizures in check. https://en.wikipedia.org/wiki/Split-brain https://en.wikipedia.org/wiki/Split-brain In these participants, specific brain "functions" such as speech were highly lateralized, meaning only one half of the brain was able to perform it to a satisfying degree. Note that these were already not neuro-typical people prior to the experiments (given the regular, debilitating epileptic seizures), so reaching general conclusions from these experiments is hard. Remember also that, like our brains, our bodies are also highly lateralized, such that the right-half of our brain controls the left-side of our body, and the left-half of the brain controls the right-side of our body. If you ever wanted proof against intelligent design, the way our brain connects to our eyes & body is one very strong argument.. Anyway, one experiment stands to mind where one half of the brain was instructed to perform some action (move the left arm, or something similar). Then the other half would be asked _why_ that arm was just moved. It would confabulate, on the spot, totally legit, but obviously bullshit, sounding reasoning. E.g. "I felt cold so I wanted to put on a coat", rather than "the experimenter instructed me to move it". So, rather than claiming "I don't know", it would just make up a plausible reasoning. It is really unimaginable to _not_ know why you moved your arm..

chriswarbo 4y ago

I agree. One nice feature of property-driven testing is that assumptions often end up causing test failures. For example (in ScalaTest): "Average of list" should "be within range" in { forAll() { (l: List[Float]) => { val avg = l.average assert(avg >= l.min && avg <= l.max) } } This test will fail, since it doesn't hold for e.g. empty lists. Requiring non-empty lists will still fail, if we have awkward values like NaNs, etc. The following version has a better chance of passing: "Average of list" should "be within range" in { forAll() { (raw: List[Float]) => { val l = raw.filter(n => !n.isNaN && !n.isInfinite) whenever (l.nonEmpty) { val avg = l.average assert(avg >= l.min && avg <= l.max) } } } Getting this test to pass required us to make those assumptions explicit. Of course, it doesn't spot everything; here's an article which explores this example in more depth (in Python) https://hypothesis.works/articles/calculating-the-mean https://hypothesis.works/articles/calculating-the-mean

debug-desperado 4y ago

So, "given when then" style tests (e.g. Spock) plus a descriptive test name. Or more than that?

ramesh31 4y ago

>You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. This is why I found Gherkin/Cucumber (and BDD in general) to be a total revelation when I first encountered it. No one should be writing tests any other way IMO. https://cucumber.io/docs/gherkin/reference/ https://cucumber.io/docs/gherkin/reference/

invalidname 4y ago
This sounds like a good theory but the practice of it is really hard. Pretty quickly you end up with tests that "say" one thing but have nuanced different behavior in the underlying implementation. Then try to debug a "document"... I like the idea. But having tried it at scale, it becomes a mess. Code I can understand. I can read English comments. I can't debug English.
- deleted 4y ago
  [deleted]
- brabel 4y ago
  We have it at scale, and no, it doesn't become a mess. We use Spock, which make "comments" a very expected thing, which helps us not let tests without comments pass a code review. Just use a tool that helps you and stop writing stupid tests whose impl code looks worse than the code being tested.
- urthor 4y ago
  https://www.scalatest.org/at_a_glance/WordSpec https://www.scalatest.org/at_a_glance/WordSpec Seems pretty straightforward.
  - invalidname 4y ago
    Then why all the comments? I know what typical code does. This code looks simple but that's misleading when you're trying to understand a failure. You want consistency and clarity. You want readablity like code is readable not like a book is readable.

randomdata 4y ago

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentation in an executable form (to be able to automatically validate that the documentation is true).

hulitu 4y ago
The test plan is the documentation. That people are cutting corners is unfortunate. A test must be reproduceable. If it is not, is not a test.
vincnetas 4y ago
So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable document where intentions and expectations are specified it's much easier to come to common (working) solution. Not all languages have capability to express complex intentions so code as documentation does not work most of the time.
- alpaca128 4y ago
  Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice that's the kind of doc I tend to have the best experience with. For example the Rust stdlib docs are auto-generated but the language also supports notes and (automatically unit-tested) examples in docstrings which means the API docs are filled with explanations & examples and mentions what assumptions are made about inputs.

quadrifoliate 4y ago

At an old job, one manager would put in his employees' annual reports stuff like "Developer X wrote N automated tests, fixed M bugs, and filed P new bugs this quarter..." The obvious result of Goodhart's Law ensued, leading to test cases like you mention. Lesson to leaders: Please stop your bad managers from pulling stupid crap like this. It wastes a lot more time in the longer run.