8 ms·

Unit testing anti-patterns: Structural Inspection

148 points by ScottWRobinson 8y ago

ThalesX 8y ago
Hmm, maybe some sort of introduction of why one might try to unit test their code structure would help some people like me. I don't think I've ever had the though of unit testing my code structure. What might I use it for?
- maxxxxx 8y ago
  I could imagine this making sense in a dynamic language like PHP. I don't know enough JavaScript to know if it's applicable there. But it seems one of the things that if used extensively make the test suite a major obstacle to refactoring because it tests way too many internal details.
  - Gibbon1 8y ago
    Even worse because refactoring and bug fixing will always break unit tests coders will get in the habit of reflexively updating unit tests when they fail.
- adrianratnapala 8y ago
  I assume people fall into doing it without realising. Unit tests are supposed to hermetically test individual components of a system. In object-orient environments, those components are often classes. But classes are typically quite small, very often a particular class exists to some implementation detail of your overall system -- this is especially true of the various Vistors, Factories, Adaptors etc. promoted by OOP theory. When you combine "unit test every class" and "express your code structure through classes" you accidentally end up with "unit test your structure".
  - projektir 8y ago
    This inevitably happens if your process insists on high test coverage.
- deleted 8y ago
  [deleted]
nevir 8y ago
It helps to ask "what is most likely to change about the code I am testing?" when writing unit tests. Try to avoid testing those things, and instead focus on testing the code's behavior (quack)
- hinkley 8y ago
  And then some pinhead starts cheerleading 100% code coverage. Usually long before you’ve gotten your test quality high enough to be sustainable.
  - adrianratnapala 8y ago
    What I have trouble with is even gauging the quality of existing tests.
    - crdoconnor 8y ago
      If you track failures and categorizing them as either failures that caught a bug or false positives then you basically have a good measure for test quality.
    - hinkley 8y ago
      Good tests are hard. Beware of anyone who tries to pretend otherwise. To build on what crdoconnor said, empirical evidence helps. Do a post mortem every time. Try to recall what was going on when the problem started and see if there are any reasonable measures you can undertake in the future. It's the typical iterative process of learning something as you go along. When you discover you made a mistake, try to figure out how not to make it again. Every time you get a regression in the code ask yourself "didn't we have a test for that?" If the answer is 'no' then think back to the last time someone (maybe you!) talked you out of spending more time on tests. Is the hassle now worth the hassle you avoided then? Occasionally this is the case, and the response is "shit happens" (if this is not rare, then you might need an attitude adjustment). If you find that prevention would have been better, try to remember that next time you're thinking about cutting back on tests. If the answer was 'yes', then something has gone wrong with your testing strategy. Was it always broken, or did subsequent code changes break it? When you fix the test(s), would the way they look now have made sense at the time you wrote them to begin with? Should new tests you write look more like these do? Should you go back right now and fix other tests to make them look this way? Just keep thinking. Verify your assumptions. Chuck the ones that don't hold water. Watch how other people deal with similar problems, and steal their ideas (or learn from their mistakes).
marcosdumay 8y ago
Action anti-patterns: Shooting yourself on the foot. Avoid this. The fact that this article is something that must be written, and it's clear why it's here and why it was upvoted is depressing.
- sametmax 8y ago
  There is nothing obvious in life. I often feel like you do, so when I notice I go this way, I read https://xkcd.com/1053/ https://xkcd.com/1053/ again
  - petersellers 8y ago
    On top of that, even if this is something you might have seen before it helps to put a name to a face, so to speak. Structural Inspection is a good name for this anti-pattern.
- deleted 8y ago
  [deleted]
- Rexxar 8y ago
  I think that the root of such test is the idea that "I write more test than you so i'm better" mentality. Similarly to evaluating productivity to number of line of code, evaluating "seriousness/professionalism" on number of tests bring us this sort of irrational behaviours.
mlthoughts2018 8y ago
Structural inspection is sometimes a very good test idea, especially in a dynamically typed language like Python. For example, a certain class might get dynamically enhanced with extra methods or extra structure at run time, depending on some config or something. I’ve found many examples like this where needing to conform the design to a static typing compiler would be a huge waste of time and involve some stupid pile of abstraction that is totally inappropriate for the problem. Instead, there may be 3 or 4 highly controlled “mix-in” options (where the choice of mix-in happens at run time and is dynamically modified or configurable), and it’s perfectly safe to write it with monkey-patching or dynamically defining a class with type(...). In the unit tests, you may sincerely want to verify that if the system is fed config that should cause some new data members or functions to be monkeypatched into some instances, that it does in fact cause this to happen. I think the point is that in some languages, the structure of the program is itself customizeable or modifiable and so there are functions whose unit of work is to effect that modification, and those functions need to be tested too. Some examples might be checking that a certain class instance gets endowed with a specialized context manager implementation or special data model overrides or special iteration behavior. I’ve experienced many cases where statically typed languages inherently disallow this as a possibility, and you end up having to engineer huge multiple-dispatch structures and highly genericized interfaces to get a similar effect of dynamically selecting structure at run time, and often that big pile of abstraction is hugely limiting and disallows you from efficiently making big changes to it, even in cases when the programmer really can guarantee safety and limits on complexity for their special use case.
- mistrial9 8y ago
  extensive OO gymnastics is an acquired taste, and best practiced by someone who is willing to deal with the dynamic mess that will inevitably result in a mature code base. Nothing you say is wrong, yet none of this is necessary to solve the vast majority of actual problems.
  - mlthoughts2018 8y ago
    I’d say the static typing analogue to the workflow I described is perhaps the canonical OO gymnastics and leads to much worse messes, thoughtless regurgitation of “design patterns” and wasted hours getting some tower of abstraction in place for something that you can just do in dynamic languages and make it perfectly safe and move on. “The dynamic mess” is mostly a myth we tell ourselves because many dominant languages are predicated on there needing to be actual material benefit to type safety. In reality, bad use of abstraction will cause a mess no matter which paradigm you use, but dynamic languages at least give you the option to eschew the bullshit and just solve the problem, while many mainstream statically typed languages, even functional languages, just give you pattern recipes that gaurantee you’ll get lost in a sea of value-destructive abstraction and end up using a bazooka for every nail.
mwkaufma 8y ago
Alternatively, don't write a runtime test for something the static type system can encode directly. In this case, have three members for the subprocessors with their concrete types, rather than a generic virtual-method-dispatch list.
mikekchar 8y ago
Although I don't disagree that the testing of structure in the example given is not particularly good practice, I disagree with the overall message. There is a difference between "unit tests" and "acceptance tests". "Does this test verify some business requirement" is a question aimed at the latter, not the former. There is nothing wrong with writing automated acceptance tests (and I highly encourage it), but they are not a replacement for unit tests. What are unit tests for? They are for helping you reason about the effect of changes on your code. There are two main questions a unit test should answer. 1: What happens when I do X (where X is calling some code)? 2: Does the behaviour change if I modify the production code? The first question is for helping you write the code in the first place (or for modifying it subsequently). It is not a question of the application operating properly, it's a question of the "unit" operating the way you expect. You should be able to use this information to reason quickly about what went wrong in the case that behaviour in the system changes unexpectedly: "Unit a" should behave like X, and I can see that it does so by looking at the tests. If that's the case, then "Unit y" must be at fault. You can think of these kinds of tests as being sort of like predefined watch points in your debugger. You should expose state in your units and add tests so that you can understand the state of the unit if it doesn't behave as expected. You should be thinking, "If the behaviour of this unit is unexpected, what things would I want to look at when determining why". You might be wondering (as many do) what a "unit" is. The answer is that it can be anything: a function, a class, a global object, a group of classes/objects acting in concert. It really doesn't matter. Some people like to fix the idea of a "unit" as a class and also add "integration tests". I have no problem with that, other than I have seen no real value in the practice. Mostly I think people find it easier to envision how to name the files that contain their tests. Many styles of testing do a relatively good job of doing what I describe above. However many people isolate their "units" too much. I think this is an unfortunate consequence of the word "unit" which seems to imply "atomic unit" to many people. The word (IMHO) refers more to simply an identifiable chunk of code. Often people avoid using real objects as collaborators in their tests in the mistaken belief that it will help with the specificity of their tests. The second question, "Does the behaviour change if I modify the production code" is as important as the first question. Imagine a scenario where you have a series of either manual or automated acceptance tests. You run those tests and you feel confident that the application is running as you expect. You then modify some code. At this point, it would be very nice to be told what behaviour (if anything) has changed in the application. You might be forgiven for thinking that's the same as a regression test. However, it is not necessary to actually write regression tests to give you that information. We don't actually need to know whether or not the behaviour is correct -- we only need to know if it is different (and ideally, what is different). As an example, I once worked on code that converted files to and from MS Word file format. I didn't write any of the original code and it was pretty stinky, TBH. Every time I made a change, I would likely break something because I couldn't reason about the code. However, we had about 2000 documents that QA had already tested and deemed acceptable. I wrote a quick hack that would allow me to compare the "accepted" output for these documents with the new outcomes. It didn't try to understand what was wrong, just that it was different. In fact, most of the time the changes were good because I was improving the conversions. Obviously, the above is not a unit test system. However, the point is that we don't need to write complicated tests to determine if the system is working correctly. All we need is to have a known baseline for the system and then be alerted when the behaviour changes -- either in a "good" or "bad" way. Then we can have a quick look to see if the change in behaviour was what we expected. In that way, the second criteria for good unit tests is that at least one unit test "fails" when I change the behaviour of the system. A lot of people equate TDD (Test Driven Development) with Test First Development. This is unfortunate because it steers people off course. Test First is a good way to learn TDD, but it's not by any stretch of the imagination the only way to do TDD. When I have a piece of code with good unit tests, I will often modify production code first and see what tests fail. If I have changed behaviour and no tests fail, I know I'm in big trouble. Similarly, if 200 tests fail and I have no specificity in my tests, I know I'm going to be in for a hard ride. These are the kinds of things you should be doing to test the quality of your tests. I'll finish this with one last thought. There are 2 classes of "customer" for tests: the end user and the programmer. The user is concerned about whether or not the application is working as desired. They don't care what is technically wrong and they don't need to reason about how to change unexpected behaviour. The programmer doesn't actually need to know what the correct behaviour is for everything in the system. They need to have a known baseline of behaviour and to be told when the behaviour has changed. They also need to know if the very specific piece of code that they are currently working on is behaving as expected. The customer for unit tests are the programmer, not the end user. Having tests for the end user is a great idea, but doesn't actually help tremendously with improving the development experience. The golden rule for programming tests as a programmer: if it helps you reason about what you are doing, then it's a good idea. If it doesn't, then you should avoid it.
lifeisstillgood 8y ago
Overall, try to constantly ask yourself a question: does this test verify some business requirement? a pretty good guide. Analogous to "test the interfaces / contract"
KKKKkkkk1 8y ago
Overall, try to constantly ask yourself a question: does this test verify some business requirement? If the answer is no, remove it. The most valuable tests are always the tests that have at least some connection to the business requirements your code base is ought to address. Not surprisingly, such tests also fall into the formal definition I brought in the beginning of this article: they are likely to give a protection against regression errors and they are unlikely to turn red without a good reason. Easier said than done. Let's say I'm writing a game AI and I wrote a test that confirms that ogres can eat humans but cannot eat other ogres. Is this testing a business requirement?
- sudhirj 8y ago
  Yes that’s exactly what it is. If you business is selling a game that works, ogres not eating other ogres is a requirement for your game to be considered bug free and sell well. (YMMV, depending on who your audience is ogre cannibal feat might also be fun). This is as opposed to testing that that the eat method on the ogre only accepts an instance of the Human class. If you do that you’ll have a brittle system on your hands, and when your ogres decide to eat horses (business decision) you’ll have to change a lot of tests and will instead procrastinate by writing a blog post about why TDD sucks.
  - domlebo70 8y ago
    A good type system makes the 2nd change you proposed a joy to make.
    - sudhirj 8y ago
      Oh, absolutely. A good type system also makes unnecessary to write tests asserting that your implementation uses exact / concrete types. A good type system would allow you to have a clean and sane 'Eatable' concept (interface / trait / protocol) that you can use and assert instead.
- lgessler 8y ago
  I think you might be taking "business requirement" in too narrow a sense. Maybe a better expression would be "external behavioral requirement", i.e. something your program has to "appear to do" from some outside perspective.
exabrial 8y ago
People still use underscores in front of variables to delineate scope? It's 2018, why are you using notepad to code. Hilariously, they also use 'var' because that communicates type information to the next guy.
gt_ 8y ago
I’ve been learning testing the last few months (been programming around a year) and this logic seems pretty apparent, and also fairly impossible to adhere to. More so, this idea of TDD seems a little ridiculous so far, and structural testing is why, but I wonder if this is just becuse I’m going through a period of wanting to refactor a lot while building something. TDD is clearly appropriate for adding features, but when architecting central components, I think it’s cost me a lot of time. Some of that time is writing and verifying extra tests but most of it is reviewing a mountain of tests each time I modify architecture. Is this just a learning phase? Or is there a long-term lesson to learn here?
- eropple 8y ago
  Testing basically breaks down into the following types (and you can flavor them however you want, but take this as a starting point): - test-last testing to test the implementation you just wrote - test-first testing to test the implementation you're gonna write - test-driven design, where the tests reflect the specification and are applied to the implementation So, with these on hand, here's my litmus test for TDD: Do I know what I'm building and why? You can't meaningfully do test-driven design if you don't know the what and the why. TDD tests need to encode "the business cares about this", and IME, the questions you are asking often arise when somebody doesn't really yet know what they're building. You're figuring it out. And that is okay, too! I'm not a TDD devotee (in that I would love to write code in a TDD fashion much more than I actually do, but I think it's probably the right answer for most situations where the unknowns have been solved or pushed out) and I think that you're describing a very common situation where writing tests while you're writing code, or maybe even after--though this is often fraught--is totally fine. I regularly build systems by writing tests to exercise bits and pieces of them in lieu of opening a REPL or running it through a command line; the Test Cops aren't gonna come get you if this works better during your exploratory phase. (It does mean your exploratory phase has some useful artifacts when it's done, though.) Put another way: if I have a spec, I can do TDD. If I don't, I can't, and I don't lose sleep over it.
  - gt_ 8y ago
    Hilarious about the test cops. Testing seems to, so far, be a fairly wishy-washy task. The spec thing makes a lot of sense. The project I’m learning with is not ideal because the test environment takes a moment to boot but it’s peculiar and challenging me to consider real-world scenarios, which may or may not be a good thing but I’m near wrapping it up. It’s a plug-in for a few different content creation softwares. I would say I knew what I was building, but there have been unknowns about which particular approach will meet my standards and still be DRY. They all worked great with prototypes but I discovered some edge cases late in construction. I found TDD very satisfying but now I have an instinctual doubt about it. Your thoughts make a lot of sense. I guess one of the main things will do next time is avoid all the extra unit testing I did early on. I think I was honestly having too much fun. Someone should do research on if automated test coverage illicits the slot-machine effect.
slaymaker1907 8y ago
Kind of specific, but I find that in the React world, Enzyme is worse than useless for the same reasons as mentioned in this article. The tests you end up writing just verify your XML (which generally has little to no value). Even if there is some tricky logic which needs testing, you can pretty much always just refactor your code a little bit and just use standard unit tests.
- onion2k 8y ago
  Enzyme is for writing integration tests rather than unit tests - a component is a whole bunch of different things interacting with the DOM. You can't really break that down in to individual units of code, so you can't unit test it. If you have complex logic in a component that needs unit tests you should probably pull that code out in to its own class and test that separately.
- Zalastax 8y ago
  Manually writing Enzyme tests looks dumb to me but what about Jest snapshots? Their only purpose is to detect changes to prevent accidental behaviour changes. There's very little work that needs to be done to maintain those tests so I want to think they provide value. An explicit git log of how the rendered view has changed seems useful to me - especially in a pull request.
- gedy 8y ago
  That is opposite my experience, we use Enzyme to assert behavior of our components and actively avoid checking the JSX. Jest snapshots sounds closer to what you are describing
sudhirj 8y ago
The reduction to absurdity is tests that essentially parse the code they're testing and assert the AST. Any unit test should really be from the point of view of checking what the code does, as opposed to checking what code was written. Structural tests by definition don't check what the code does - they're way to parse the code and check the AST is structured a certain way.
jmartrican 8y ago
Either you KNOW you didn't introduce any new bugs or you don't know and are relying on hope. Testing makes the difference. I really do not care how you arrive to 100% certainty, just get there. If it means testing some structural part of your code (and other unit-testing), or it means settings up integration tests that do end to end tests on live systems, just get there. We can talk about increasing efficiency in this process as a team. For some systems it might be easier to do integration tests, for others its more efficient to do unit-tests.

Hmm, maybe some sort of introduction of why one might try to unit test their code structure would help some people like me. I don't think I've ever had the though of unit testing my code structure. What might I use it for?

maxxxxx 8y ago
I could imagine this making sense in a dynamic language like PHP. I don't know enough JavaScript to know if it's applicable there. But it seems one of the things that if used extensively make the test suite a major obstacle to refactoring because it tests way too many internal details.
- Gibbon1 8y ago
  Even worse because refactoring and bug fixing will always break unit tests coders will get in the habit of reflexively updating unit tests when they fail.
adrianratnapala 8y ago
I assume people fall into doing it without realising. Unit tests are supposed to hermetically test individual components of a system. In object-orient environments, those components are often classes. But classes are typically quite small, very often a particular class exists to some implementation detail of your overall system -- this is especially true of the various Vistors, Factories, Adaptors etc. promoted by OOP theory. When you combine "unit test every class" and "express your code structure through classes" you accidentally end up with "unit test your structure".
- projektir 8y ago
  This inevitably happens if your process insists on high test coverage.
deleted 8y ago
[deleted]

nevir 8y ago

It helps to ask "what is most likely to change about the code I am testing?" when writing unit tests. Try to avoid testing those things, and instead focus on testing the code's behavior (quack)

hinkley 8y ago
And then some pinhead starts cheerleading 100% code coverage. Usually long before you’ve gotten your test quality high enough to be sustainable.
- adrianratnapala 8y ago
  What I have trouble with is even gauging the quality of existing tests.
  - crdoconnor 8y ago
    If you track failures and categorizing them as either failures that caught a bug or false positives then you basically have a good measure for test quality.
  - hinkley 8y ago
    Good tests are hard. Beware of anyone who tries to pretend otherwise. To build on what crdoconnor said, empirical evidence helps. Do a post mortem every time. Try to recall what was going on when the problem started and see if there are any reasonable measures you can undertake in the future. It's the typical iterative process of learning something as you go along. When you discover you made a mistake, try to figure out how not to make it again. Every time you get a regression in the code ask yourself "didn't we have a test for that?" If the answer is 'no' then think back to the last time someone (maybe you!) talked you out of spending more time on tests. Is the hassle now worth the hassle you avoided then? Occasionally this is the case, and the response is "shit happens" (if this is not rare, then you might need an attitude adjustment). If you find that prevention would have been better, try to remember that next time you're thinking about cutting back on tests. If the answer was 'yes', then something has gone wrong with your testing strategy. Was it always broken, or did subsequent code changes break it? When you fix the test(s), would the way they look now have made sense at the time you wrote them to begin with? Should new tests you write look more like these do? Should you go back right now and fix other tests to make them look this way? Just keep thinking. Verify your assumptions. Chuck the ones that don't hold water. Watch how other people deal with similar problems, and steal their ideas (or learn from their mistakes).

marcosdumay 8y ago

Action anti-patterns: Shooting yourself on the foot. Avoid this. The fact that this article is something that must be written, and it's clear why it's here and why it was upvoted is depressing.

sametmax 8y ago
There is nothing obvious in life. I often feel like you do, so when I notice I go this way, I read https://xkcd.com/1053/ https://xkcd.com/1053/ again
- petersellers 8y ago
  On top of that, even if this is something you might have seen before it helps to put a name to a face, so to speak. Structural Inspection is a good name for this anti-pattern.
deleted 8y ago
[deleted]
Rexxar 8y ago
I think that the root of such test is the idea that "I write more test than you so i'm better" mentality. Similarly to evaluating productivity to number of line of code, evaluating "seriousness/professionalism" on number of tests bring us this sort of irrational behaviours.

mlthoughts2018 8y ago

Structural inspection is sometimes a very good test idea, especially in a dynamically typed language like Python. For example, a certain class might get dynamically enhanced with extra methods or extra structure at run time, depending on some config or something. I’ve found many examples like this where needing to conform the design to a static typing compiler would be a huge waste of time and involve some stupid pile of abstraction that is totally inappropriate for the problem. Instead, there may be 3 or 4 highly controlled “mix-in” options (where the choice of mix-in happens at run time and is dynamically modified or configurable), and it’s perfectly safe to write it with monkey-patching or dynamically defining a class with type(...). In the unit tests, you may sincerely want to verify that if the system is fed config that should cause some new data members or functions to be monkeypatched into some instances, that it does in fact cause this to happen. I think the point is that in some languages, the structure of the program is itself customizeable or modifiable and so there are functions whose unit of work is to effect that modification, and those functions need to be tested too. Some examples might be checking that a certain class instance gets endowed with a specialized context manager implementation or special data model overrides or special iteration behavior. I’ve experienced many cases where statically typed languages inherently disallow this as a possibility, and you end up having to engineer huge multiple-dispatch structures and highly genericized interfaces to get a similar effect of dynamically selecting structure at run time, and often that big pile of abstraction is hugely limiting and disallows you from efficiently making big changes to it, even in cases when the programmer really can guarantee safety and limits on complexity for their special use case.

mistrial9 8y ago
extensive OO gymnastics is an acquired taste, and best practiced by someone who is willing to deal with the dynamic mess that will inevitably result in a mature code base. Nothing you say is wrong, yet none of this is necessary to solve the vast majority of actual problems.
- mlthoughts2018 8y ago
  I’d say the static typing analogue to the workflow I described is perhaps the canonical OO gymnastics and leads to much worse messes, thoughtless regurgitation of “design patterns” and wasted hours getting some tower of abstraction in place for something that you can just do in dynamic languages and make it perfectly safe and move on. “The dynamic mess” is mostly a myth we tell ourselves because many dominant languages are predicated on there needing to be actual material benefit to type safety. In reality, bad use of abstraction will cause a mess no matter which paradigm you use, but dynamic languages at least give you the option to eschew the bullshit and just solve the problem, while many mainstream statically typed languages, even functional languages, just give you pattern recipes that gaurantee you’ll get lost in a sea of value-destructive abstraction and end up using a bazooka for every nail.

mwkaufma 8y ago

Alternatively, don't write a runtime test for something the static type system can encode directly. In this case, have three members for the subprocessors with their concrete types, rather than a generic virtual-method-dispatch list.

mikekchar 8y ago

Although I don't disagree that the testing of structure in the example given is not particularly good practice, I disagree with the overall message. There is a difference between "unit tests" and "acceptance tests". "Does this test verify some business requirement" is a question aimed at the latter, not the former. There is nothing wrong with writing automated acceptance tests (and I highly encourage it), but they are not a replacement for unit tests. What are unit tests for? They are for helping you reason about the effect of changes on your code. There are two main questions a unit test should answer. 1: What happens when I do X (where X is calling some code)? 2: Does the behaviour change if I modify the production code? The first question is for helping you write the code in the first place (or for modifying it subsequently). It is not a question of the application operating properly, it's a question of the "unit" operating the way you expect. You should be able to use this information to reason quickly about what went wrong in the case that behaviour in the system changes unexpectedly: "Unit a" should behave like X, and I can see that it does so by looking at the tests. If that's the case, then "Unit y" must be at fault. You can think of these kinds of tests as being sort of like predefined watch points in your debugger. You should expose state in your units and add tests so that you can understand the state of the unit if it doesn't behave as expected. You should be thinking, "If the behaviour of this unit is unexpected, what things would I want to look at when determining why". You might be wondering (as many do) what a "unit" is. The answer is that it can be anything: a function, a class, a global object, a group of classes/objects acting in concert. It really doesn't matter. Some people like to fix the idea of a "unit" as a class and also add "integration tests". I have no problem with that, other than I have seen no real value in the practice. Mostly I think people find it easier to envision how to name the files that contain their tests. Many styles of testing do a relatively good job of doing what I describe above. However many people isolate their "units" too much. I think this is an unfortunate consequence of the word "unit" which seems to imply "atomic unit" to many people. The word (IMHO) refers more to simply an identifiable chunk of code. Often people avoid using real objects as collaborators in their tests in the mistaken belief that it will help with the specificity of their tests. The second question, "Does the behaviour change if I modify the production code" is as important as the first question. Imagine a scenario where you have a series of either manual or automated acceptance tests. You run those tests and you feel confident that the application is running as you expect. You then modify some code. At this point, it would be very nice to be told what behaviour (if anything) has changed in the application. You might be forgiven for thinking that's the same as a regression test. However, it is not necessary to actually write regression tests to give you that information. We don't actually need to know whether or not the behaviour is correct -- we only need to know if it is different (and ideally, what is different). As an example, I once worked on code that converted files to and from MS Word file format. I didn't write any of the original code and it was pretty stinky, TBH. Every time I made a change, I would likely break something because I couldn't reason about the code. However, we had about 2000 documents that QA had already tested and deemed acceptable. I wrote a quick hack that would allow me to compare the "accepted" output for these documents with the new outcomes. It didn't try to understand what was wrong, just that it was different. In fact, most of the time the changes were good because I was improving the conversions. Obviously, the above is not a unit test system. However, the point is that we don't need to write complicated tests to determine if the system is working correctly. All we need is to have a known baseline for the system and then be alerted when the behaviour changes -- either in a "good" or "bad" way. Then we can have a quick look to see if the change in behaviour was what we expected. In that way, the second criteria for good unit tests is that at least one unit test "fails" when I change the behaviour of the system. A lot of people equate TDD (Test Driven Development) with Test First Development. This is unfortunate because it steers people off course. Test First is a good way to learn TDD, but it's not by any stretch of the imagination the only way to do TDD. When I have a piece of code with good unit tests, I will often modify production code first and see what tests fail. If I have changed behaviour and no tests fail, I know I'm in big trouble. Similarly, if 200 tests fail and I have no specificity in my tests, I know I'm going to be in for a hard ride. These are the kinds of things you should be doing to test the quality of your tests. I'll finish this with one last thought. There are 2 classes of "customer" for tests: the end user and the programmer. The user is concerned about whether or not the application is working as desired. They don't care what is technically wrong and they don't need to reason about how to change unexpected behaviour. The programmer doesn't actually need to know what the correct behaviour is for everything in the system. They need to have a known baseline of behaviour and to be told when the behaviour has changed. They also need to know if the very specific piece of code that they are currently working on is behaving as expected. The customer for unit tests are the programmer, not the end user. Having tests for the end user is a great idea, but doesn't actually help tremendously with improving the development experience. The golden rule for programming tests as a programmer: if it helps you reason about what you are doing, then it's a good idea. If it doesn't, then you should avoid it.

lifeisstillgood 8y ago

Overall, try to constantly ask yourself a question: does this test verify some business requirement? a pretty good guide. Analogous to "test the interfaces / contract"

KKKKkkkk1 8y ago

Overall, try to constantly ask yourself a question: does this test verify some business requirement? If the answer is no, remove it. The most valuable tests are always the tests that have at least some connection to the business requirements your code base is ought to address. Not surprisingly, such tests also fall into the formal definition I brought in the beginning of this article: they are likely to give a protection against regression errors and they are unlikely to turn red without a good reason. Easier said than done. Let's say I'm writing a game AI and I wrote a test that confirms that ogres can eat humans but cannot eat other ogres. Is this testing a business requirement?

sudhirj 8y ago
Yes that’s exactly what it is. If you business is selling a game that works, ogres not eating other ogres is a requirement for your game to be considered bug free and sell well. (YMMV, depending on who your audience is ogre cannibal feat might also be fun). This is as opposed to testing that that the eat method on the ogre only accepts an instance of the Human class. If you do that you’ll have a brittle system on your hands, and when your ogres decide to eat horses (business decision) you’ll have to change a lot of tests and will instead procrastinate by writing a blog post about why TDD sucks.
- domlebo70 8y ago
  A good type system makes the 2nd change you proposed a joy to make.
  - sudhirj 8y ago
    Oh, absolutely. A good type system also makes unnecessary to write tests asserting that your implementation uses exact / concrete types. A good type system would allow you to have a clean and sane 'Eatable' concept (interface / trait / protocol) that you can use and assert instead.
lgessler 8y ago
I think you might be taking "business requirement" in too narrow a sense. Maybe a better expression would be "external behavioral requirement", i.e. something your program has to "appear to do" from some outside perspective.

exabrial 8y ago

People still use underscores in front of variables to delineate scope? It's 2018, why are you using notepad to code. Hilariously, they also use 'var' because that communicates type information to the next guy.

gt_ 8y ago

I’ve been learning testing the last few months (been programming around a year) and this logic seems pretty apparent, and also fairly impossible to adhere to. More so, this idea of TDD seems a little ridiculous so far, and structural testing is why, but I wonder if this is just becuse I’m going through a period of wanting to refactor a lot while building something. TDD is clearly appropriate for adding features, but when architecting central components, I think it’s cost me a lot of time. Some of that time is writing and verifying extra tests but most of it is reviewing a mountain of tests each time I modify architecture. Is this just a learning phase? Or is there a long-term lesson to learn here?

eropple 8y ago
Testing basically breaks down into the following types (and you can flavor them however you want, but take this as a starting point): - test-last testing to test the implementation you just wrote - test-first testing to test the implementation you're gonna write - test-driven design, where the tests reflect the specification and are applied to the implementation So, with these on hand, here's my litmus test for TDD: Do I know what I'm building and why? You can't meaningfully do test-driven design if you don't know the what and the why. TDD tests need to encode "the business cares about this", and IME, the questions you are asking often arise when somebody doesn't really yet know what they're building. You're figuring it out. And that is okay, too! I'm not a TDD devotee (in that I would love to write code in a TDD fashion much more than I actually do, but I think it's probably the right answer for most situations where the unknowns have been solved or pushed out) and I think that you're describing a very common situation where writing tests while you're writing code, or maybe even after--though this is often fraught--is totally fine. I regularly build systems by writing tests to exercise bits and pieces of them in lieu of opening a REPL or running it through a command line; the Test Cops aren't gonna come get you if this works better during your exploratory phase. (It does mean your exploratory phase has some useful artifacts when it's done, though.) Put another way: if I have a spec, I can do TDD. If I don't, I can't, and I don't lose sleep over it.
- gt_ 8y ago
  Hilarious about the test cops. Testing seems to, so far, be a fairly wishy-washy task. The spec thing makes a lot of sense. The project I’m learning with is not ideal because the test environment takes a moment to boot but it’s peculiar and challenging me to consider real-world scenarios, which may or may not be a good thing but I’m near wrapping it up. It’s a plug-in for a few different content creation softwares. I would say I knew what I was building, but there have been unknowns about which particular approach will meet my standards and still be DRY. They all worked great with prototypes but I discovered some edge cases late in construction. I found TDD very satisfying but now I have an instinctual doubt about it. Your thoughts make a lot of sense. I guess one of the main things will do next time is avoid all the extra unit testing I did early on. I think I was honestly having too much fun. Someone should do research on if automated test coverage illicits the slot-machine effect.

slaymaker1907 8y ago

Kind of specific, but I find that in the React world, Enzyme is worse than useless for the same reasons as mentioned in this article. The tests you end up writing just verify your XML (which generally has little to no value). Even if there is some tricky logic which needs testing, you can pretty much always just refactor your code a little bit and just use standard unit tests.

onion2k 8y ago
Enzyme is for writing integration tests rather than unit tests - a component is a whole bunch of different things interacting with the DOM. You can't really break that down in to individual units of code, so you can't unit test it. If you have complex logic in a component that needs unit tests you should probably pull that code out in to its own class and test that separately.
Zalastax 8y ago
Manually writing Enzyme tests looks dumb to me but what about Jest snapshots? Their only purpose is to detect changes to prevent accidental behaviour changes. There's very little work that needs to be done to maintain those tests so I want to think they provide value. An explicit git log of how the rendered view has changed seems useful to me - especially in a pull request.
gedy 8y ago
That is opposite my experience, we use Enzyme to assert behavior of our components and actively avoid checking the JSX. Jest snapshots sounds closer to what you are describing

sudhirj 8y ago

The reduction to absurdity is tests that essentially parse the code they're testing and assert the AST. Any unit test should really be from the point of view of checking what the code does, as opposed to checking what code was written. Structural tests by definition don't check what the code does - they're way to parse the code and check the AST is structured a certain way.

jmartrican 8y ago

Either you KNOW you didn't introduce any new bugs or you don't know and are relying on hope. Testing makes the difference. I really do not care how you arrive to 100% certainty, just get there. If it means testing some structural part of your code (and other unit-testing), or it means settings up integration tests that do end to end tests on live systems, just get there. We can talk about increasing efficiency in this process as a team. For some systems it might be easier to do integration tests, for others its more efficient to do unit-tests.