4 ms·

> in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases No.

> in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases No. In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. With more careful testing at their boundaries, so that you can root out edge and corner cases. The results of those tests will then find the ways your software can fail for you. Thinking about it this way, it starts to become clear that a huge part of your job is finding ways to simplify that input space as much as possible. Simplifying at this stage makes getting it right so much easier. The fewer special, edge, and corner cases you allow in the first place, the fewer you have to code for, the fewer you have to test for, the fewer you might accidentally miss, and the fewer others might accidentally run afoul of. This, incidentally, is the key reason why it's good to take some time to define all your tests before you start writing your implementation. I'm not necessarily a huge advocate for "red green refactor", but I do think that at least identifying all your test cases before you ever start implementing can potentially save you boatloads of time, by helping you recognize opportunities to simplify your design before you get locked into an unnecessarily complicated one by a couple hours or days (or months) of sunk cost. It's also the real reason (IMO) why functional programming - as a style, not a kind of language - is such a valuable discipline. The challenge with programming in an imperative style is that it turns your module's entire past history into one of the inputs you need to consider, and that turns your test and specification surface into something that is just so much bigger. Considering that it opens up the possibility of injecting bugs into a routine without ever actually editing the routine itself, or even any of the functions it calls, it's possibly even fair to say that it's unbounded.

pjc50 8y ago
> think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes This is an extremely good way of putting it. Doing some of this at the requirements stage if possible can be very helpful too, because you can detect nonsensical or conflicting requirements before you even write any code.
- TeMPOraL 8y ago
  Seconded. I haven't seen it summarized so well before, and the sentence you quoted is the very one that made me favourite that comment.
  - specialist 8y ago
    Metaphorically, this is functional style programming, mapping inputs to outputs, minimize state and side effects. His conclusion is the money quote for me: "...your job is finding ways to simplify that input space as much as possible." I do this. Instinctively? I feel more or less alone in this practice. My prime motivator is to do less work. Less is more. Concision is a virtue. Blah, blah, blah. The trouble with my strategy is that simple is hard. Much harder than kanban themed JIRA mediated velocity maximizing slapdashery. Also harder to measure. So compared to my peers, it looks like I'm moving in slow motion. To illustrate, someone who fixes the same code pathway multiple times is scored higher than someone (like me) who fixes it (mostly) right the first time. The tortoise vs the hare. -- "If I Had More Time, I Would Have Written a Shorter Letter" -- Blaise Pascal https://quoteinvestigator.com/2012/04/28/shorter-letter/ "-2000 Lines of Code" http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt http://www.folklore.org/StoryView.py?story=Negative_2000_Lin... -- Aside: I was a QA/Test manager for a while. I really cared about this stuff, back in the day. Now it seems no one cares.
- bunderbunder 8y ago
  I'm rather fond of a book, Specification by Example by Gojko Adzic, that's partially a manifesto for, and partially a guide to, doing this.
  - chrisweekly 8y ago
    Thanks for the rec! Just bought for $5 (kindle ed).
    - shady-lady 8y ago
      what site? i can only see manning for $36. not on amazons.
skybrian 8y ago
These are great techniques and should be done more. But I'll add a caveat: they don't cover all the ways software can fail and we probably shouldn't pretend they do. The environment in which the software runs is also an input, and you often don't get to pick that. Consider new browser releases, browser addons, and browser bugs. Performance is a global property of a system that crosses most abstraction boundaries. Outside hard real-time systems, we get few performance guarantees. Some domains are simple on one level and very complicated on another. The set of all <=280 character strings may seem simple as far as the data goes, but setting policy for Twitter is not. Still, being able to decompose problems and treat bugs as either functionality or performance (and not a complicated mixture of both) is pretty helpful.
zbentley 8y ago
> In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. OK, so let's revise the statement to "in order to write effective tests, you must know how your software should behave in the face of all kinds of inputs". In my mind, that's an equivalent, or nearly equivalent* proposition. It's still above and beyond the capabilities or available resources of people developing very complex software. That's not excusing anyone. Just saying that your correction isn't that different from the original statement. * Sure, "someone unplugged the computer while it was running my code" is a failure mode that's not a function of (what most people would consider to be) the input domain. Genuinely external failures like that, while damaging, are rare. What you usually end up defending against are the downstream results of those failures (i.e. "a network partition meant that half of my input dataset was null pointers").
- bunderbunder 8y ago
  Well, that's the idea behind breaking it into equivalence classes. Even if the input space is infinite - say, the set of all tuples of two natural numbers and a string - you should be able to determine that the domain can be carved into only a relatively small number of sets that are interesting. If you find you can't, well, that's what's great about figuring out the test cases first - because now you know that you've got something complicated, and you have a chance to figure out how to make it simpler. Then, ideally, you don't have to even worry about coming up with a mess of individual test cases. You arm yourself with a good QuickCheck style testing framework, so that all you have to do is write down your invariants, your equivalence classes, and what sort of output to expect for each one, and you're mostly done. Just a few hand-coded unit tests to explicitly call out the edge cases, stuff like that. Obviously this s a much bigger job if you're limiting yourself to hand-coding an explicit test method for every single test case. Don't do that. There are so many higher-leverage tools - data-driven tests, property tests, fuzz tests, approval tests, etc. - to use in addition to the basics. It's very much worthwhile to get good at them. Even if you don't always use them, learning how to is good practice for learning how to carve a big problem up into manageable pieces.
wdewind 8y ago
> has to think of the piece of software's entire input domain If you're a startup this is not clear and changes frequently. Practicing TDD in this environment has very real downsides that exist less in mature companies with clearer understanding of business requirements.
rstuart4133 8y ago
My philosophy is a little different. It grew out of writing a thesis on software quality in a post grad course. We studied lots and lots of ways to improve software quality. I can't count the number of papers I read on the subject. The first paper you read is all gung-ho about their particular method of course so you think you are done - you've found the holly grail. Then you find the second paper is equally gung-ho about an entirely different method. Eventually you become jaded by what is really opinions on the effectiveness of the technique their paper explores (dare I say delivered with the almost religious zeal of a true believer) disguised by blizzard of equations, tables and lemma's that take hours to decipher. Being a post grad thing I had been writing software for many years before reading these bloody things. I did learn a whole pile of new techniques (my thesis got a high distinction with the comment "you sure know a lot about improving software quality"), but for most of it I was none the wiser on what I should apply to my everyday work, and that was the point of doing the post grad in the first place. So I changed tack, and started reading papers that reports the results in productivity. It's really hard to measure of course, and these papers were very thin on the ground and their figures involved a lot of hand waving. Even so I was pretty confident in the end the _only_ technique that had a significant effect on productivity was unit testing. Note that considering quality and productivity together is very different from looking at quality alone. The most effective way of reducing defects per line is the method is probably the one developed by IBM when developing code for NASA's early space program. It relies on heavily on formal inspection. In that system programmers weren't allowed to compile their own code - and compile errors where counted in their defect rates. It was spectacularly effective at developing software with low defect rates. Programmers don't usually think of their profession of being shining example of the quality engineers of all types can achieve when they put there minds to it - but this code was just that. 100's of thousands of lines of real time code, not a single failure in flight. Contrast that to the efforts from other engineering disciplines like the O rings, tiles falling off, yada, yada. But most of us don't develop code that must work on first deployment, with catastrophic life and death billion dollar consequences if it doesn't. In the trenches where most of us spend out working lives unit testing delivers more bang per buck then any other sort of quality measure. The reason is simple - it automates testing that would otherwise have to be done manually. Automates it so effectively that it's almost free. If you are wondering how that could have effect net productivity just read the story - the entire team stops work for days to rescue a burning site, people too scared to deploy code. But that's probably dwarfed by the lost sales while the site was down. Still, writing tests take time. Writing too few doesn't get you the quality boost. Writing too many slams productivity. How many tests to write is a puzzle. Your post about all the tests you could write to test everything is certainly pertinent - but if you go down that rabbit hole too far you will spend all your time writing tests. My way of deciding that point isn't like yours. In the end I decided the most important thing an easy way to add test for every line of code. The only proof I can believe is a coverage tool reporting 100% code coverage (or better yet 100% branch coverage if your tool supports it). Handing waving to explain the rest could be right, but to verify it requires a 2nd person to read code _every time it is modified_. Remember we are on about maximising productivity here, so creating manual work like this for every release is really hard to justify. Thus the first step is to write tests for your code until you have 100% code coverage. But it goes further - it must be really easy to find what test covers a particular line of code. This usually means the same line will be tested multiple times - once by the test that targets it, and many other times by tests that happen to use it. Someone programmers view this is inefficient, and it is in a way, but you will see later there is a reason to spend this extra effort. Sadly it is time consuming to verify this has been done in every case as you would have to check every line of the code being tested, but it is amenable to auditing: if you select 10 random lines and they are all good the odds another random line doesn't have an easy to find test is 0.1024%. As anyone who has written a lot of tests knows, this insistence on 100% test coverage has flow on effects to the code being tested. Arranging code so it is easy to spot how you get to the line you want to test can be the most tedious step. It's not unusual for the "writing tests" step to force the code being tested to be refactored for visibility. It is this "refactor your code so it is easy for others that code later to add more tests" that is real goal of the 100% test coverage condition. Once they've hit 100% then, and only then, can the programmer take into consideration the sorts of things you describe. It's difficult to check if they have done that of course - at doing such checks really requires 2 people to think up the tests required, which doubles the work. But perhaps it doesn't matter as at lot will happen anyway if your programmers are diligent in developing test for every subsequent bug you hit. Here diligent means thinking about all the other places this bug may arise, and testing it there too. Relying on this is effectively leveraging off the all the systems testing that follows to produce the sort of tests you describe automagically. Adding these new tests is easy of course because you already have 100% test coverage, and its obvious what code tests each line. You can ensure adding these new tests happens by insisting every programmer report every bug they find in their own library or someone else's into a bug database, and later record the test. Doing it this way does not guarantee your code will have a 0% defect rate. But then nothing, not even what NASA did can do that. What it does guarantee is over time your defect rate will drop, and it will do that with minimal extra work. I find the "extra work" bit is important, because when you are in the "creating the new shiny" mode finding the energy and money to do the work isn't so hard. Hopes should be high is the next big thing that hits the jackpot. But once you are in maintenance mode reality has hit home. Every new addition has to justify every cent spent on it. I suspect the that is because existing KPI's are in place, everybody has a reasonable feel for the impact a change might make - and most changes are small increments. Yet over the years most development happens via the cumulative weight of these small increments. Ensuring the cost of maintaining 100% test coverage during these small incremental changes is minimal is only hope you have of ensuring it's done.

pjc50 8y ago

> think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes This is an extremely good way of putting it. Doing some of this at the requirements stage if possible can be very helpful too, because you can detect nonsensical or conflicting requirements before you even write any code.

TeMPOraL 8y ago
Seconded. I haven't seen it summarized so well before, and the sentence you quoted is the very one that made me favourite that comment.
- specialist 8y ago
  Metaphorically, this is functional style programming, mapping inputs to outputs, minimize state and side effects. His conclusion is the money quote for me: "...your job is finding ways to simplify that input space as much as possible." I do this. Instinctively? I feel more or less alone in this practice. My prime motivator is to do less work. Less is more. Concision is a virtue. Blah, blah, blah. The trouble with my strategy is that simple is hard. Much harder than kanban themed JIRA mediated velocity maximizing slapdashery. Also harder to measure. So compared to my peers, it looks like I'm moving in slow motion. To illustrate, someone who fixes the same code pathway multiple times is scored higher than someone (like me) who fixes it (mostly) right the first time. The tortoise vs the hare. -- "If I Had More Time, I Would Have Written a Shorter Letter" -- Blaise Pascal https://quoteinvestigator.com/2012/04/28/shorter-letter/ "-2000 Lines of Code" http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt http://www.folklore.org/StoryView.py?story=Negative_2000_Lin... -- Aside: I was a QA/Test manager for a while. I really cared about this stuff, back in the day. Now it seems no one cares.
bunderbunder 8y ago
I'm rather fond of a book, Specification by Example by Gojko Adzic, that's partially a manifesto for, and partially a guide to, doing this.
- chrisweekly 8y ago
  Thanks for the rec! Just bought for $5 (kindle ed).
  - shady-lady 8y ago
    what site? i can only see manning for $36. not on amazons.

skybrian 8y ago

These are great techniques and should be done more. But I'll add a caveat: they don't cover all the ways software can fail and we probably shouldn't pretend they do. The environment in which the software runs is also an input, and you often don't get to pick that. Consider new browser releases, browser addons, and browser bugs. Performance is a global property of a system that crosses most abstraction boundaries. Outside hard real-time systems, we get few performance guarantees. Some domains are simple on one level and very complicated on another. The set of all <=280 character strings may seem simple as far as the data goes, but setting policy for Twitter is not. Still, being able to decompose problems and treat bugs as either functionality or performance (and not a complicated mixture of both) is pretty helpful.

zbentley 8y ago

> In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. OK, so let's revise the statement to "in order to write effective tests, you must know how your software should behave in the face of all kinds of inputs". In my mind, that's an equivalent, or nearly equivalent* proposition. It's still above and beyond the capabilities or available resources of people developing very complex software. That's not excusing anyone. Just saying that your correction isn't that different from the original statement. * Sure, "someone unplugged the computer while it was running my code" is a failure mode that's not a function of (what most people would consider to be) the input domain. Genuinely external failures like that, while damaging, are rare. What you usually end up defending against are the downstream results of those failures (i.e. "a network partition meant that half of my input dataset was null pointers").

bunderbunder 8y ago
Well, that's the idea behind breaking it into equivalence classes. Even if the input space is infinite - say, the set of all tuples of two natural numbers and a string - you should be able to determine that the domain can be carved into only a relatively small number of sets that are interesting. If you find you can't, well, that's what's great about figuring out the test cases first - because now you know that you've got something complicated, and you have a chance to figure out how to make it simpler. Then, ideally, you don't have to even worry about coming up with a mess of individual test cases. You arm yourself with a good QuickCheck style testing framework, so that all you have to do is write down your invariants, your equivalence classes, and what sort of output to expect for each one, and you're mostly done. Just a few hand-coded unit tests to explicitly call out the edge cases, stuff like that. Obviously this s a much bigger job if you're limiting yourself to hand-coding an explicit test method for every single test case. Don't do that. There are so many higher-leverage tools - data-driven tests, property tests, fuzz tests, approval tests, etc. - to use in addition to the basics. It's very much worthwhile to get good at them. Even if you don't always use them, learning how to is good practice for learning how to carve a big problem up into manageable pieces.

wdewind 8y ago

> has to think of the piece of software's entire input domain If you're a startup this is not clear and changes frequently. Practicing TDD in this environment has very real downsides that exist less in mature companies with clearer understanding of business requirements.

rstuart4133 8y ago

My philosophy is a little different. It grew out of writing a thesis on software quality in a post grad course. We studied lots and lots of ways to improve software quality. I can't count the number of papers I read on the subject. The first paper you read is all gung-ho about their particular method of course so you think you are done - you've found the holly grail. Then you find the second paper is equally gung-ho about an entirely different method. Eventually you become jaded by what is really opinions on the effectiveness of the technique their paper explores (dare I say delivered with the almost religious zeal of a true believer) disguised by blizzard of equations, tables and lemma's that take hours to decipher. Being a post grad thing I had been writing software for many years before reading these bloody things. I did learn a whole pile of new techniques (my thesis got a high distinction with the comment "you sure know a lot about improving software quality"), but for most of it I was none the wiser on what I should apply to my everyday work, and that was the point of doing the post grad in the first place. So I changed tack, and started reading papers that reports the results in productivity. It's really hard to measure of course, and these papers were very thin on the ground and their figures involved a lot of hand waving. Even so I was pretty confident in the end the _only_ technique that had a significant effect on productivity was unit testing. Note that considering quality and productivity together is very different from looking at quality alone. The most effective way of reducing defects per line is the method is probably the one developed by IBM when developing code for NASA's early space program. It relies on heavily on formal inspection. In that system programmers weren't allowed to compile their own code - and compile errors where counted in their defect rates. It was spectacularly effective at developing software with low defect rates. Programmers don't usually think of their profession of being shining example of the quality engineers of all types can achieve when they put there minds to it - but this code was just that. 100's of thousands of lines of real time code, not a single failure in flight. Contrast that to the efforts from other engineering disciplines like the O rings, tiles falling off, yada, yada. But most of us don't develop code that must work on first deployment, with catastrophic life and death billion dollar consequences if it doesn't. In the trenches where most of us spend out working lives unit testing delivers more bang per buck then any other sort of quality measure. The reason is simple - it automates testing that would otherwise have to be done manually. Automates it so effectively that it's almost free. If you are wondering how that could have effect net productivity just read the story - the entire team stops work for days to rescue a burning site, people too scared to deploy code. But that's probably dwarfed by the lost sales while the site was down. Still, writing tests take time. Writing too few doesn't get you the quality boost. Writing too many slams productivity. How many tests to write is a puzzle. Your post about all the tests you could write to test everything is certainly pertinent - but if you go down that rabbit hole too far you will spend all your time writing tests. My way of deciding that point isn't like yours. In the end I decided the most important thing an easy way to add test for every line of code. The only proof I can believe is a coverage tool reporting 100% code coverage (or better yet 100% branch coverage if your tool supports it). Handing waving to explain the rest could be right, but to verify it requires a 2nd person to read code _every time it is modified_. Remember we are on about maximising productivity here, so creating manual work like this for every release is really hard to justify. Thus the first step is to write tests for your code until you have 100% code coverage. But it goes further - it must be really easy to find what test covers a particular line of code. This usually means the same line will be tested multiple times - once by the test that targets it, and many other times by tests that happen to use it. Someone programmers view this is inefficient, and it is in a way, but you will see later there is a reason to spend this extra effort. Sadly it is time consuming to verify this has been done in every case as you would have to check every line of the code being tested, but it is amenable to auditing: if you select 10 random lines and they are all good the odds another random line doesn't have an easy to find test is 0.1024%. As anyone who has written a lot of tests knows, this insistence on 100% test coverage has flow on effects to the code being tested. Arranging code so it is easy to spot how you get to the line you want to test can be the most tedious step. It's not unusual for the "writing tests" step to force the code being tested to be refactored for visibility. It is this "refactor your code so it is easy for others that code later to add more tests" that is real goal of the 100% test coverage condition. Once they've hit 100% then, and only then, can the programmer take into consideration the sorts of things you describe. It's difficult to check if they have done that of course - at doing such checks really requires 2 people to think up the tests required, which doubles the work. But perhaps it doesn't matter as at lot will happen anyway if your programmers are diligent in developing test for every subsequent bug you hit. Here diligent means thinking about all the other places this bug may arise, and testing it there too. Relying on this is effectively leveraging off the all the systems testing that follows to produce the sort of tests you describe automagically. Adding these new tests is easy of course because you already have 100% test coverage, and its obvious what code tests each line. You can ensure adding these new tests happens by insisting every programmer report every bug they find in their own library or someone else's into a bug database, and later record the test. Doing it this way does not guarantee your code will have a 0% defect rate. But then nothing, not even what NASA did can do that. What it does guarantee is over time your defect rate will drop, and it will do that with minimal extra work. I find the "extra work" bit is important, because when you are in the "creating the new shiny" mode finding the energy and money to do the work isn't so hard. Hopes should be high is the next big thing that hits the jackpot. But once you are in maintenance mode reality has hit home. Every new addition has to justify every cent spent on it. I suspect the that is because existing KPI's are in place, everybody has a reasonable feel for the impact a change might make - and most changes are small increments. Yet over the years most development happens via the cumulative weight of these small increments. Ensuring the cost of maintaining 100% test coverage during these small incremental changes is minimal is only hope you have of ensuring it's done.