3 ms·

I'm interested in your test later approach. When you're attempting 100% branch coverage, do you sometimes find writing the test later causes redesign of the pro

I'm interested in your test later approach. When you're attempting 100% branch coverage, do you sometimes find writing the test later causes redesign of the product code? On average, how many days does it take to complete a cycle of: write the code and then write automated unit test code? What ratio of that time is spent doing test automation versus writing production code?

eesmith 9y ago
I rarely try for 100% branch coverage. I find that the extra tests aren't worthwhile. I don't even try for 100% statement coverage. I use coverage to help identify important missing test cases. For example, some of my code will read/write a binary format of my own devising. The format is used in a "friendly" environment, so I don't have to worry about malicious users trying to cause failures. (The code is in Python, which eliminates a lot of the memory error that a malicious user could use.) I could get 100% coverage by creating corrupted binary files in all of the different ways to fail. But why go through the work of testing unrealistic cases? Instead, I use my experience and intuition to only test the realistic cases. I also know that my code is not mission critical, so if one of my users reports a failure, I'll send them a patch or point release. I can't give you a breakdown of time because I don't track it. My usual estimate is 1/3 programming, 1/3 testing, and 1/3 documentation, for commercial-quality packages. I find that writing good documentation helps find bugs that users are likely to experience. (Now that I think of it, that binary format took several months to write. The first two attempts didn't scale and I had to toss them. They had almost no tests, so think of them as spikes.) For consulting, it depends on the project. In one two-week project I had no tests. The control flow was simple and most of the code was executed during normal use. Plus, the support staff was able to maintain the software in case of errors, and we did a thorough code review for knowledge transfer. In another project, I was brought in to optimize some existing code, which didn't have tests but which had been extensively tested manually. It was presumed to be correct. The new code I wrote also didn't have unit tests. Instead, I cross-compared a few million inputs to check that they gave the same results, at about 10x performance. In yet another project, I had unit tests for the user-facing parts, but not for the main algorithm. Rather than explain the program, I'll use an analogy. Consider a program to factor a number into its primes. This is difficult. On the other hand, it's easy to test that the result is correct. Multiply the numbers to see if they give the original number, and use a primality test to ensure all of the factors are prime. In essence, I have a #define which enables verification. This also slows things down, so it's not enabled by default. I then (once again) processed a few million inputs. This sort of integration testing is much more complete than manual unit tests. (I do have a simple unit test used as a burn test, so it's not completely untested.) For an extreme case, see https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou... . Why write unit tests when it's possible to examine all 2^32 possibilities in 90 seconds?
- lancerkind 9y ago
  Thank you for that. I've got the idea now. In these projects, are you responsible for the maintence and later code changes? Or are these projects one time deliveries and someone else handles things from then on? Also, if you don't mind my asking, how many years of experience did you have in working in the IT industry when first trying your hand at TDD?
  - eesmith 9y ago
    I don't think that's an informative question, because the answer is "yes." Which cases? For software I sell, I am responsible. For software I write as a contractor/consultant, it depends on the contract. After all, I'm not going to maintain it forever, for free. For the two week contract I said I would fix things for 3 months [0], if it was something I should have caught during development. I don't recall them asking for any changes, but like I said, we did a code review at the end to get the two local people up to speed with what it does, and they are good people. The project used a combination of technologies which they hadn't used before. The "yet another project" was a series of work orders across 18 months, where I did two major rewrites to handle new capabilities. Someone local was also reviewing the code, doing validation, and making a few patches. Nearly all of it was me, including maintence and code changes. I've done several projects for that company for about 5 years. Again, I'll fix errors that should have been caught during development. On that last project, which finished in February or so, there was a bug report two weeks ago which I investigated and suggested two possible fixes. They chose one and implemented it themselves. I started programming in 1983 and got my first full-time job in the IT industry in 1995. I didn't try TDD until around 2007 or so. Our local programming user's group does code katas. The organizer is an Agile/XP/TDD facilitator and testing advocate, so I've also learned and had practice that way. Also, she was the project lead for a project where I was involved as a contractor, so when we pair programmed [1] we did it as TDD. [0] Their standard contract said 2 years, which I said was extreme for a two week contract, but if they wanted that then I would adjust my rates accordingly. We settled on 3 months. [1] This wasn't often. I mostly worked on a back-end server component, while she worked on front-end and middleware.
    - 9y ago

eesmith 9y ago

I rarely try for 100% branch coverage. I find that the extra tests aren't worthwhile. I don't even try for 100% statement coverage. I use coverage to help identify important missing test cases. For example, some of my code will read/write a binary format of my own devising. The format is used in a "friendly" environment, so I don't have to worry about malicious users trying to cause failures. (The code is in Python, which eliminates a lot of the memory error that a malicious user could use.) I could get 100% coverage by creating corrupted binary files in all of the different ways to fail. But why go through the work of testing unrealistic cases? Instead, I use my experience and intuition to only test the realistic cases. I also know that my code is not mission critical, so if one of my users reports a failure, I'll send them a patch or point release. I can't give you a breakdown of time because I don't track it. My usual estimate is 1/3 programming, 1/3 testing, and 1/3 documentation, for commercial-quality packages. I find that writing good documentation helps find bugs that users are likely to experience. (Now that I think of it, that binary format took several months to write. The first two attempts didn't scale and I had to toss them. They had almost no tests, so think of them as spikes.) For consulting, it depends on the project. In one two-week project I had no tests. The control flow was simple and most of the code was executed during normal use. Plus, the support staff was able to maintain the software in case of errors, and we did a thorough code review for knowledge transfer. In another project, I was brought in to optimize some existing code, which didn't have tests but which had been extensively tested manually. It was presumed to be correct. The new code I wrote also didn't have unit tests. Instead, I cross-compared a few million inputs to check that they gave the same results, at about 10x performance. In yet another project, I had unit tests for the user-facing parts, but not for the main algorithm. Rather than explain the program, I'll use an analogy. Consider a program to factor a number into its primes. This is difficult. On the other hand, it's easy to test that the result is correct. Multiply the numbers to see if they give the original number, and use a primality test to ensure all of the factors are prime. In essence, I have a #define which enables verification. This also slows things down, so it's not enabled by default. I then (once again) processed a few million inputs. This sort of integration testing is much more complete than manual unit tests. (I do have a simple unit test used as a burn test, so it's not completely untested.) For an extreme case, see https://randomascii.wordpress.com/2014/01/27/theres-only-four-billion-floatsso-test-them-all/ https://randomascii.wordpress.com/2014/01/27/theres-only-fou... . Why write unit tests when it's possible to examine all 2^32 possibilities in 90 seconds?

lancerkind 9y ago
Thank you for that. I've got the idea now. In these projects, are you responsible for the maintence and later code changes? Or are these projects one time deliveries and someone else handles things from then on? Also, if you don't mind my asking, how many years of experience did you have in working in the IT industry when first trying your hand at TDD?
- eesmith 9y ago
  I don't think that's an informative question, because the answer is "yes." Which cases? For software I sell, I am responsible. For software I write as a contractor/consultant, it depends on the contract. After all, I'm not going to maintain it forever, for free. For the two week contract I said I would fix things for 3 months [0], if it was something I should have caught during development. I don't recall them asking for any changes, but like I said, we did a code review at the end to get the two local people up to speed with what it does, and they are good people. The project used a combination of technologies which they hadn't used before. The "yet another project" was a series of work orders across 18 months, where I did two major rewrites to handle new capabilities. Someone local was also reviewing the code, doing validation, and making a few patches. Nearly all of it was me, including maintence and code changes. I've done several projects for that company for about 5 years. Again, I'll fix errors that should have been caught during development. On that last project, which finished in February or so, there was a bug report two weeks ago which I investigated and suggested two possible fixes. They chose one and implemented it themselves. I started programming in 1983 and got my first full-time job in the IT industry in 1995. I didn't try TDD until around 2007 or so. Our local programming user's group does code katas. The organizer is an Agile/XP/TDD facilitator and testing advocate, so I've also learned and had practice that way. Also, she was the project lead for a project where I was involved as a contractor, so when we pair programmed [1] we did it as TDD. [0] Their standard contract said 2 years, which I said was extreme for a two week contract, but if they wanted that then I would adjust my rates accordingly. We settled on 3 months. [1] This wasn't often. I mostly worked on a back-end server component, while she worked on front-end and middleware.
  - 9y ago