4 ms·

To be honest, I think this is one of the few areas where Joel is off the mark. In my experience, the "tester" is a useless position that usually only exists be

To be honest, I think this is one of the few areas where Joel is off the mark. In my experience, the "tester" is a useless position that usually only exists because of a deficiency in one of two areas. The first reason you need testers is that your programmers can't - for whatever reason - turn out code that works. Either they don't sufficiently understand the problem scope, they're overworked, or they're just lazy and don't bother running through enough of their own quality controls before releasing changes. The second area is that you aren't getting the right feedback from customers. Joel uses an example in his article: With long-cycle shrinkwrap software, it can take a year or more to hear feedback from customers. The problem is the cycle is too long. Fix that. I find mostly this is due to more basic things, like separating client service from programming (hello feedback!), allowing salesmen to mollycoddle their clients, or not even bothering to ask or listen. The "tester" is the response to the existence of either one of both of the symptom above. By their very existence, they are a sign that you are either working in a company that has placed itself in a less than ideal market (selling enterprise software, for example), is poorly led, or has leadership that doesn't listen. All this thinking hearkens back to computer engineering as building a house; and analogy so flawed that even PG saw fit to write a book with a more apt one. This has been my experience and I have yet to see a situation where it isn't true.

argv_empty 17y ago
The first reason you need testers is that your programmers can't - for whatever reason - turn out code that works. And it's pretty unreasonable to expect them to do so. They can run their own small set of test cases, but a full test of a significant program is a lot of work, which (as mentioned in another thread) includes a lot of hard-to-automate grinding. That grind work can be done much cheaper (and probably more reliably) by a dedicated tester.
- lincolnq 17y ago
  Yes -- much more reliably. The testers on my team save me so much time. My code runs on six platforms. Every time I switch platforms it takes a lot of time and energy to switch physical and mental contexts. I have no hope of thoroughly testing all the platforms for each little change I make. So I usually just compile the code on two or three platforms, test on one, and check it in. The incremental build servers will tell me if I broke the build within a few minutes, and the testers will tell me if I made some platform-specific breakage the next day. Furthermore, for each bug, the tester has narrowed it to specific repro steps, including what DOESN'T work, and usually has a screenshot of the problem and what's expected. Usually by reading the report, I immediately have a good idea of what's broken and that saves me tons of time too (as opposed to a vague "the xyz component is broken, fix plz" which I have seen in lots of shops where non-testers write bug reports). I love our testers.
- run4yourlives 17y ago
  That grind work can be done much cheaper (and probably more reliably) by a dedicated tester. I won't dispute that, as that is a major reason why companies love testers. The problem is that while the testing is done cheaper and perhaps better, that's not really the goal of the whole process now, is it? The goal of the process is to match the product to the customers expectation/need. I submit that the best possible way to do this is to have programmers that are also managing that need, even if it is more expensive and/or less efficient when view specifically.
  - jister 17y ago
    I think the best way is to have a specification coming from the customer indicating their needs and expectation. Any feature that's not mentioned in the document cannot be considered a "bug". Based from this specification, testers/QA can not write their test cases.
    - nostrademons 17y ago
      You will never get that specification. Most people don't know what they need until you show it to them. Specification-driven development is a great way to write software that gets you paid, gets some purchasing manager a bonus, and yet pisses off every end user who comes in contact with it.
pixelbath 17y ago
To me, your experience sounds an awful lot like not having much experience in a production environment. In my experience, context-switching between programming and testing gets quite cumbersome with difficult problems, and can lead to overlooking smaller issues. If I'm busy testing every single edge case I can think of, who's doing the development? With separate testers, you get the benefit of concurrency (the software can continue being tested while issues from the previous version are fixed), and subsequently more software developed. "The problem is the cycle is too long. Fix that." How so? What if I'm developing a new product that has a full specification and design document, and is expected to meet customer acceptance goals? I don't see a deficiency in the development cycle if I need someone else to help test my software in a manner that I simply don't have time for. Does your assertion mean that any software that contains bugs is caused by lack of programmer skill? I, too, would like to learn the secret to writing completely issue-free code if you're willing to share it. Now, please don't take what I say to mean programmers shouldn't test their own creations. If you hand something that crashes 100% of the time to a tester, you're being lazy. The purpose of testers goes far beyond simply checking to see if it's "code that works."
- run4yourlives 17y ago
  To me, your experience sounds an awful lot like not having much experience in a production environment. Actually on the contrary, it comes from a growing understanding that the 'production environment' is in fact the problem! In my experience, context-switching between programming and testing gets quite cumbersome with difficult problems, and can lead to overlooking smaller issues. Whether or not a programmer or a designated tester is actually doing the work has no bearing on over-testing. In fact I'd argue that this is more likely to occur if you use a tester. * If I'm busy testing every single edge case I can think of, who's doing the development?* Nobody! That's the crux of the matter right there: 1. Testing sucks, so if someone else is doing it then it's better for you, and, more importantly 2. the pressure to continue to develop new features at speeds that are unsuistainable all but ensure that you'll eventually find yourself with so much technical debt that it will kill you. What if I'm developing a new product that has a full specification and design document, and is expected to meet customer acceptance goals? I'd say run away from this waterfall design process as fast as possible, to be frank. I don't see a deficiency in the development cycle if I need someone else to help test my software in a manner that I simply don't have time for. I do. I see a development cycle where you haven't been allocated enough time! Does your assertion mean that any software that contains bugs is caused by lack of programmer skill? No, because in order for a bug to be a bug it must not only exist in the code but also in the feature set. (i.e. it must be found) The fact of the matter is that nobody - not you, not your testers, and certainly not your customers knows exactly how the software will be used until it is actually being used.
RiderOfGiraffes 17y ago
With respect, I suggest your experience is limited. I deploy embedded safety-critical software into establishments such as nuclear naval bases. We need testers because while the code works (we use formal and semi-formal software derivation techniques) there are, on occasion, unexpected interactions between features. You need to run through combinations of configurations and actions, and some of it can't be scripted because the interactions are unexpected - the Black Swan principle. Our cycle is a long one in part because acquisition programs in government and quasi-governmental organisations are like that. We can't deploy changes every day into the field. For one thing, we usually have access. For another, any change needs associated training and manuals. It would be a waste of a talented programmer to run through the sorts of detailed and exhaustive testing our software has to pass before it will be accepted. Perhaps you simply think that my company is placed in a less than ideal market. I assure you, it's worth being in the game.
- run4yourlives 17y ago
  Perhaps you simply think that my company is placed in a less than ideal market. I assure you, it's worth being in the game. I do, only in the sense that in your market, your costs can justify your operations in a way that not many markets can match. Joel makes bug trackers, so I stayed in that context although I spoke in very black and white terms. I'm not intending to profess the ultimate solution to everyone's process, but I am trying to speak to a majority of start-up centric developers-cum-managers. Doing what you do... hell hire 25 testers for every component; just don't kill anyone.
  - jister 17y ago
    "Doing what you do... hell hire 25 testers for every component; just don't kill anyone. I think I miss this one but how did you know he needed 25 testers for every component?
    - nostrademons 17y ago
      Hyperbole.
dasil003 17y ago
The second area is that you aren't getting the right feedback from customers. Yes, if you are bringing your customers into the office and actually watching them use your product. Usability testing is and always has been one of the most lacking areas of web development (probably other types of consumer software as well, but web is my area of expertise). A good tester can address these issues up front more than anyone else in the organization. Everyone else is too close to project to give it a fresh look. It's no substitute for real usability testing, but it's a good deal closer than anything else internal.
ericb 17y ago
> Either they don't sufficiently understand the problem scope, they're overworked, or they're just lazy and don't bother running through enough of their own quality controls before releasing changes. The number of possible system states in larger systems can be enormous. I feel like to have this opinion, perhaps you have worked mainly on smaller systems? In larger systems, there can be so many possibilities that it would take several times longer to test than to code, and without reading spec docs, it is very difficult to tell what correct even is. For example, a student loan system I worked on fit this bill. The glib answer would be to say it was so complex because of poor programming, but that's not true. The legal reality it was modeling was complex and so the software could not be less complex than the underlying system's rules. I have worked on several such systems in different industries. Now, you could argue that all programmers should become programming/QA people, but that would be self-defeating as programmer time is generally more expensive per-hour than QA time. Further, the type of person who is good at running through 400 steps of detail-oriented rote testing is not always the same person that you would like doing high-level design. Specialization makes life more efficient. Even if they did happen to be the same person, if they enjoyed one task more, why wouldn't they go to a job where they could focus on that type of task?
- run4yourlives 17y ago
  I feel like to have this opinion, perhaps you have worked mainly on smaller systems? No, I've got a good mix, and I agree with you that this is a product of large systems more than simple web apps and the like. The complexity of large systems being required is what's the problem though; not that I have an immediate solution. That and too many programmers that simply aren't able to do anything but program.
  - derefr 17y ago
    You can't say that complexity itself is the problem, because the whole of programming is encoding complex processes so as to simplify them from a human perspective. Without complexity, it's not testing that would go away, it's programming. You've basically said something like "the problem with space shuttles is that they have to aim their thrust precisely in order to achieve lift"—that's not the problem with space shuttles, that's the idea of space shuttles.

argv_empty 17y ago

The first reason you need testers is that your programmers can't - for whatever reason - turn out code that works. And it's pretty unreasonable to expect them to do so. They can run their own small set of test cases, but a full test of a significant program is a lot of work, which (as mentioned in another thread) includes a lot of hard-to-automate grinding. That grind work can be done much cheaper (and probably more reliably) by a dedicated tester.

lincolnq 17y ago
Yes -- much more reliably. The testers on my team save me so much time. My code runs on six platforms. Every time I switch platforms it takes a lot of time and energy to switch physical and mental contexts. I have no hope of thoroughly testing all the platforms for each little change I make. So I usually just compile the code on two or three platforms, test on one, and check it in. The incremental build servers will tell me if I broke the build within a few minutes, and the testers will tell me if I made some platform-specific breakage the next day. Furthermore, for each bug, the tester has narrowed it to specific repro steps, including what DOESN'T work, and usually has a screenshot of the problem and what's expected. Usually by reading the report, I immediately have a good idea of what's broken and that saves me tons of time too (as opposed to a vague "the xyz component is broken, fix plz" which I have seen in lots of shops where non-testers write bug reports). I love our testers.
run4yourlives 17y ago
That grind work can be done much cheaper (and probably more reliably) by a dedicated tester. I won't dispute that, as that is a major reason why companies love testers. The problem is that while the testing is done cheaper and perhaps better, that's not really the goal of the whole process now, is it? The goal of the process is to match the product to the customers expectation/need. I submit that the best possible way to do this is to have programmers that are also managing that need, even if it is more expensive and/or less efficient when view specifically.
- jister 17y ago
  I think the best way is to have a specification coming from the customer indicating their needs and expectation. Any feature that's not mentioned in the document cannot be considered a "bug". Based from this specification, testers/QA can not write their test cases.
  - nostrademons 17y ago
    You will never get that specification. Most people don't know what they need until you show it to them. Specification-driven development is a great way to write software that gets you paid, gets some purchasing manager a bonus, and yet pisses off every end user who comes in contact with it.

pixelbath 17y ago

To me, your experience sounds an awful lot like not having much experience in a production environment. In my experience, context-switching between programming and testing gets quite cumbersome with difficult problems, and can lead to overlooking smaller issues. If I'm busy testing every single edge case I can think of, who's doing the development? With separate testers, you get the benefit of concurrency (the software can continue being tested while issues from the previous version are fixed), and subsequently more software developed. "The problem is the cycle is too long. Fix that." How so? What if I'm developing a new product that has a full specification and design document, and is expected to meet customer acceptance goals? I don't see a deficiency in the development cycle if I need someone else to help test my software in a manner that I simply don't have time for. Does your assertion mean that any software that contains bugs is caused by lack of programmer skill? I, too, would like to learn the secret to writing completely issue-free code if you're willing to share it. Now, please don't take what I say to mean programmers shouldn't test their own creations. If you hand something that crashes 100% of the time to a tester, you're being lazy. The purpose of testers goes far beyond simply checking to see if it's "code that works."

run4yourlives 17y ago
To me, your experience sounds an awful lot like not having much experience in a production environment. Actually on the contrary, it comes from a growing understanding that the 'production environment' is in fact the problem! In my experience, context-switching between programming and testing gets quite cumbersome with difficult problems, and can lead to overlooking smaller issues. Whether or not a programmer or a designated tester is actually doing the work has no bearing on over-testing. In fact I'd argue that this is more likely to occur if you use a tester. * If I'm busy testing every single edge case I can think of, who's doing the development?* Nobody! That's the crux of the matter right there: 1. Testing sucks, so if someone else is doing it then it's better for you, and, more importantly 2. the pressure to continue to develop new features at speeds that are unsuistainable all but ensure that you'll eventually find yourself with so much technical debt that it will kill you. What if I'm developing a new product that has a full specification and design document, and is expected to meet customer acceptance goals? I'd say run away from this waterfall design process as fast as possible, to be frank. I don't see a deficiency in the development cycle if I need someone else to help test my software in a manner that I simply don't have time for. I do. I see a development cycle where you haven't been allocated enough time! Does your assertion mean that any software that contains bugs is caused by lack of programmer skill? No, because in order for a bug to be a bug it must not only exist in the code but also in the feature set. (i.e. it must be found) The fact of the matter is that nobody - not you, not your testers, and certainly not your customers knows exactly how the software will be used until it is actually being used.

RiderOfGiraffes 17y ago

With respect, I suggest your experience is limited. I deploy embedded safety-critical software into establishments such as nuclear naval bases. We need testers because while the code works (we use formal and semi-formal software derivation techniques) there are, on occasion, unexpected interactions between features. You need to run through combinations of configurations and actions, and some of it can't be scripted because the interactions are unexpected - the Black Swan principle. Our cycle is a long one in part because acquisition programs in government and quasi-governmental organisations are like that. We can't deploy changes every day into the field. For one thing, we usually have access. For another, any change needs associated training and manuals. It would be a waste of a talented programmer to run through the sorts of detailed and exhaustive testing our software has to pass before it will be accepted. Perhaps you simply think that my company is placed in a less than ideal market. I assure you, it's worth being in the game.

run4yourlives 17y ago
Perhaps you simply think that my company is placed in a less than ideal market. I assure you, it's worth being in the game. I do, only in the sense that in your market, your costs can justify your operations in a way that not many markets can match. Joel makes bug trackers, so I stayed in that context although I spoke in very black and white terms. I'm not intending to profess the ultimate solution to everyone's process, but I am trying to speak to a majority of start-up centric developers-cum-managers. Doing what you do... hell hire 25 testers for every component; just don't kill anyone.
- jister 17y ago
  "Doing what you do... hell hire 25 testers for every component; just don't kill anyone. I think I miss this one but how did you know he needed 25 testers for every component?
  - nostrademons 17y ago
    Hyperbole.

dasil003 17y ago

The second area is that you aren't getting the right feedback from customers. Yes, if you are bringing your customers into the office and actually watching them use your product. Usability testing is and always has been one of the most lacking areas of web development (probably other types of consumer software as well, but web is my area of expertise). A good tester can address these issues up front more than anyone else in the organization. Everyone else is too close to project to give it a fresh look. It's no substitute for real usability testing, but it's a good deal closer than anything else internal.

ericb 17y ago

> Either they don't sufficiently understand the problem scope, they're overworked, or they're just lazy and don't bother running through enough of their own quality controls before releasing changes. The number of possible system states in larger systems can be enormous. I feel like to have this opinion, perhaps you have worked mainly on smaller systems? In larger systems, there can be so many possibilities that it would take several times longer to test than to code, and without reading spec docs, it is very difficult to tell what correct even is. For example, a student loan system I worked on fit this bill. The glib answer would be to say it was so complex because of poor programming, but that's not true. The legal reality it was modeling was complex and so the software could not be less complex than the underlying system's rules. I have worked on several such systems in different industries. Now, you could argue that all programmers should become programming/QA people, but that would be self-defeating as programmer time is generally more expensive per-hour than QA time. Further, the type of person who is good at running through 400 steps of detail-oriented rote testing is not always the same person that you would like doing high-level design. Specialization makes life more efficient. Even if they did happen to be the same person, if they enjoyed one task more, why wouldn't they go to a job where they could focus on that type of task?

run4yourlives 17y ago
I feel like to have this opinion, perhaps you have worked mainly on smaller systems? No, I've got a good mix, and I agree with you that this is a product of large systems more than simple web apps and the like. The complexity of large systems being required is what's the problem though; not that I have an immediate solution. That and too many programmers that simply aren't able to do anything but program.
- derefr 17y ago
  You can't say that complexity itself is the problem, because the whole of programming is encoding complex processes so as to simplify them from a human perspective. Without complexity, it's not testing that would go away, it's programming. You've basically said something like "the problem with space shuttles is that they have to aim their thrust precisely in order to achieve lift"—that's not the problem with space shuttles, that's the idea of space shuttles.