5 ms·

I agree, it makes no sense, not only from a planning perspective from an implementation perspective, specifically... "In order to keep costs low on testing you

I agree, it makes no sense, not only from a planning perspective from an implementation perspective, specifically... "In order to keep costs low on testing you need to test more software, so you do a bigger release with more changes - economies of scale thinking." If you wanted to keep testing costs low, you wouldn't do bigger releases, you'd create automated tests. You may spend more effort up front on building the tests, but as long as you make the test components modular and target the tests at the right level of your application the 'cost' of testing will decrease over time.

AstralStorm 10y ago
Yeah, sure, everything can be tested in an automated way. I can name a few projects that tried it. They were released with a hilarious amount of bugs. Also writing good automated tests requires a great test developer. The thing is, anyone with such credentials would be a great developer and as such, not a tester. Even if you go fully test-driven, which makes it much cheaper, the cost of test a lot development model is surprisingly high for any application of useful size. Just imagine trying to write even something a simple as MS Paint with good test coverage.
- rco8786 10y ago
  I'm not sure what would be so special or difficult about writing unit tests for Ms Paint.
  - Retric 10y ago
    MS paint includes user selected faunts. That's an endless rabbit hole of complexity.
    - ZenoArrow 10y ago
      Validation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.
      - Retric 10y ago
        You're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.
        ZenoArrow 10y ago
        You target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.
        ktRolster 10y ago
        Users don't care if a bug is due to your code or the code in a library you depend on. It's up to you to make sure things work either way.
        ZenoArrow 10y ago
        If you find bugs in 3rd party components whilst testing, of course you try and do something about it. The point is, at some point you have to trust that the components you're working with are relatively bug-free otherwise every application will end up testing all the features of the OS it sits upon. Do you really expect developers to run a full test suite for Windows every time they develop a painting application? Note that I'm talking about actively looking for bugs, not the bugs that are apparent from some happy path testing.
    - beat 10y ago
      No, it's not. It's quite simple. There are only a few types of fonts available - TrueType, PostScript, bitmap, maybe others? At any rate, you test a couple of examples of each type. Paint is a bitmap graphic generator, right? Apply letters in a test font to a bitmap, and compare the resulting bitmap to an expected value. We don't need to test every possible font there is. We only need to test enough fonts to cover the likely failure cases and known bugs. Speaking of bugs, suppose we find a bug with a specific font, or a specific size of font, or something like that. We write a test that exercises the bug, then fix the bug. Because it's a one-off test, it's not a tremendous burden to write. Moreover, that bug will never come back undetected again.
- ZenoArrow 10y ago
  Writing good automated tests doesn't necessarily rely on a great developer, it all depends on how you approach testing. For example, let's say you want to use Selenium to write web UI tests. One common approach is to have developers create a page object model, which testers can then use to write readable and robust tests. Creating a page object model is a simple task, and working with that page object model is a simple task, as the model guidelines define an easy to implement and easy to follow control structure (essentially all page object methods must return another page object, which means you can chain them together). Oh and MS Paint would be easy to create a good test suite for. For what it's worth I'm a software tester by trade, so perhaps it's straightforward for someone who creates tests for a living to know how to approach it, I guess someone who was inexperienced wouldn't necessarily know how to approach it.
  - lovich 10y ago
    Do you have any suggestions on where to learn more about testing? I've gone through numerous tutorials and read through The Art of Unit Testing but I still feel like I'm writing tests just to write tests. It's not really clicking for me
    - ZenoArrow 10y ago
      Sure. First of all, even though you're clearly a developer (as you mentioned writing unit tests), I'd recommend starting at a higher level of abstraction with BDD tools, basically anything that implements the Gherkin language (I can't tell you which BDD tool to use as I don't know what language you're coding in, if you can tell me I can give you a more specific recommendation). The idea behind this is that you're writing your code to meet a specification that your client can also work with, so you can be sure the specification you're coding against is what the client wants. As an added bonus, the Gherkin scripts form a type of living documentation, providing clear information about what the application does whilst also remaining up to date (so long as all tests must pass before a new version is sent out). Second recommendation is to look into how to avoid test-induced damage, which is where code gets bloated and more complicated in order to make it more testable. One major source of problems in this area is the need to create mock objects. I'd recommend this video from Mark Seemann as a good starting point in this area, as it looks at how you can create unit tests without mocks: https://www.infoq.com/presentations/mock-fsharp-tdd https://www.infoq.com/presentations/mock-fsharp-tdd If you can let me know which language you're mostly coding in I'll see if I can give you some more recommendations.
- landryraccoon 10y ago
  Developers should be writing their own tests. If a developer thinks writing tests for his own code is a waste of time that's a red flag for me.
  - nradov 10y ago
    Developers should be doing a lot of things. In my experience the trouble is that most developers are too optimistic and tend to neglect negative test cases. So most teams still need specialist testers to ensure acceptable quality.
    - Guyag 10y ago
      Is this not a case for developer education? If developers are neglecting corner cases that's a real issue that needs to be addressed directly rather than having it be the job of someone else to clean up after them.
      - nradov 10y ago
        Fish or cut bait? Obviously I train developers on my teams to be better testers but that takes time, meanwhile we have to actually get working releases out. The testers work directly with the developers, they aren't cleaning up after them. (And please no one tell me that I should only hire developers who are already also expert testers. I have to operate in the real world.)
        ktRolster 10y ago
        Yeah, it's too hard to hire good programmers and too expensive as well. If you have the capability to teach programmers, then that's a real competitive advantage.
        nradov 10y ago
        It's not hard to hire good programmers. There are plenty of good programmers available at the market clearing rate. However there are never any perfect programmers available anywhere at any price. Everyone has some flaws and knowledge gaps. So managers have to keep their expectations realistic.
- tehlike 10y ago
  Edit: Some spacing, and grammar, and some sentence structure. +1. Putting too much faith on the test coverage is a sign of too much faith in humans. Some of my time goes to mobile development/android these days, and here are facts: 1. On android, there is occasionally this one vendor that crashes randomly for some piece of code. Because they modified the source code for android when they shouldn't. And it's just one model, among thousands of others. And it's different model for different code pieces. 2. Android API is not exactly one would assume. They return nulls for cases when they should return empty lists, and vice versa. They do this occasionally. So you don't really know when an api returns null vs empty list, and some apis (like get running processes) should never return empty, except that it does in some bizarre cases. So your code would need to test both empty list and null, which adds to your test scenarios. Aside from null/empty list, there are many other examples. 3. When you release, all your tests work fine, at least for what they test. There is this one client that uses things a bit differently than others. You also fail 10% of the time for that client because of how they use an API. Especially in Android, how they use an API does matter. A simple example is using Application Context vs Activity Context vs Custom Context implementations. You'd assume they should all work fine, because Context is an abstraction and Liskov substitution principle should apply, but it doesn't because it's leaky abstraction. Thus, somethings works with some context while others work with other contexts, and you don't really know which. I came to the conclusion that for most cases, testing is a must to verify correctness of known scenarios, and known regressions. However, for automated upgrades/updates, testing will not work. Not at all. Doing automated updates/upgrades, it should be a controlled release. it makes more sense to make the release A/B test way, if possible, where small population (say 1% of the users) get control/previous release, and another population of the same size but disjoint gets the experiment/new release. Then you compare your baseline metrics. Could be # of http connections, number of user messages sent, number of crashes, etc. Testing can only cover what humans can think of. Unfortunately, we don't know what we don't know. That's the main problem.
  - ZenoArrow 10y ago
    Even if you do automated testing you should still do exploratory testing, it's not an either/or situation. Automated testing allows you to cover the bulk of your system functionality, exploratory testing allows you to pick up on the unexpected quirks that crop up. Automated testing frees up the time of testers to do exploratory testing as they don't have to run through all tests manually.

AstralStorm 10y ago

Yeah, sure, everything can be tested in an automated way. I can name a few projects that tried it. They were released with a hilarious amount of bugs. Also writing good automated tests requires a great test developer. The thing is, anyone with such credentials would be a great developer and as such, not a tester. Even if you go fully test-driven, which makes it much cheaper, the cost of test a lot development model is surprisingly high for any application of useful size. Just imagine trying to write even something a simple as MS Paint with good test coverage.

rco8786 10y ago
I'm not sure what would be so special or difficult about writing unit tests for Ms Paint.
- Retric 10y ago
  MS paint includes user selected faunts. That's an endless rabbit hole of complexity.
  - ZenoArrow 10y ago
    Validation of new fonts is handled by Windows rather than Paint, I don't see why a new font that meets the criteria set out for Windows would cause any massive test burden for Paint.
    - Retric 10y ago
      You're suggestion is to not test a massive chunk of functionality. That may be reasonable choice, but it's a long way from total test coverage.
      - ZenoArrow 10y ago
        You target the functionality that resides the application, and its integration with other software. However, you don't need to test Windows functionality, that's for Windows developers to do. 'Total test coverage' is about testing the application under test, not every component of the OS it builds upon.
        ktRolster 10y ago
        Users don't care if a bug is due to your code or the code in a library you depend on. It's up to you to make sure things work either way.
        ZenoArrow 10y ago
        If you find bugs in 3rd party components whilst testing, of course you try and do something about it. The point is, at some point you have to trust that the components you're working with are relatively bug-free otherwise every application will end up testing all the features of the OS it sits upon. Do you really expect developers to run a full test suite for Windows every time they develop a painting application? Note that I'm talking about actively looking for bugs, not the bugs that are apparent from some happy path testing.
  - beat 10y ago
    No, it's not. It's quite simple. There are only a few types of fonts available - TrueType, PostScript, bitmap, maybe others? At any rate, you test a couple of examples of each type. Paint is a bitmap graphic generator, right? Apply letters in a test font to a bitmap, and compare the resulting bitmap to an expected value. We don't need to test every possible font there is. We only need to test enough fonts to cover the likely failure cases and known bugs. Speaking of bugs, suppose we find a bug with a specific font, or a specific size of font, or something like that. We write a test that exercises the bug, then fix the bug. Because it's a one-off test, it's not a tremendous burden to write. Moreover, that bug will never come back undetected again.
ZenoArrow 10y ago
Writing good automated tests doesn't necessarily rely on a great developer, it all depends on how you approach testing. For example, let's say you want to use Selenium to write web UI tests. One common approach is to have developers create a page object model, which testers can then use to write readable and robust tests. Creating a page object model is a simple task, and working with that page object model is a simple task, as the model guidelines define an easy to implement and easy to follow control structure (essentially all page object methods must return another page object, which means you can chain them together). Oh and MS Paint would be easy to create a good test suite for. For what it's worth I'm a software tester by trade, so perhaps it's straightforward for someone who creates tests for a living to know how to approach it, I guess someone who was inexperienced wouldn't necessarily know how to approach it.
- lovich 10y ago
  Do you have any suggestions on where to learn more about testing? I've gone through numerous tutorials and read through The Art of Unit Testing but I still feel like I'm writing tests just to write tests. It's not really clicking for me
  - ZenoArrow 10y ago
    Sure. First of all, even though you're clearly a developer (as you mentioned writing unit tests), I'd recommend starting at a higher level of abstraction with BDD tools, basically anything that implements the Gherkin language (I can't tell you which BDD tool to use as I don't know what language you're coding in, if you can tell me I can give you a more specific recommendation). The idea behind this is that you're writing your code to meet a specification that your client can also work with, so you can be sure the specification you're coding against is what the client wants. As an added bonus, the Gherkin scripts form a type of living documentation, providing clear information about what the application does whilst also remaining up to date (so long as all tests must pass before a new version is sent out). Second recommendation is to look into how to avoid test-induced damage, which is where code gets bloated and more complicated in order to make it more testable. One major source of problems in this area is the need to create mock objects. I'd recommend this video from Mark Seemann as a good starting point in this area, as it looks at how you can create unit tests without mocks: https://www.infoq.com/presentations/mock-fsharp-tdd https://www.infoq.com/presentations/mock-fsharp-tdd If you can let me know which language you're mostly coding in I'll see if I can give you some more recommendations.
landryraccoon 10y ago
Developers should be writing their own tests. If a developer thinks writing tests for his own code is a waste of time that's a red flag for me.
- nradov 10y ago
  Developers should be doing a lot of things. In my experience the trouble is that most developers are too optimistic and tend to neglect negative test cases. So most teams still need specialist testers to ensure acceptable quality.
  - Guyag 10y ago
    Is this not a case for developer education? If developers are neglecting corner cases that's a real issue that needs to be addressed directly rather than having it be the job of someone else to clean up after them.
    - nradov 10y ago
      Fish or cut bait? Obviously I train developers on my teams to be better testers but that takes time, meanwhile we have to actually get working releases out. The testers work directly with the developers, they aren't cleaning up after them. (And please no one tell me that I should only hire developers who are already also expert testers. I have to operate in the real world.)
      - ktRolster 10y ago
        Yeah, it's too hard to hire good programmers and too expensive as well. If you have the capability to teach programmers, then that's a real competitive advantage.
        nradov 10y ago
        It's not hard to hire good programmers. There are plenty of good programmers available at the market clearing rate. However there are never any perfect programmers available anywhere at any price. Everyone has some flaws and knowledge gaps. So managers have to keep their expectations realistic.
tehlike 10y ago
Edit: Some spacing, and grammar, and some sentence structure. +1. Putting too much faith on the test coverage is a sign of too much faith in humans. Some of my time goes to mobile development/android these days, and here are facts: 1. On android, there is occasionally this one vendor that crashes randomly for some piece of code. Because they modified the source code for android when they shouldn't. And it's just one model, among thousands of others. And it's different model for different code pieces. 2. Android API is not exactly one would assume. They return nulls for cases when they should return empty lists, and vice versa. They do this occasionally. So you don't really know when an api returns null vs empty list, and some apis (like get running processes) should never return empty, except that it does in some bizarre cases. So your code would need to test both empty list and null, which adds to your test scenarios. Aside from null/empty list, there are many other examples. 3. When you release, all your tests work fine, at least for what they test. There is this one client that uses things a bit differently than others. You also fail 10% of the time for that client because of how they use an API. Especially in Android, how they use an API does matter. A simple example is using Application Context vs Activity Context vs Custom Context implementations. You'd assume they should all work fine, because Context is an abstraction and Liskov substitution principle should apply, but it doesn't because it's leaky abstraction. Thus, somethings works with some context while others work with other contexts, and you don't really know which. I came to the conclusion that for most cases, testing is a must to verify correctness of known scenarios, and known regressions. However, for automated upgrades/updates, testing will not work. Not at all. Doing automated updates/upgrades, it should be a controlled release. it makes more sense to make the release A/B test way, if possible, where small population (say 1% of the users) get control/previous release, and another population of the same size but disjoint gets the experiment/new release. Then you compare your baseline metrics. Could be # of http connections, number of user messages sent, number of crashes, etc. Testing can only cover what humans can think of. Unfortunately, we don't know what we don't know. That's the main problem.
- ZenoArrow 10y ago
  Even if you do automated testing you should still do exploratory testing, it's not an either/or situation. Automated testing allows you to cover the bulk of your system functionality, exploratory testing allows you to pick up on the unexpected quirks that crop up. Automated testing frees up the time of testers to do exploratory testing as they don't have to run through all tests manually.