6 ms·

Completely agree. People tend to dismiss testing rather than balance the depth of testing that they do. 100% code coverage doesn't mean you've tested every co

Completely agree. People tend to dismiss testing rather than balance the depth of testing that they do. 100% code coverage doesn't mean you've tested every conceivable combination of parameters to a method. One of the biggest benefits of testing in my mind is improving the design of code. If you have code that is very difficult to test, there is likely something wrong in your design. Any way you cut it, you need to become knowledgeable about testing to be able to apply it effectively.

spc476 15y ago
At work, for one small subsection of the project I'm on, I wrote the regression test. I'm testing a "program" that consists of 56 processes (what I'm testing) across three machines (and requires around four other processes across two machines to stub out some services we require but aren't technically part of what I'm testing). It can take up to half an hour to set up (one of the reasons it's not fully automated is that the third party network stack we rely upon will shut down if there are too many errors) and it takes around four hours to run (except for two test cases that require manual intervention to run properly). And that's just for the back-end processing (nearly 300k lines of C/C++ code). Unit testing? Okay, for large values of "unit", and most of the "units" being tested require almost as much set up as the entire "program". Is something wrong with the design? Given the constraints and how the project evolved, I can't see it being any simpler. And I'm somewhat overwhelmed with the thought of testing the frontend (which requires Android phones).
- wpietri 15y ago
  My metric here is always finding the most valuable way to use my time long term. In the short term, test automation always seems wasteful. But in the long term, it's great. Solid product, little debugging, and minimal manual QA. You're in a situation with a lot of legacy code. Testing shapes design, but it sounds like you're trying to retrofit testability onto an existing mess. People cut corners for years, and now it's your problem. That sucks. In your shoes I'd either start improving it or find a new job. I think life's too short to spend my time doing something a computer could and should be doing.
  - spc476 15y ago
    Heh ... it's actually a new project. Yes, the majority of the code is third party software. And while we do have a bit of "legacy code" in the project (in the form of the third party proprietary network stack that literally is in pure maintenance mode) we're mostly working in a legacy system (telephony network) that requires very high degrees of redundancy (hence the number of processes on the number of machines). And for the most part, I was able to get the regression test for the backend process(es) to run unattended once started (thankfully---I (along with two others) did it once manually and it was horrible). I have no idea of how to do that for the frontend Android cell phone client. Sure, we can run tests on an emulator, but there are issues with the Android emulator (it exhibits different buggy behavior than the the physical hardware) so that only gets you so far. It's an interesting (if somewhat overwhelming) problem.
    - jonstjohn 15y ago
      Would you consider this testing 'unit testing', though? Sounds more like higher-level integration tests with a lot of dependencies. Honest question.
      - spc476 15y ago
        I'm not sure. Yes, we can (to some degree) independently test parts, but like I said, each part requires a significant portion of the environment to be up (or simulated). And "unit" testing (that is, testing an individual routine or module) doesn't really make sense given how the code is written (receive a message via SS7---the network stack I mentioned) and convert it to an IP based message). To test the portion that talks to the telephony network requires a telephony network (very hard to mock out---lord knows I would love to) and another major unit we wrote (which is another part I test) to even be testable. And to test that other part? Well, it requires I mock out the previous unit (or run it), plus three three other parts (one including a cell phone---which is really a simple script at this point). And again, it doesn't really make sense to test individual routines because this takes the translated IP packets from the SS7 module, and makes several queries to other IP based services. So a lot of what's going on is just simple translations (in a multithreaded/multiprocessor environment---more fun!).

spc476 15y ago

At work, for one small subsection of the project I'm on, I wrote the regression test. I'm testing a "program" that consists of 56 processes (what I'm testing) across three machines (and requires around four other processes across two machines to stub out some services we require but aren't technically part of what I'm testing). It can take up to half an hour to set up (one of the reasons it's not fully automated is that the third party network stack we rely upon will shut down if there are too many errors) and it takes around four hours to run (except for two test cases that require manual intervention to run properly). And that's just for the back-end processing (nearly 300k lines of C/C++ code). Unit testing? Okay, for large values of "unit", and most of the "units" being tested require almost as much set up as the entire "program". Is something wrong with the design? Given the constraints and how the project evolved, I can't see it being any simpler. And I'm somewhat overwhelmed with the thought of testing the frontend (which requires Android phones).

wpietri 15y ago
My metric here is always finding the most valuable way to use my time long term. In the short term, test automation always seems wasteful. But in the long term, it's great. Solid product, little debugging, and minimal manual QA. You're in a situation with a lot of legacy code. Testing shapes design, but it sounds like you're trying to retrofit testability onto an existing mess. People cut corners for years, and now it's your problem. That sucks. In your shoes I'd either start improving it or find a new job. I think life's too short to spend my time doing something a computer could and should be doing.
- spc476 15y ago
  Heh ... it's actually a new project. Yes, the majority of the code is third party software. And while we do have a bit of "legacy code" in the project (in the form of the third party proprietary network stack that literally is in pure maintenance mode) we're mostly working in a legacy system (telephony network) that requires very high degrees of redundancy (hence the number of processes on the number of machines). And for the most part, I was able to get the regression test for the backend process(es) to run unattended once started (thankfully---I (along with two others) did it once manually and it was horrible). I have no idea of how to do that for the frontend Android cell phone client. Sure, we can run tests on an emulator, but there are issues with the Android emulator (it exhibits different buggy behavior than the the physical hardware) so that only gets you so far. It's an interesting (if somewhat overwhelming) problem.
  - jonstjohn 15y ago
    Would you consider this testing 'unit testing', though? Sounds more like higher-level integration tests with a lot of dependencies. Honest question.
    - spc476 15y ago
      I'm not sure. Yes, we can (to some degree) independently test parts, but like I said, each part requires a significant portion of the environment to be up (or simulated). And "unit" testing (that is, testing an individual routine or module) doesn't really make sense given how the code is written (receive a message via SS7---the network stack I mentioned) and convert it to an IP based message). To test the portion that talks to the telephony network requires a telephony network (very hard to mock out---lord knows I would love to) and another major unit we wrote (which is another part I test) to even be testable. And to test that other part? Well, it requires I mock out the previous unit (or run it), plus three three other parts (one including a cell phone---which is really a simple script at this point). And again, it doesn't really make sense to test individual routines because this takes the translated IP packets from the SS7 module, and makes several queries to other IP based services. So a lot of what's going on is just simple translations (in a multithreaded/multiprocessor environment---more fun!).