2 ms·

Heh ... it's actually a new project. Yes, the majority of the code is third party software. And while we do have a bit of "legacy code" in the project (in the

by spc476 15y ago

Heh ... it's actually a new project. Yes, the majority of the code is third party software. And while we do have a bit of "legacy code" in the project (in the form of the third party proprietary network stack that literally is in pure maintenance mode) we're mostly working in a legacy system (telephony network) that requires very high degrees of redundancy (hence the number of processes on the number of machines). And for the most part, I was able to get the regression test for the backend process(es) to run unattended once started (thankfully---I (along with two others) did it once manually and it was horrible). I have no idea of how to do that for the frontend Android cell phone client. Sure, we can run tests on an emulator, but there are issues with the Android emulator (it exhibits different buggy behavior than the the physical hardware) so that only gets you so far. It's an interesting (if somewhat overwhelming) problem.

jonstjohn 15y ago
Would you consider this testing 'unit testing', though? Sounds more like higher-level integration tests with a lot of dependencies. Honest question.
- spc476 15y ago
  I'm not sure. Yes, we can (to some degree) independently test parts, but like I said, each part requires a significant portion of the environment to be up (or simulated). And "unit" testing (that is, testing an individual routine or module) doesn't really make sense given how the code is written (receive a message via SS7---the network stack I mentioned) and convert it to an IP based message). To test the portion that talks to the telephony network requires a telephony network (very hard to mock out---lord knows I would love to) and another major unit we wrote (which is another part I test) to even be testable. And to test that other part? Well, it requires I mock out the previous unit (or run it), plus three three other parts (one including a cell phone---which is really a simple script at this point). And again, it doesn't really make sense to test individual routines because this takes the translated IP packets from the SS7 module, and makes several queries to other IP based services. So a lot of what's going on is just simple translations (in a multithreaded/multiprocessor environment---more fun!).
  - yxhuvud 15y ago
    Heh, I wonder if we have the same SS7-stack. Your description sounds disturbingly similar to our experience. Do your stack also have a wait of several minutes before reporting that starting it up went ok? (to the logfile of course. It is to good to actually report that to the console or to have service scripts that can be trusted). That wait is very popular with our testers. Oh well, at least in our case our signals originate in IP and we only have to check against a HLR, which do have a semidecent mockup.
    - spc476 15y ago
      From what I understand, there are only two commercially available SS7 stacks, and the one we use is the better of the two (which I find a frightening thought). So there's a 50/50 chance. I don't know enough of the stack to start it (or restart it) so I can't say for sure if that's how our stack works. And in our case, our signals originate in SS7 ...
  - jes5199 15y ago
    I went to an Agile class where the lecturer compared unit tests to double-entry bookkeeping. An accountant doesn't say "oh, I don't need to add up both columns here, I know it's just trivial addition". Once I got in the habit of writing tests of even the most simple transformations, the code complexity and my test complexity grew at the same rate, so it's much harder to end up with a giant untestable mass.
    - spc476 15y ago
      I once spent over a month tracking down a bug (in a different project than the one I mentioned above) that I have a hard time seeing how unit testing would have caught. The program: a simple process (no threads, no multiprocessing) that would, depending on which system it ran, would crash with a seg fault. The resulting core files were useless as each crash was in a different location. It turned out I was calling a non-re-entrant function (indirectly) in a signal handler (so technically it was multithreaded) and the crash really depended on one function being interrupted at just the right location by the right signal. That's why it took a month of staring at the code before I found the issue. Individually, every function worked exactly as designed. Together, they all worked except in one odd-ball edge case that varied from system to system (on my development system, the program could run for days before a crash; on the production system it would crash after a few hours). The fix was straightforward once the bug was found, but finding it was a pain. So please, I would love to know how unit tests would have helped find that bug. Yes, it is possible to write code to hopefully trigger that situation (run the process---run another process that continuously sends signals the program handles) but how long do I run the test for? How do I know it passed?
wpietri 15y ago
That sounds painful. And it's a regrettably common experience working with older code, code not written with testability in mind. Regarding Android, Netflix has a nice article about what they do to test everything: http://techblog.netflix.com/2012/03/testing-netflix-on-android.html http://techblog.netflix.com/2012/03/testing-netflix-on-andro... Personally, most of my testing code is unit tests. I try to isolate my code from third-party services in a variety of ways. I do have some integration tests that check that it all works together, but those are never as useful or as maintainable as I'd like.

by spc476 15y ago

jonstjohn 15y ago

Would you consider this testing 'unit testing', though? Sounds more like higher-level integration tests with a lot of dependencies. Honest question.

spc476 15y ago
I'm not sure. Yes, we can (to some degree) independently test parts, but like I said, each part requires a significant portion of the environment to be up (or simulated). And "unit" testing (that is, testing an individual routine or module) doesn't really make sense given how the code is written (receive a message via SS7---the network stack I mentioned) and convert it to an IP based message). To test the portion that talks to the telephony network requires a telephony network (very hard to mock out---lord knows I would love to) and another major unit we wrote (which is another part I test) to even be testable. And to test that other part? Well, it requires I mock out the previous unit (or run it), plus three three other parts (one including a cell phone---which is really a simple script at this point). And again, it doesn't really make sense to test individual routines because this takes the translated IP packets from the SS7 module, and makes several queries to other IP based services. So a lot of what's going on is just simple translations (in a multithreaded/multiprocessor environment---more fun!).
- yxhuvud 15y ago
  Heh, I wonder if we have the same SS7-stack. Your description sounds disturbingly similar to our experience. Do your stack also have a wait of several minutes before reporting that starting it up went ok? (to the logfile of course. It is to good to actually report that to the console or to have service scripts that can be trusted). That wait is very popular with our testers. Oh well, at least in our case our signals originate in IP and we only have to check against a HLR, which do have a semidecent mockup.
  - spc476 15y ago
    From what I understand, there are only two commercially available SS7 stacks, and the one we use is the better of the two (which I find a frightening thought). So there's a 50/50 chance. I don't know enough of the stack to start it (or restart it) so I can't say for sure if that's how our stack works. And in our case, our signals originate in SS7 ...
- jes5199 15y ago
  I went to an Agile class where the lecturer compared unit tests to double-entry bookkeeping. An accountant doesn't say "oh, I don't need to add up both columns here, I know it's just trivial addition". Once I got in the habit of writing tests of even the most simple transformations, the code complexity and my test complexity grew at the same rate, so it's much harder to end up with a giant untestable mass.
  - spc476 15y ago
    I once spent over a month tracking down a bug (in a different project than the one I mentioned above) that I have a hard time seeing how unit testing would have caught. The program: a simple process (no threads, no multiprocessing) that would, depending on which system it ran, would crash with a seg fault. The resulting core files were useless as each crash was in a different location. It turned out I was calling a non-re-entrant function (indirectly) in a signal handler (so technically it was multithreaded) and the crash really depended on one function being interrupted at just the right location by the right signal. That's why it took a month of staring at the code before I found the issue. Individually, every function worked exactly as designed. Together, they all worked except in one odd-ball edge case that varied from system to system (on my development system, the program could run for days before a crash; on the production system it would crash after a few hours). The fix was straightforward once the bug was found, but finding it was a pain. So please, I would love to know how unit tests would have helped find that bug. Yes, it is possible to write code to hopefully trigger that situation (run the process---run another process that continuously sends signals the program handles) but how long do I run the test for? How do I know it passed?

wpietri 15y ago

That sounds painful. And it's a regrettably common experience working with older code, code not written with testability in mind. Regarding Android, Netflix has a nice article about what they do to test everything: http://techblog.netflix.com/2012/03/testing-netflix-on-android.html http://techblog.netflix.com/2012/03/testing-netflix-on-andro... Personally, most of my testing code is unit tests. I try to isolate my code from third-party services in a variety of ways. I do have some integration tests that check that it all works together, but those are never as useful or as maintainable as I'd like.