5 ms·

Meh. I keep running into this thing.. I want to test a (legacy) system but it wasn't designed with testability in mind. Every time I look for advice, I run in

by clarry 7y ago

Meh. I keep running into this thing.. I want to test a (legacy) system but it wasn't designed with testability in mind. Every time I look for advice, I run into a stumbling block where the advice seems to apply to systems that were designed to be testable. In this case, right at step 1. Capturing the output and making sense of it while simultaneously allowing for normal variation e.g. due to timing is one heck of a big job and requires a lot of mocking. It's not a job that can be done in hours; probably weeks, if you want something that doesn't get thrown away the next time you have time to spend on testing.

kodablah 7y ago
With these systems, I've often found that black-box testing is what is required. This is at a systems level, even higher than this blog suggests. Yes, there are specific error scenarios and timing situations you may not cover with such tests. However, if you tackle the system holistically and make your tests from an external actor perspective, you can get a long way without concerning yourself with implementation details beyond the ones needed to deploy it and all dependencies in a repeatable environment. It does take time to build the harness to have reusable test pieces, but it's often worth it. As a bonus, you'll have a specification to meet as you try to replace the system (though the suite is rarely enough to give confidence on its own).
- clarry 7y ago
  Yes, that's where I'd like to start (and I sort of did, but the approach I took was nowhere near good enough). Like you say, it does take time to build the harness. Kind of a hard sell with our client.. I'd need to mock hardware I don't have, possibly mock some kernel interfaces, set up a physical test endpoint that can observe things on the wire (as in raw ethernet packets, but also application level protocols..). And well, really capturing timing issues and such is kinda important because things like flow control, multipath routing & failover are critical features of the software. Of course some of the things you want to test are inherently timing related issues. Like, a certain signal must be asserted if a particular message isn't received within a specified (possibly configurable or not) time window after some event. The core of the software is a heavily asynchronous & multithreaded mess.. Getting all the scaffolding in place and then automating everything is going to require lots and lots of time. And reverse engineering. I tried playing with bash scripts and docker based virtual deployments but it completely misses the hardware side of things and really doesn't get me far.
  - akavel 7y ago
    Hm, do you know of any tool/framework for capturing and then re-running (mocking) syscall responses? Could be interesting to keep in a toolbox for future...
  - pydry 7y ago
    I did this once on a large project I recently joined. It took a long time to build these sophisticated mocks (3-4 weeks) and it was a fairly hard sell but it did have the advantage that it's somewhat immune from mythical man month issues. That is, you can hire a new person, they can do all this work and it doesn't affect the speed that the rest of the team delivers at much. They don't need to know extensive amounts about the underlying core code - just the way it interfaces with the outside world. It's also a pretty effective way of onboarding somebody on to the project. That isn't the case for any kind of unit testing approach, where you have to be able to work with, understand and refactor the core code. That requires core dev team members and will take their attention away from customer facing tasks.
praptak 7y ago
"I want to test a (legacy) system but it wasn't designed with testability in mind." Ugly advice: fall back to manual "unit tests" with a debugger, temporary printf or whatever you have at hand. Does this eat a lot of time? Yes. Is it grungywork? Yes. Why do it then? Basically to surface the real cost of poor testability.
room271 7y ago
Small changes. If possible, pick a bit of logic and extract it into a (unit) testable function. Release it, see if anyone shouts. Repeat. Even then, it is often hard to justify this work/find the time. It depends on how long you think you'll be working with the codebase really. Edit: the above assumes you have something like CD/fast release cycle. If not, then it's not really viable unfortunately.
- Retric 7y ago
  That approach is almost guaranteed to introduce bugs into complex systems. It’s fine if you’re doing something directly useful at the same time, but just be aware of the risks. Though that dependent on what you mean by legacy. If nobody’s touched it in 5+ years it might very well be unchained when the system is completely replaced with something else. It’s well worth adding tests if something is or will be in active development.
- Davertron 7y ago
  > BUT you know that you should. If you don’t, you’re making things worse! If you don’t write the tests, you may even break something and you won’t realize!!! Slowly adding tests is fine if your goal is to increase the total code under test, but it's not going to give you much confidence that you didn't just break something when making updates to the system.
deleted 7y ago
[deleted]
commandlinefan 7y ago
What I run into, over and over again, is code that can't be executed in isolation. I mostly work in Java, and I keep ending up maintaining codebases where nearly everything is declared "static"... including the code that connects to databases and remote services and initializes data to start the service. I've seen this "pattern" across four different employers over two decades now.
dllthomas 7y ago
This is precisely the focus of the book "Working Effectively With Legacy Code", and my impression on reading it was that the advice seemed sensible (although I haven't tested any of it deliberately). Worth looking into if you haven't.

by clarry 7y ago

kodablah 7y ago

With these systems, I've often found that black-box testing is what is required. This is at a systems level, even higher than this blog suggests. Yes, there are specific error scenarios and timing situations you may not cover with such tests. However, if you tackle the system holistically and make your tests from an external actor perspective, you can get a long way without concerning yourself with implementation details beyond the ones needed to deploy it and all dependencies in a repeatable environment. It does take time to build the harness to have reusable test pieces, but it's often worth it. As a bonus, you'll have a specification to meet as you try to replace the system (though the suite is rarely enough to give confidence on its own).

clarry 7y ago
Yes, that's where I'd like to start (and I sort of did, but the approach I took was nowhere near good enough). Like you say, it does take time to build the harness. Kind of a hard sell with our client.. I'd need to mock hardware I don't have, possibly mock some kernel interfaces, set up a physical test endpoint that can observe things on the wire (as in raw ethernet packets, but also application level protocols..). And well, really capturing timing issues and such is kinda important because things like flow control, multipath routing & failover are critical features of the software. Of course some of the things you want to test are inherently timing related issues. Like, a certain signal must be asserted if a particular message isn't received within a specified (possibly configurable or not) time window after some event. The core of the software is a heavily asynchronous & multithreaded mess.. Getting all the scaffolding in place and then automating everything is going to require lots and lots of time. And reverse engineering. I tried playing with bash scripts and docker based virtual deployments but it completely misses the hardware side of things and really doesn't get me far.
- akavel 7y ago
  Hm, do you know of any tool/framework for capturing and then re-running (mocking) syscall responses? Could be interesting to keep in a toolbox for future...
- pydry 7y ago
  I did this once on a large project I recently joined. It took a long time to build these sophisticated mocks (3-4 weeks) and it was a fairly hard sell but it did have the advantage that it's somewhat immune from mythical man month issues. That is, you can hire a new person, they can do all this work and it doesn't affect the speed that the rest of the team delivers at much. They don't need to know extensive amounts about the underlying core code - just the way it interfaces with the outside world. It's also a pretty effective way of onboarding somebody on to the project. That isn't the case for any kind of unit testing approach, where you have to be able to work with, understand and refactor the core code. That requires core dev team members and will take their attention away from customer facing tasks.

praptak 7y ago

"I want to test a (legacy) system but it wasn't designed with testability in mind." Ugly advice: fall back to manual "unit tests" with a debugger, temporary printf or whatever you have at hand. Does this eat a lot of time? Yes. Is it grungywork? Yes. Why do it then? Basically to surface the real cost of poor testability.

room271 7y ago

Small changes. If possible, pick a bit of logic and extract it into a (unit) testable function. Release it, see if anyone shouts. Repeat. Even then, it is often hard to justify this work/find the time. It depends on how long you think you'll be working with the codebase really. Edit: the above assumes you have something like CD/fast release cycle. If not, then it's not really viable unfortunately.

Retric 7y ago
That approach is almost guaranteed to introduce bugs into complex systems. It’s fine if you’re doing something directly useful at the same time, but just be aware of the risks. Though that dependent on what you mean by legacy. If nobody’s touched it in 5+ years it might very well be unchained when the system is completely replaced with something else. It’s well worth adding tests if something is or will be in active development.
Davertron 7y ago
> BUT you know that you should. If you don’t, you’re making things worse! If you don’t write the tests, you may even break something and you won’t realize!!! Slowly adding tests is fine if your goal is to increase the total code under test, but it's not going to give you much confidence that you didn't just break something when making updates to the system.

deleted 7y ago

[deleted]

commandlinefan 7y ago

What I run into, over and over again, is code that can't be executed in isolation. I mostly work in Java, and I keep ending up maintaining codebases where nearly everything is declared "static"... including the code that connects to databases and remote services and initializes data to start the service. I've seen this "pattern" across four different employers over two decades now.

dllthomas 7y ago

This is precisely the focus of the book "Working Effectively With Legacy Code", and my impression on reading it was that the advice seemed sensible (although I haven't tested any of it deliberately). Worth looking into if you haven't.