13 ms·

So I do a lot of systems programming type development. In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range betw

So I do a lot of systems programming type development. In these cases, whether you're PXE booting a server or talking to 500 different APIs that all range between the latest database or cloud API, flashing firmware on a RAID controller (shudder), whatever, there's a point of diminishing returns. There's also most of the bugs that you won't be able to simulate, because they come from external factors. The magic lies in automated integration tests. QA automation is AMAZING stuff. Set up real servers and real test environments when you can. Test real database upgrades. Mock testing can have it's place, but often you find yourself making internal API contracts more rigid than you want, which can limit refactoring.

humanrebar 12y ago
How do you test your system's response to error conditions? Do you have hooks in your integrated system to force errors (throw exceptions, violate contracts, etc.)?
- mpdehaan2 12y ago
  In some cases, yes. You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters. The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem. Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out. Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing. And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.
pjc50 12y ago
I too do systemsish programming in C++. All the places I've ever done this have "unit" tests that link in almost all of the executable and inject test data through the normal frontend. There's very little mocking; sometimes there will be a setFooForUnitTests() function to short-circuit some normal configuration path.
goblin89 12y ago
Another magical thing (that I'm yet to try myself) is automated testing against live system. This ensures that the system actually works in production, not just on staging or test environment set up specifically for QA. I was initially against it, but reading about it recently[0] made me reconsider my opinion for some reason. [0] Meditations by Sean Cassidy, http://blog.seancassidy.me/meditations.html http://blog.seancassidy.me/meditations.html

humanrebar 12y ago

How do you test your system's response to error conditions? Do you have hooks in your integrated system to force errors (throw exceptions, violate contracts, etc.)?

mpdehaan2 12y ago
In some cases, yes. You can do things like set up scenarios you know won't work, such as connecting to machines that don't exist or sending the wrong parameters. The main thing is if the external inputs can't be made to "violate a contract", the internals aren't the problem. Ultimately, it depends on what you are building. Simple CRUD web applications are different from systems development type applications in many cases. Many of these classes can be more easily mocked out. Often the breaking change you'll see is an upstream API or SaaS endpoint changing in ways you didn't expect (or the difference between a Linux utility on the latest Ubuntu, an older RHEL, and Solaris), so it's often better to concentrate on testing the real thing. And these tests are expensive, so an automated test matrix that deploys a VM fleet with lots of permutations is usually necessary.

pjc50 12y ago

I too do systemsish programming in C++. All the places I've ever done this have "unit" tests that link in almost all of the executable and inject test data through the normal frontend. There's very little mocking; sometimes there will be a setFooForUnitTests() function to short-circuit some normal configuration path.

goblin89 12y ago

Another magical thing (that I'm yet to try myself) is automated testing against live system. This ensures that the system actually works in production, not just on staging or test environment set up specifically for QA. I was initially against it, but reading about it recently[0] made me reconsider my opinion for some reason. [0] Meditations by Sean Cassidy, http://blog.seancassidy.me/meditations.html http://blog.seancassidy.me/meditations.html