4 ms·

> If you have a test suite that you trust so much that you are willing to deploy the system based solely on those tests passing; and if that test suite can be e

> If you have a test suite that you trust so much that you are willing to deploy the system based solely on those tests passing; and if that test suite can be executed in seconds, or minutes, then you can quickly and easily clean the code without fear. My spider sense always starts tingling when someone says something like this. Blind trust in an automated test-suite is a nice thing to strive for. But even for projects in which I have every confidence that the test suite is complete enough to actually trust it I'm still hestitant to "insta-deploy-on-pass". Call me cautious, call me chicken, whatever... I'll not drink this flavour of kool-aid

dagw 12y ago
Completely agree. Every time I read something like that I always ask myself if I'm really the only person in the world who's had a bug in his test suite.
- lmm 12y ago
  Maybe you're the only person in the world who's never made a mistake when manually testing new code? My test suite isn't perfect but it's more reliable than I would be. (if you're thinking "do both": if I thought the test suite wasn't reliable enough, I'd get a better return on time spent improving the test suite than I do on doing manual testing)
daleharvey 12y ago
Its one of those practices where the end goal may be unattainable, but the process of trying to achieve it has a huge benefit imo.
brudgers 12y ago
I'll call you uncomfortable with the idea that the critical element of test driven development is making the programmer's life easier rather than producing software user's can trust. The correlation may be the reason for using TDD, but the goal is software that doesn't harm users. Making the programmer's life easier is only a potential side effect.
cyberneticcook 12y ago
I think it's dangerous to rely solely on a unit test suite to guarantee the correctness of your program. Each unit-tested component may pass its own tests, but that doesn't guarantee that by interacting with other (present and future) components it won't exhibit any bug. I guess immutability would go a long way forward in realizing this guarantee, but I've never had the pleasure of working on a software where each component was truly immutable and independent of each other.
- jarrett 12y ago
  Very true. Here's an example of a bug that has bitten me more than once, has everything to do with immutability, and isn't reasonably within the scope of traditional testing tools: A crucial class writes files to a certain folder. The test sandbox provides such a folder, and the class knows to use that folder when it's running in test mode. Obviously, the production environment is also supposed to provide that folder. But oops, it doesn't! The tests exercised writing a file to disk, the tests were all green, and the system crashed anyway.
- MartinCron 12y ago
  I always progressively test at different levels before insta-deploy, and run a set of "is this still OK" tests against production post-deploy. Unit tests are at the heart of the test gauntlet, but absolutely can't be trusted by themselves.
- bulte-rs 12y ago
  And this is - IMO - where the magic happens. It's somewhat doable to "waterproof" your unit tests; but getting every (possible? Required? Anticopated?) interaction tested? Magic. I'm not saying tests are evil, I actually love my test suites. But as always; as soon as someone uses ALWAYS and/or NEVER, those tiny hairs in the back of my neck do a sort of wave and I try to ignore their message, which is a shame since (in this saga) both Uncle Bob and Nephew David have good advice to offer. As for immutability; add in referential transparency, static ish typing and you've got yourself a nice development setup ;)
MartinCron 12y ago
I have done two major projects with insta-deploy-on-pass and I would be hesitant to do it any other way. You still do manual and exploratory testing, you just automate as much of the "does thing X still work" that you don't have to manually test everything in order to ship anything.

dagw 12y ago

Completely agree. Every time I read something like that I always ask myself if I'm really the only person in the world who's had a bug in his test suite.

lmm 12y ago
Maybe you're the only person in the world who's never made a mistake when manually testing new code? My test suite isn't perfect but it's more reliable than I would be. (if you're thinking "do both": if I thought the test suite wasn't reliable enough, I'd get a better return on time spent improving the test suite than I do on doing manual testing)

daleharvey 12y ago

Its one of those practices where the end goal may be unattainable, but the process of trying to achieve it has a huge benefit imo.

brudgers 12y ago

I'll call you uncomfortable with the idea that the critical element of test driven development is making the programmer's life easier rather than producing software user's can trust. The correlation may be the reason for using TDD, but the goal is software that doesn't harm users. Making the programmer's life easier is only a potential side effect.

cyberneticcook 12y ago

I think it's dangerous to rely solely on a unit test suite to guarantee the correctness of your program. Each unit-tested component may pass its own tests, but that doesn't guarantee that by interacting with other (present and future) components it won't exhibit any bug. I guess immutability would go a long way forward in realizing this guarantee, but I've never had the pleasure of working on a software where each component was truly immutable and independent of each other.

jarrett 12y ago
Very true. Here's an example of a bug that has bitten me more than once, has everything to do with immutability, and isn't reasonably within the scope of traditional testing tools: A crucial class writes files to a certain folder. The test sandbox provides such a folder, and the class knows to use that folder when it's running in test mode. Obviously, the production environment is also supposed to provide that folder. But oops, it doesn't! The tests exercised writing a file to disk, the tests were all green, and the system crashed anyway.
MartinCron 12y ago
I always progressively test at different levels before insta-deploy, and run a set of "is this still OK" tests against production post-deploy. Unit tests are at the heart of the test gauntlet, but absolutely can't be trusted by themselves.
bulte-rs 12y ago
And this is - IMO - where the magic happens. It's somewhat doable to "waterproof" your unit tests; but getting every (possible? Required? Anticopated?) interaction tested? Magic. I'm not saying tests are evil, I actually love my test suites. But as always; as soon as someone uses ALWAYS and/or NEVER, those tiny hairs in the back of my neck do a sort of wave and I try to ignore their message, which is a shame since (in this saga) both Uncle Bob and Nephew David have good advice to offer. As for immutability; add in referential transparency, static ish typing and you've got yourself a nice development setup ;)

MartinCron 12y ago

I have done two major projects with insta-deploy-on-pass and I would be hesitant to do it any other way. You still do manual and exploratory testing, you just automate as much of the "does thing X still work" that you don't have to manually test everything in order to ship anything.