6 ms·

> I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what

> I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what needs to be tested, then there is an implication that some (I'd venture to say most) of what you write doesn't need to be tested. And that's a liberating concept. You aren't obligated to test everything, but you should test what really matters. In -my- anecdotal experience, this is the excuse developers use while they write bug-ridden code without tests. Maybe they save a little time at the early stage of the development cycle, but the real reason seems to be avoiding work they don't find to be as fun as hacking out code. After that code is hacked out, we then waste a bunch of cycles across multiple teams while we find and fix out the most obvious of their bugs. After the release goes out, we spend even more time finding previously undiscovered bugs and fixing them -- which hits actual user satisfaction. All because some lazy sod thought he was too smart to need tests, and that he could be trusted to decide when his code didn't require them. Fuck that. I've been doing this too long and seen this happen too many times. If you're on my team, you write the damn tests.

sturgill 14y ago
We have a slightly different approach to the same problem: if people keep putting out crap code that breaks (or worse, introduces regression bugs) and I will let them go. I have no time for people who do crap work. You use tests to weed those people out. That's fine, if it works for you. I'm anal about knowing everything that's happening in my code. I compare git commits to ticket numbers to keep abreast of how things are changing over time. A bug comes through on Goalee and I know which handful of files it will be in. Honestly, I'm quite uptight about this stuff. I catch most shortcuts personally before they go out, and have the appropriate conversation with the responsible party.
- pifflesnort 14y ago
  I've never seen untested code that wasn't crap, or didn't require just as much work (if not more) to ferret out all the bugs (including watching everyone like a hawk).
  - RyanZAG 14y ago
    A huge percentage of the code you use every day doesn't have tests - Linux kernel? No tests - TCP/IP stack? No tests - Windows Kernel (up till WinXP anyway) ? No tests - Android? No tests - Quake engine ? No tests Don't make stupidly broad statements like "I've never seen untested code that wasn't crap". Well unless by "untested" you mean "code that was never even ran or QA'd a little". I'm pretty sure you mean "code that didn't go through TDD" though, in which case - you are very, very wrong.
    - MaulingMonkey 14y ago
      Even if we make your assumption that he's specifically referring to TDD only, just because good code that didn't go through the more rigorous definitions of TDD exists doesn't mean he's seen it. And plenty of crap code ships. I'd argue Linux has significantly more than 'no' tests: http://stackoverflow.com/questions/3177338/how-is-linux-kernel-tested http://stackoverflow.com/questions/3177338/how-is-linux-kern... -- it's not TDD, and it's not centralized, but then again, neither is Linux development. And while you might convince me that pre-XP kernels went without unit testing, I'd point out we hardly call them bastions of stability and security. Try certifying a 360 or ModernUI title under the Microsoft banner and see if they'll let you get away without writing a single test. I'd wager anyone who successfully accomplishes this has spent far more effort arguing why it should be allowed than writing some tests would take. And, given the sheer girth of their security processes (patching, QA certification, tracking) and toolset (through virtualization, fuzzing, unit testing, CI build servers), it would take much more effort to convince me that they do absolutely no unit testing whatsoever on their kernel in this day and age. Far more than you could reasonably give: I suspect such convincing would require breaking several NDAs and committing several felonies.
    - pifflesnort 14y ago
      I'm curious as to whether you've actually worked on the Linux kernel, or the UNIX descendants, or TCP/IP stacks. I have. The only reason they work at all is the enormous number of man years spent banging on the poorly tested code after its written. The number of regressions that pop up in core subsystems of the kernel are staggering. VM bugs still happen in commercially shipping systems like OS X because of small changes to extremely fragile and untested code. Linux isn't a paragon of code quality, and neither are most Network stacks. If anything, you're proving my point -- code written in that methodology has required enourmous amounts of work to stabilize and prevent/fix regressions going forward. Case in point; a few years ago, I found a trivial signed/unsigned comparison bug but in FreeBSD device node cloning that was triggered by adding/removing devices in a certain order; this triggered a kernel panic, and took teo days to track down. The most trivial of unit tests would have caught this bug immediately, but instead it sat there languishing until someone wrote some network interface cloning code (me) that happened to trigger this bug faaar down below the section of the kernel I was working in. This kind of thing happens all the time in the projects you list, and its incredibly costly compared to using testing to not ship bugs in the first place.
gfodor 14y ago
Ok, so if you have people that are legitimately lazy, then they should be fired, full stop. However there is probably another reason they are not writing enough tests: - They don't see the benefit because they are forced to write lots of tests that do not, and never would have, provided value. (like stupid getter and setter tests being the canonical example) - Writing tests is hard or slow, either due to things not being tooled out or because the code has not been architected to be testable. - Writing tests causes lots of false positives, fails to catch real bugs, or end up being extremely brittle. Classic example here is religious mocking of the database in the name of performance, yet fails to catch most bugs since it's, you know, not actually hitting a database. - The suite is slow or causes deployments to slow down such that adding tests adds a "tax" to everything else people do. I've had enough experiences with all of the above phenomena so as to become completely skeptical of extensive testing since unless you are a real pro you will inevitably wane on at least one of these things.

sturgill 14y ago

We have a slightly different approach to the same problem: if people keep putting out crap code that breaks (or worse, introduces regression bugs) and I will let them go. I have no time for people who do crap work. You use tests to weed those people out. That's fine, if it works for you. I'm anal about knowing everything that's happening in my code. I compare git commits to ticket numbers to keep abreast of how things are changing over time. A bug comes through on Goalee and I know which handful of files it will be in. Honestly, I'm quite uptight about this stuff. I catch most shortcuts personally before they go out, and have the appropriate conversation with the responsible party.

pifflesnort 14y ago
I've never seen untested code that wasn't crap, or didn't require just as much work (if not more) to ferret out all the bugs (including watching everyone like a hawk).
- RyanZAG 14y ago
  A huge percentage of the code you use every day doesn't have tests - Linux kernel? No tests - TCP/IP stack? No tests - Windows Kernel (up till WinXP anyway) ? No tests - Android? No tests - Quake engine ? No tests Don't make stupidly broad statements like "I've never seen untested code that wasn't crap". Well unless by "untested" you mean "code that was never even ran or QA'd a little". I'm pretty sure you mean "code that didn't go through TDD" though, in which case - you are very, very wrong.
  - MaulingMonkey 14y ago
    Even if we make your assumption that he's specifically referring to TDD only, just because good code that didn't go through the more rigorous definitions of TDD exists doesn't mean he's seen it. And plenty of crap code ships. I'd argue Linux has significantly more than 'no' tests: http://stackoverflow.com/questions/3177338/how-is-linux-kernel-tested http://stackoverflow.com/questions/3177338/how-is-linux-kern... -- it's not TDD, and it's not centralized, but then again, neither is Linux development. And while you might convince me that pre-XP kernels went without unit testing, I'd point out we hardly call them bastions of stability and security. Try certifying a 360 or ModernUI title under the Microsoft banner and see if they'll let you get away without writing a single test. I'd wager anyone who successfully accomplishes this has spent far more effort arguing why it should be allowed than writing some tests would take. And, given the sheer girth of their security processes (patching, QA certification, tracking) and toolset (through virtualization, fuzzing, unit testing, CI build servers), it would take much more effort to convince me that they do absolutely no unit testing whatsoever on their kernel in this day and age. Far more than you could reasonably give: I suspect such convincing would require breaking several NDAs and committing several felonies.
  - pifflesnort 14y ago
    I'm curious as to whether you've actually worked on the Linux kernel, or the UNIX descendants, or TCP/IP stacks. I have. The only reason they work at all is the enormous number of man years spent banging on the poorly tested code after its written. The number of regressions that pop up in core subsystems of the kernel are staggering. VM bugs still happen in commercially shipping systems like OS X because of small changes to extremely fragile and untested code. Linux isn't a paragon of code quality, and neither are most Network stacks. If anything, you're proving my point -- code written in that methodology has required enourmous amounts of work to stabilize and prevent/fix regressions going forward. Case in point; a few years ago, I found a trivial signed/unsigned comparison bug but in FreeBSD device node cloning that was triggered by adding/removing devices in a certain order; this triggered a kernel panic, and took teo days to track down. The most trivial of unit tests would have caught this bug immediately, but instead it sat there languishing until someone wrote some network interface cloning code (me) that happened to trigger this bug faaar down below the section of the kernel I was working in. This kind of thing happens all the time in the projects you list, and its incredibly costly compared to using testing to not ship bugs in the first place.

gfodor 14y ago

Ok, so if you have people that are legitimately lazy, then they should be fired, full stop. However there is probably another reason they are not writing enough tests: - They don't see the benefit because they are forced to write lots of tests that do not, and never would have, provided value. (like stupid getter and setter tests being the canonical example) - Writing tests is hard or slow, either due to things not being tooled out or because the code has not been architected to be testable. - Writing tests causes lots of false positives, fails to catch real bugs, or end up being extremely brittle. Classic example here is religious mocking of the database in the name of performance, yet fails to catch most bugs since it's, you know, not actually hitting a database. - The suite is slow or causes deployments to slow down such that adding tests adds a "tax" to everything else people do. I've had enough experiences with all of the above phenomena so as to become completely skeptical of extensive testing since unless you are a real pro you will inevitably wane on at least one of these things.