8 ms·

tl;dr: Tests are overhyped because we once wrote tests that became irrelevant when the thing we were testing changed Really? I mean, tests might be overhyped,

by azov 14y ago

tl;dr: Tests are overhyped because we once wrote tests that became irrelevant when the thing we were testing changed Really? I mean, tests might be overhyped, but so far it looks like the author draws incredibly general conclusions from some isolated incident. Why did the tests become irrelevant? Is their scoring algorithm now doing something completely different? Do they now have different use cases? Were they hard-coding scores when the actual thing that matters was, say, the ordering of the things they score? > Only test what needs to be tested well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested?

jaredsohn 14y ago
>well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested? I am not the OP, but I think unit tests are more useful for code that has a lot of edge cases (such as string parsing) and for code which causes the most bugs (as seen in black box testing or integration tests.) Also, some code is just easier to develop if you create a harness that runs it directly instead of having to work your way to the point in the program where it would execute it. If you do this, you might as well turn it into a test.
jamesaguilar 14y ago
I've heard this saying: "Test until fear turns to boredom." Then you need to make sure you get bored at a properly calibrated rate (i.e. not too soon given your business' criticality, not too late given your need to move quickly). For example, if a lot of expensive bugs during final cause analysis are found to be preventable via unit tests, then your calibration is probably tuned to being bored too quickly.
- mmcnickle 14y ago
  That's an excellent saying. It's exactly the line I try to tread with testing. I don't know what to do with the angst of getting the tradeoff wrong though.
- gfodor 14y ago
  This is a great line. I think it goes deeper though, in that after doing this for a while, if it hurts, you are probably doing it wrong. Writing code with too little tests hurts, because you are scared shitless. Writing code with too many tests hurts, because you are performing a religious rite not actually building something useful. The trouble is that you have to do this for a while to get to the point where you can both feel and recognize pain, or the lack thereof.
sturgill 14y ago
OP here. My conclusion wasn't drawn from an isolated incident; I shared the incident to highlight what I imagine is a common pitfall in testing. I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what needs to be tested, then there is an implication that some (I'd venture to say most) of what you write doesn't need to be tested. And that's a liberating concept. You aren't obligated to test everything, but you should test what really matters. What needs to be tested is probably directly related to the size and stability of the company (assuming size and stability have a positive correlation). I would venture to surmise that young start-ups have almost no need to test anything. That comment isn't meant to be inflammatory, but I look at testing too early like optimizing too early. There's no reason to shard a database until you absolutely have to, and I don't think you need to test something until you know it's mission critical. For example, Stripe offers a service that were components of it to fail would jeopardize their entire business. Their code base is obligated, by its nature, to have more stringent tests. But the startup in the garage next to yours who is still trying to determine MVP and will probably end up pivoting five times in the next two weeks? Save yourself a lot of grief and don't worry about the tests. Once you know what you are going to be, once you know what you can't afford to lose, well, wrap those portions up with some solid tests.
- pifflesnort 14y ago
  > I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what needs to be tested, then there is an implication that some (I'd venture to say most) of what you write doesn't need to be tested. And that's a liberating concept. You aren't obligated to test everything, but you should test what really matters. In -my- anecdotal experience, this is the excuse developers use while they write bug-ridden code without tests. Maybe they save a little time at the early stage of the development cycle, but the real reason seems to be avoiding work they don't find to be as fun as hacking out code. After that code is hacked out, we then waste a bunch of cycles across multiple teams while we find and fix out the most obvious of their bugs. After the release goes out, we spend even more time finding previously undiscovered bugs and fixing them -- which hits actual user satisfaction. All because some lazy sod thought he was too smart to need tests, and that he could be trusted to decide when his code didn't require them. Fuck that. I've been doing this too long and seen this happen too many times. If you're on my team, you write the damn tests.
wildwood 14y ago
1) Test your APIs. A public API, especially one that is key to your business, should be near 100% coverage. And attacked to look for security/usability/load problems. 2) Test enough during development to support later regression tests, and to make sure that the design is testable. This can usually be achieved with less than 20% coverage. But if you write production code that's so screwy it can't be regression tested, then you've got big problems. 3) Test any parts that scare you or confuse you or make you nervous. Use "test until you're more bored than scared" here.
- endymi0n 14y ago
  Terse, but extremely helpful and well formulated. I'll print these out for further reference ;-) I'd add 4) Continuous integration testing doubling as operations monitoring. I know that this doesn't "really" qualify as testing, but from my experience, it's far easier getting bitten by components not working together than from a screwed component alone. The key here is monitoring and testing business parameters from end to end. It doesn't help you finding out where the bug is, but it lets you know early on if there is one.
dspeyer 14y ago
Principles I've found: * A test should accept any correct behaviour from the tested code. Anything which is not in the requirements, should not be enforced by the test. * A test should not use the same logic as the code to find the "right" answer. * A function whose semantics are likely to be changed in the next refactor should not be invoked from a test. * Whenever a test fails, make a note of whether you got it to pass by changing the test or by changing the code. If it's the test more than 2/3 of the time, it's a bad test. * If you can't write a good test, don't write a test at all. See if you can write a good test for the code that calls this code instead.

by azov 14y ago

jaredsohn 14y ago

>well, thanks for the helpful advice :) Care to share what, in your opinion, needs to be tested? I am not the OP, but I think unit tests are more useful for code that has a lot of edge cases (such as string parsing) and for code which causes the most bugs (as seen in black box testing or integration tests.) Also, some code is just easier to develop if you create a harness that runs it directly instead of having to work your way to the point in the program where it would execute it. If you do this, you might as well turn it into a test.

jamesaguilar 14y ago

I've heard this saying: "Test until fear turns to boredom." Then you need to make sure you get bored at a properly calibrated rate (i.e. not too soon given your business' criticality, not too late given your need to move quickly). For example, if a lot of expensive bugs during final cause analysis are found to be preventable via unit tests, then your calibration is probably tuned to being bored too quickly.

mmcnickle 14y ago
That's an excellent saying. It's exactly the line I try to tread with testing. I don't know what to do with the angst of getting the tradeoff wrong though.
gfodor 14y ago
This is a great line. I think it goes deeper though, in that after doing this for a while, if it hurts, you are probably doing it wrong. Writing code with too little tests hurts, because you are scared shitless. Writing code with too many tests hurts, because you are performing a religious rite not actually building something useful. The trouble is that you have to do this for a while to get to the point where you can both feel and recognize pain, or the lack thereof.

sturgill 14y ago

OP here. My conclusion wasn't drawn from an isolated incident; I shared the incident to highlight what I imagine is a common pitfall in testing. I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what needs to be tested, then there is an implication that some (I'd venture to say most) of what you write doesn't need to be tested. And that's a liberating concept. You aren't obligated to test everything, but you should test what really matters. What needs to be tested is probably directly related to the size and stability of the company (assuming size and stability have a positive correlation). I would venture to surmise that young start-ups have almost no need to test anything. That comment isn't meant to be inflammatory, but I look at testing too early like optimizing too early. There's no reason to shard a database until you absolutely have to, and I don't think you need to test something until you know it's mission critical. For example, Stripe offers a service that were components of it to fail would jeopardize their entire business. Their code base is obligated, by its nature, to have more stringent tests. But the startup in the garage next to yours who is still trying to determine MVP and will probably end up pivoting five times in the next two weeks? Save yourself a lot of grief and don't worry about the tests. Once you know what you are going to be, once you know what you can't afford to lose, well, wrap those portions up with some solid tests.

pifflesnort 14y ago
> I used the phrase "only test what needs to be tested" intentionally. How should I know what you need to test? But if you accept that you should only test what needs to be tested, then there is an implication that some (I'd venture to say most) of what you write doesn't need to be tested. And that's a liberating concept. You aren't obligated to test everything, but you should test what really matters. In -my- anecdotal experience, this is the excuse developers use while they write bug-ridden code without tests. Maybe they save a little time at the early stage of the development cycle, but the real reason seems to be avoiding work they don't find to be as fun as hacking out code. After that code is hacked out, we then waste a bunch of cycles across multiple teams while we find and fix out the most obvious of their bugs. After the release goes out, we spend even more time finding previously undiscovered bugs and fixing them -- which hits actual user satisfaction. All because some lazy sod thought he was too smart to need tests, and that he could be trusted to decide when his code didn't require them. Fuck that. I've been doing this too long and seen this happen too many times. If you're on my team, you write the damn tests.

wildwood 14y ago

1) Test your APIs. A public API, especially one that is key to your business, should be near 100% coverage. And attacked to look for security/usability/load problems. 2) Test enough during development to support later regression tests, and to make sure that the design is testable. This can usually be achieved with less than 20% coverage. But if you write production code that's so screwy it can't be regression tested, then you've got big problems. 3) Test any parts that scare you or confuse you or make you nervous. Use "test until you're more bored than scared" here.

endymi0n 14y ago
Terse, but extremely helpful and well formulated. I'll print these out for further reference ;-) I'd add 4) Continuous integration testing doubling as operations monitoring. I know that this doesn't "really" qualify as testing, but from my experience, it's far easier getting bitten by components not working together than from a screwed component alone. The key here is monitoring and testing business parameters from end to end. It doesn't help you finding out where the bug is, but it lets you know early on if there is one.

dspeyer 14y ago

Principles I've found: * A test should accept any correct behaviour from the tested code. Anything which is not in the requirements, should not be enforced by the test. * A test should not use the same logic as the code to find the "right" answer. * A function whose semantics are likely to be changed in the next refactor should not be invoked from a test. * Whenever a test fails, make a note of whether you got it to pass by changing the test or by changing the code. If it's the test more than 2/3 of the time, it's a bad test. * If you can't write a good test, don't write a test at all. See if you can write a good test for the code that calls this code instead.