4 ms·

I don't know. When working in an unfamiliar codebase, tests are very helpful (even when some percentage of them will always be annoying). The problem is, the in

I don't know. When working in an unfamiliar codebase, tests are very helpful (even when some percentage of them will always be annoying). The problem is, the industry has focused on having lots of tests instead of having good code.

TylerE 3y ago
One problem I don't see talked about enough - to the point where I don't even have a name for it - goes something like this, You have a giant, long lived system, like decade plus. You've got huge number of both unit and integration tests. You're cruising along, implementing a new feature. Everything's looking good from your local manual tests and the tests you wrote over the new behavior. Then you ship the branch up to the test runner to let the full suite cook, and you get it back, and there's like 9 random test fails that aren't in master. Has your change had unintended consequences? Is the test just out of date? If the test is out of date, how do we update it such that it both passes and we are at least reasonably confident that the test is still actually testing what it was intended to test? Where it gets real obnoxious is when the test isn't testing anything even vaguely related to your change so it can turn into a real snipe hunt. Five minutes here, ten minutes there, it adds up. What's frustrating is that the number of times this has found an actual issue that I can personally remember.. I won't swear it's zero, but I'm struggling to recall a specific example.
- osigurdson 3y ago
  I guess it all depends on the value of developer time vs the cost of releasing with an undiscovered bug. In any case, I'm glad the industry is starting to realize that tests are code, code can have bugs and code can be stupid.
  - TylerE 3y ago
    What’s kinda funny is that there have been a few where we’ve had old code tests written sort of in the form of “try this thing which this system should reject as impossible” and then you realize that after your change that in fact the business rules actually handle this corner case gracefully instead of exposing… I’ll admit this trade off is probably helped by our clients being mostly state govt, so they’re often grateful to get anything at all functional. One never wants to ship bugs, we’ve actually had plenty of positive client interaction along the lines of “Hey, we’re just impressed you caught it and are the process of pushing a fix before we even noticed”.
    - osigurdson 3y ago
      This is a nitpick' but I wish the phrase "business logic" should go away. Are we not just testing logic? It makes all software sound like a tax preparation program.

TylerE 3y ago

One problem I don't see talked about enough - to the point where I don't even have a name for it - goes something like this, You have a giant, long lived system, like decade plus. You've got huge number of both unit and integration tests. You're cruising along, implementing a new feature. Everything's looking good from your local manual tests and the tests you wrote over the new behavior. Then you ship the branch up to the test runner to let the full suite cook, and you get it back, and there's like 9 random test fails that aren't in master. Has your change had unintended consequences? Is the test just out of date? If the test is out of date, how do we update it such that it both passes and we are at least reasonably confident that the test is still actually testing what it was intended to test? Where it gets real obnoxious is when the test isn't testing anything even vaguely related to your change so it can turn into a real snipe hunt. Five minutes here, ten minutes there, it adds up. What's frustrating is that the number of times this has found an actual issue that I can personally remember.. I won't swear it's zero, but I'm struggling to recall a specific example.

osigurdson 3y ago
I guess it all depends on the value of developer time vs the cost of releasing with an undiscovered bug. In any case, I'm glad the industry is starting to realize that tests are code, code can have bugs and code can be stupid.
- TylerE 3y ago
  What’s kinda funny is that there have been a few where we’ve had old code tests written sort of in the form of “try this thing which this system should reject as impossible” and then you realize that after your change that in fact the business rules actually handle this corner case gracefully instead of exposing… I’ll admit this trade off is probably helped by our clients being mostly state govt, so they’re often grateful to get anything at all functional. One never wants to ship bugs, we’ve actually had plenty of positive client interaction along the lines of “Hey, we’re just impressed you caught it and are the process of pushing a fix before we even noticed”.
  - osigurdson 3y ago
    This is a nitpick' but I wish the phrase "business logic" should go away. Are we not just testing logic? It makes all software sound like a tax preparation program.