4 ms·

I heard the "customers don't pay for tests" line recently, from someone boasting about how they migrated hundreds of thousands of lines of PHP from 5.x to 7.x.

I heard the "customers don't pay for tests" line recently, from someone boasting about how they migrated hundreds of thousands of lines of PHP from 5.x to 7.x. I didn't have an off-the-cuff reply to this, but thinking about it afterwards the issue became clear to me: customers don't pay for code either! Customers pay for solutions to their problems. I think the key phrase is "how do we know?": maybe those hundreds of thousands of lines of PHP were needed to solve the customers' problems, but how do we know? After all of that work, does it actually do what's needed? When the requirements inevitably change, how do we know when we've finished our patches? We can never know for sure, but there are ways to gain confidence in what we've done. Automated testing is a really low-cost way to gain quite a lot of confidence, which is also relatively stable over time. In contrast, manual tests are either very expensive or woefully shallow; and re-running them in the future takes just as much effort each time. Static analysis, code read-throughs, formal verification, etc. can give us more confidence than tests, but at a much higher cost. Simplifying the codebase can also help (code is a liability, not an asset!), but again that can be expensive. We should get the most bang for our buck: usually that means adding more tests. Occasionally, if that's not enough, we might sit down and prove something, or manually step though a print out, etc. but usually we could get more out of the same time by writing tests.

arkh 8y ago
> customers don't pay for code either! Customers pay for solutions to their problems. And that's why you don't test code. You test that your application comply with your customer's requirements. I'm happy to see things changing about tests with people realizing fine grained unit tests are often an hindrance and you should prioritize end to end testing. Test the interface of what you're selling, not the inner workings.
- purerandomness 8y ago
  Are you aware of the Test Pyramid, and why it exists?
  - arkh 8y ago
    > Are you aware of the Test Pyramid Yes. > and why it exists? Because the xUnit consultant crowd worked a lot, it is easier to make unit tests (which fossilize your code) so there's a lot of tooling around it. And the reliance on external services with no sandbox or ready to use mocks mean E2E is harder to implement. But being harder just mean you have to do your job. When I buy your software I don't care about how you implemented some pattern. What I care is that when I click on this gizmo in that situation it does what it should in some time-frame using some resources.
- TarpitCarnivore 8y ago
  > you should prioritize end to end testing. Test the interface of what you're selling, not the inner workings. E2E tests can be slow and extremely fragile though. They have a place and purpose, but they should not be the only form. Why would you skip entirely over "inner workings" tests that could catch bugs sooner?
- chriswarbo 8y ago
  I agree with what you say. That's why I tend to use the phrase "automated testing", rather than something more specific like "unit testing". The tests I write would mostly be called property tests and integration tests, although I think the terminology around testing is too loaded and ambiguous ( http://chriswarbo.net/blog/2017-11-10-unit_testing_terminology.html http://chriswarbo.net/blog/2017-11-10-unit_testing_terminolo... ). In any case, whilst I think it's good and healthy to debate the different forms of testing, their merits and tradeoffs, etc. I don't think it's appropriate when the alternative is not testing at all (e.g. "customers don't pay for tests"). I've worked at three organisations and managed to introduce automated testing to two of them. Even then, the test suites were "my responsibility", since (a) nobody else was running them and (b) I had to hand-hold people whenever I spotted a breakage (which they inevitably blamed on the test being wrong). The heuristics I've come to follow are: - Having automated tests is better than not having automated tests - It's easier to improve a bad test suite than it is to get a test suite added/accepted - There usually aren't enough tests There are exceptions to these rules, but I don't believe them without evidence. For example, "the test suite is slow" won't convince me that there are too many tests; demonstrating systematic redundancy and fragility in the test suite might convince me (e.g. exposing privates in order to test them). Only if people are on board with the idea of automated testing, will I bother to get more opinionated about the specifics.

arkh 8y ago

> customers don't pay for code either! Customers pay for solutions to their problems. And that's why you don't test code. You test that your application comply with your customer's requirements. I'm happy to see things changing about tests with people realizing fine grained unit tests are often an hindrance and you should prioritize end to end testing. Test the interface of what you're selling, not the inner workings.

purerandomness 8y ago
Are you aware of the Test Pyramid, and why it exists?
- arkh 8y ago
  > Are you aware of the Test Pyramid Yes. > and why it exists? Because the xUnit consultant crowd worked a lot, it is easier to make unit tests (which fossilize your code) so there's a lot of tooling around it. And the reliance on external services with no sandbox or ready to use mocks mean E2E is harder to implement. But being harder just mean you have to do your job. When I buy your software I don't care about how you implemented some pattern. What I care is that when I click on this gizmo in that situation it does what it should in some time-frame using some resources.
TarpitCarnivore 8y ago
> you should prioritize end to end testing. Test the interface of what you're selling, not the inner workings. E2E tests can be slow and extremely fragile though. They have a place and purpose, but they should not be the only form. Why would you skip entirely over "inner workings" tests that could catch bugs sooner?
chriswarbo 8y ago
I agree with what you say. That's why I tend to use the phrase "automated testing", rather than something more specific like "unit testing". The tests I write would mostly be called property tests and integration tests, although I think the terminology around testing is too loaded and ambiguous ( http://chriswarbo.net/blog/2017-11-10-unit_testing_terminology.html http://chriswarbo.net/blog/2017-11-10-unit_testing_terminolo... ). In any case, whilst I think it's good and healthy to debate the different forms of testing, their merits and tradeoffs, etc. I don't think it's appropriate when the alternative is not testing at all (e.g. "customers don't pay for tests"). I've worked at three organisations and managed to introduce automated testing to two of them. Even then, the test suites were "my responsibility", since (a) nobody else was running them and (b) I had to hand-hold people whenever I spotted a breakage (which they inevitably blamed on the test being wrong). The heuristics I've come to follow are: - Having automated tests is better than not having automated tests - It's easier to improve a bad test suite than it is to get a test suite added/accepted - There usually aren't enough tests There are exceptions to these rules, but I don't believe them without evidence. For example, "the test suite is slow" won't convince me that there are too many tests; demonstrating systematic redundancy and fragility in the test suite might convince me (e.g. exposing privates in order to test them). Only if people are on board with the idea of automated testing, will I bother to get more opinionated about the specifics.