6 ms·

The most important question to ask when writing tests: Can I do this faster if I don’t write a test? The answer is always no. Even if you are the only person b

The most important question to ask when writing tests: Can I do this faster if I don’t write a test? The answer is always no. Even if you are the only person building something, future you will lick your boots clean in gratitude if there are tests. Because even the best developers have to work with their own code sometimes.

mooreds 9y ago
This! I became a believer in automated testing when I worked with a large ETL process that worked with real estate data. This data was input by real estate agents, varied wildly in quality, and had both image files and structured text data. Releasing this code before automated testing was fearful. We'd push changes to staging and then wait for three (!) days of data, then examine the staging and production databases manually and see if there were material differences. Needless to say, we didn't like to release this code very often. When I left, we weren't doing automated testing on the image processing, but we were on the text side of things. It became far easier to do a release because we had a set of regression tests that gave us confidence we weren't breaking anything. If something did pop up, we could add it to the suite. Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.
- maksimum 9y ago
  > Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests. This! What's also worrisome is that sometimes your incentives are misaligned; e.g. when product development and maintenance are handled by different teams, or features have tight deadlines and screw the debt that will be the future and we must deal with now.
qaq 9y ago
I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.
- simonhamp 9y ago
  Of course, just because you haven’t personally written tests for parts of a stack doesn’t mean that you shouldn’t write tests for other parts, especially the parts you’ve created. Also don’t forget that certain things like Linux are extremely battle-tested. It’s generally unlikely that any piece of software you write will receive that much real-world testing, so a growing set of automated tests will help you cover your own arse.
crdoconnor 9y ago
>Can I do this faster if I don’t write a test? The answer is always no. I think the answer is mostly no but it's dangerous to think that it's always no. I've been given many stories in the past for which writing a realistic automated test would have taken days, manual verification took minutes and the code was fairly isolated and did not get changed very frequently. Writing a test under those circumstances is actually a pretty poor investment.
- KirinDave 9y ago
  That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.
  - crdoconnor 9y ago
    5x 5 minute manual test < 1 day Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.
    - KirinDave 9y ago
      And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.
      - crdoconnor 9y ago
        Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level). I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma). >How many customer minutes lost = 1 developer minute Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true. I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute. As I mentioned above, I really don't think it pays to be a test fundamentalist. >That said, I think tests are the most expensive and most brittle way to address this problem. Or a type system fundamentalist.
KirinDave 9y ago
I wish people would extend this same logic to type systems and formal methods.
- mooreds 9y ago
  Isn't the wave of FP causing some folks to do this? It takes time for people to change their attitudes. Consider that we're still having this conversation about automated testing, which was promoted by the XP book in the early 2000s.
muzani 9y ago
Perhaps this always applies for manual testing. When it comes to automated testing and something like Android, it can potentially take much longer, especially with the architecture that allows easy testing.

mooreds 9y ago

This! I became a believer in automated testing when I worked with a large ETL process that worked with real estate data. This data was input by real estate agents, varied wildly in quality, and had both image files and structured text data. Releasing this code before automated testing was fearful. We'd push changes to staging and then wait for three (!) days of data, then examine the staging and production databases manually and see if there were material differences. Needless to say, we didn't like to release this code very often. When I left, we weren't doing automated testing on the image processing, but we were on the text side of things. It became far easier to do a release because we had a set of regression tests that gave us confidence we weren't breaking anything. If something did pop up, we could add it to the suite. Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests.

maksimum 9y ago
> Human beings systematically undervalue their future selves. This is why we have a hard time saving for retirement, exercising and writing tests. Think of your future self and write tests. This! What's also worrisome is that sometimes your incentives are misaligned; e.g. when product development and maintenance are handled by different teams, or features have tight deadlines and screw the debt that will be the future and we must deal with now.

qaq 9y ago

I like having tests but I would bet if you audit you production stack it's very likely that a very large portion of it does not really have tests yet is prob most stable part of your stack like Linux kernel etc.

simonhamp 9y ago
Of course, just because you haven’t personally written tests for parts of a stack doesn’t mean that you shouldn’t write tests for other parts, especially the parts you’ve created. Also don’t forget that certain things like Linux are extremely battle-tested. It’s generally unlikely that any piece of software you write will receive that much real-world testing, so a growing set of automated tests will help you cover your own arse.

crdoconnor 9y ago

>Can I do this faster if I don’t write a test? The answer is always no. I think the answer is mostly no but it's dangerous to think that it's always no. I've been given many stories in the past for which writing a realistic automated test would have taken days, manual verification took minutes and the code was fairly isolated and did not get changed very frequently. Writing a test under those circumstances is actually a pretty poor investment.

KirinDave 9y ago
That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.
- crdoconnor 9y ago
  5x 5 minute manual test < 1 day Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.
  - KirinDave 9y ago
    And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.
    - crdoconnor 9y ago
      Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level). I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma). >How many customer minutes lost = 1 developer minute Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true. I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute. As I mentioned above, I really don't think it pays to be a test fundamentalist. >That said, I think tests are the most expensive and most brittle way to address this problem. Or a type system fundamentalist.

KirinDave 9y ago

I wish people would extend this same logic to type systems and formal methods.

mooreds 9y ago
Isn't the wave of FP causing some folks to do this? It takes time for people to change their attitudes. Consider that we're still having this conversation about automated testing, which was promoted by the XP book in the early 2000s.

muzani 9y ago

Perhaps this always applies for manual testing. When it comes to automated testing and something like Android, it can potentially take much longer, especially with the architecture that allows easy testing.