5 ms·

>Can I do this faster if I don’t write a test? The answer is always no. I think the answer is mostly no but it's dangerous to think that it's always no. I've

>Can I do this faster if I don’t write a test? The answer is always no. I think the answer is mostly no but it's dangerous to think that it's always no. I've been given many stories in the past for which writing a realistic automated test would have taken days, manual verification took minutes and the code was fairly isolated and did not get changed very frequently. Writing a test under those circumstances is actually a pretty poor investment.

KirinDave 9y ago
That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.
- crdoconnor 9y ago
  5x 5 minute manual test < 1 day Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.
  - KirinDave 9y ago
    And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.
    - crdoconnor 9y ago
      Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level). I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma). >How many customer minutes lost = 1 developer minute Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true. I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute. As I mentioned above, I really don't think it pays to be a test fundamentalist. >That said, I think tests are the most expensive and most brittle way to address this problem. Or a type system fundamentalist.
      - KirinDave 9y ago
        Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests. Miserably bad. Unforgivably bad. Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing. Because otherwise, all we'll do is continue to wallow in pride and failure, claiming it can't be helped. All the while using language like "case by case," that I have taken to mean: "I will never do that unless you force me to." Fortunately, I think the scope of failure and fraud in the software industry has grown so late that folks are starting to take correctness as a requirement and not a nice to have. Another Equifax or two and maybe a nice DAO hack or something and folks are going to start saying, "Maybe it's just too bad we all learn to make bad software," turning to new techniques and practices. Far-fetched? Maybe. But it is happening with AI...
        crdoconnor 9y ago
        >that I have taken to mean: "I will never do that unless you force me to." I actually created my own open source BDD framework and a ton of tooling to help automate stories. 19 out of 20 was a pretty conservative estimate of how much I automate - it's probably more like 39 out of 40. I'm a little obsessive about it because I want to dogfood my work properly so I automate quite a few things where the cost/benefit for a normal programmer would seem a bit low. I'm very cognizant that the industry as a whole is terrible at testing and I'm hoping my software can one day do a small part to help with that.
simonhamp 9y ago
I would be very surprised if an automated test literally took days to write when a manual test is just minutes. Also the time savings still pay off later, as automated tests usually take seconds to run and there’s no training required - once it’s in the test suite and the test suite runs are automated, it will always run and quickly identify a failure - no “oops, we forgot to show Jim the Intern that he had to test that part manually...” Setting up your test suite and automation is longer for sure, but not days. Even a complex manual process can be automated relatively quickly... the manual process should be fairly scriptable in any OS nowadays, and most platforms have great test frameworks.
- andrewflnr 9y ago
  Think about something that necessarily involves hardware interaction, or a GUI, or where all the interesting error cases are non-deterministic (concurrency, network error handling). Ok, on the last one you're pretty much hosed anyway. But we're not all writing nice data-in/data-out apps.
  - crdoconnor 9y ago
    Actually, I probably wouldn't count network error handling, because you can probably use something like vaurien - https://github.com/community-libs/vaurien https://github.com/community-libs/vaurien - to deterministically mimic bad network conditions in an integration test in a fairly reasonable amount of time. Also, integrating that tool would probably have applications beyond a single story so even if the change takes 5 minutes and making the test work with vaurien takes half a day, it's probably still worthwhile. Assuming that no tool like vaurien exists, though (and there are plenty of scenarios out there where you'd have to build it from scratch), building the test scenario could become prohibitively expensive.
- crdoconnor 9y ago
  Trust me you'll come across examples like this one day if you make a concerted effort to automate a test for every story. It's a rough guess but I'd say I don't automate 1 out of 20 stories - I'm not exactly blithely unaware of the benefits an automated test suite brings. Most of the examples that require "days" (or weeks) for the test and 5 minutes for the test would involve the building or amending an elaborate mock/stub. Some examples where this happens include rare interactions with weird hardware, odd legacy APIs that are scheduled to be replaced, race conditions and obscure edge cases with mocked services. I document all of these special cases and they should clearly remain special but I'm not going to blindly assume that automating the story will have a positive ROI.

KirinDave 9y ago

That's a decision you can make, but all you're doing is costing yourself more time later for the benefit of less time now.

crdoconnor 9y ago
5x 5 minute manual test < 1 day Automatd test fundamentalists are subject to the same kinds of folly that other fundamentalists are.
- KirinDave 9y ago
  And when someone else modifies the code and introduces a subtle bug, will you attribute the costs handling that to this same timebucket? How many customer minutes lost = 1 developer minute? That said, I think tests are the most expensive and most brittle way to address this problem. They're necessary, but should be deployed sparingly.
  - crdoconnor 9y ago
    Yes, I would. However: A) In such cases I think testing against the real thing often has a greater chance of catching subtle bugs rather than the automated test scenario against the elaborate mock which is highly likely to share many of the same assumptions that caused the bug. B) Code reviews ought to flag that a piece of code that is not under automated test is being modified and appropriate care should be taken (ideally this alert should be automated but I haven't yet reached that level). I think it pays to approach these things on a case by case basis, and if a pattern of subtle bugs does appear that's a strong indication that you should change your behavior (I'm a stronger believer in bug retros than I am in any kind of testing dogma). >How many customer minutes lost = 1 developer minute Is that a relevant question to ask? If you introduce the presumption that an automated scenario test is more likely to catch a bug than a manual test then I guess it's relevant, but honestly for these types of scenario I think the opposite is true. I didn't mention it before also but if you have manual testers on hand that changes the calculus too. I'd say it's normal for 3-4 manual tester minutes to be equivalent to about 1 developer minute. As I mentioned above, I really don't think it pays to be a test fundamentalist. >That said, I think tests are the most expensive and most brittle way to address this problem. Or a type system fundamentalist.
    - KirinDave 9y ago
      Actually, my personal belief is that people, humans like you and I, are all awful at writing software. We're even worse at enumerating and writing tests. Miserably bad. Unforgivably bad. Slowly refining and testing systematizations of correct software building processes is perhaps the most important thing we can do in the first 2 centuries of "software" as a thing. Because otherwise, all we'll do is continue to wallow in pride and failure, claiming it can't be helped. All the while using language like "case by case," that I have taken to mean: "I will never do that unless you force me to." Fortunately, I think the scope of failure and fraud in the software industry has grown so late that folks are starting to take correctness as a requirement and not a nice to have. Another Equifax or two and maybe a nice DAO hack or something and folks are going to start saying, "Maybe it's just too bad we all learn to make bad software," turning to new techniques and practices. Far-fetched? Maybe. But it is happening with AI...
      - crdoconnor 9y ago
        >that I have taken to mean: "I will never do that unless you force me to." I actually created my own open source BDD framework and a ton of tooling to help automate stories. 19 out of 20 was a pretty conservative estimate of how much I automate - it's probably more like 39 out of 40. I'm a little obsessive about it because I want to dogfood my work properly so I automate quite a few things where the cost/benefit for a normal programmer would seem a bit low. I'm very cognizant that the industry as a whole is terrible at testing and I'm hoping my software can one day do a small part to help with that.

simonhamp 9y ago

I would be very surprised if an automated test literally took days to write when a manual test is just minutes. Also the time savings still pay off later, as automated tests usually take seconds to run and there’s no training required - once it’s in the test suite and the test suite runs are automated, it will always run and quickly identify a failure - no “oops, we forgot to show Jim the Intern that he had to test that part manually...” Setting up your test suite and automation is longer for sure, but not days. Even a complex manual process can be automated relatively quickly... the manual process should be fairly scriptable in any OS nowadays, and most platforms have great test frameworks.

andrewflnr 9y ago
Think about something that necessarily involves hardware interaction, or a GUI, or where all the interesting error cases are non-deterministic (concurrency, network error handling). Ok, on the last one you're pretty much hosed anyway. But we're not all writing nice data-in/data-out apps.
- crdoconnor 9y ago
  Actually, I probably wouldn't count network error handling, because you can probably use something like vaurien - https://github.com/community-libs/vaurien https://github.com/community-libs/vaurien - to deterministically mimic bad network conditions in an integration test in a fairly reasonable amount of time. Also, integrating that tool would probably have applications beyond a single story so even if the change takes 5 minutes and making the test work with vaurien takes half a day, it's probably still worthwhile. Assuming that no tool like vaurien exists, though (and there are plenty of scenarios out there where you'd have to build it from scratch), building the test scenario could become prohibitively expensive.
crdoconnor 9y ago
Trust me you'll come across examples like this one day if you make a concerted effort to automate a test for every story. It's a rough guess but I'd say I don't automate 1 out of 20 stories - I'm not exactly blithely unaware of the benefits an automated test suite brings. Most of the examples that require "days" (or weeks) for the test and 5 minutes for the test would involve the building or amending an elaborate mock/stub. Some examples where this happens include rare interactions with weird hardware, odd legacy APIs that are scheduled to be replaced, race conditions and obscure edge cases with mocked services. I document all of these special cases and they should clearly remain special but I'm not going to blindly assume that automating the story will have a positive ROI.