4 ms·

I've found 80% to be a good number for unit tests. BUT things like integration testing and end-to-end user story testing can really make all the difference, and

I've found 80% to be a good number for unit tests. BUT things like integration testing and end-to-end user story testing can really make all the difference, and complement a solid unit test suite. Also focussing on the numbers can leave you going down the wrong path and trying to shoehorn as much testing in as possible without giving any tangible benefit for the sake of a higher number. As it's a young startup I would focus on writing tests for the business-critical use cases, which should allow a team the confidence to "move fast and break things" knowing comfortably that the day-to-day is still functioning as per usual. Ultimately, I would say most code should be covered with unit tests, but functionality covered by automated integration/end-to-end testing. To me that would be 'well-tested'. (and no flakiness!)

ajeet_dhaliwal 7y ago
I broadly agree with this approach. Automated functional end-to-end tests may not pinpoint what line of code has an issue but they will identify there’s a problem somewhere which is better than not knowing. Your product code and tests can be setup to help isolate where the source of the issue may be. Have developers continue to write unit tests but if you’re big enough to it have one developer focused on writing end-to-end automated functional tests it will give everyone a sense of assurance at each release that nothing is fundamentally broken. If you’re not big enough, like the parent comment said, try to do the major ones eg. sign up flow, core functionality. Someone else made the point about using data effectively and I agree with that too. I’ve been involved with automated testing and tooling for a few years now and solid test reporting and test case management is something I advocate. I’m the lead dev and founder at Tesults (http://www.tesults.com http://www.tesults.com) - try it and push your results data to it and house your manual test cases. There’s a free tier and if you contact me I’m happy to expand it. Basically if you set up your CI system to run your tests regularly, or even if you schedule them to run regularly (E.g. nightly/daily) then having something like this boosts your ROI because you can track history easily, link bugs, and do things like get indications about which tests may be flaky. We don’t know exactly how small your team is, it may be you have to rely on manual testing if there’s no one to spare but I’d try to move some of your end-to-end to be automated sooner rather later if you end up growing otherwise you’ll make testing a bottleneck to releasing, full regression testing of even a moderately complex application can end up taking days.
- tcbasche 7y ago
  Why would I want to link tests to bugs? Or track the history of tests? Or assign someone to a failing test? I've found they either fail or they pass, and you find why they failed and you fix the failure. Just genuinely curious about the purpose of this software, as it's quite expensive, and most CI tools provide enough information for history / scheduling etc.
  - ajeet_dhaliwal 7y ago
    If you don’t see it as something you need, you probably don’t ... at least not yet. Ive been an Engineering Manager and Head of Quality Engineering and when you end up with hundreds or thousands of test cases, either in one product or service or across many, managing the output of testing becomes a challenge. There may be low priority regressions that you want to fix later and create bugs for and link to failing tests so you know you’re waiting on those fixes before you expect the tests to pass. If there is a problem with a test itself then assigning it to a team member to fix is useful. Tracking history helps identify flaky test cases - they’re automatically highlighted as being flaky. Consolidating tests from several sources helps understand the overall health of a project. If you’re maintaining a product or service that goes through a lot of change and there’s say 10 people or more on a team managing all of this becomes increasingly difficult. With respect to pricing it’s free to try and $25/month per target so not particularly expensive.

ajeet_dhaliwal 7y ago

I broadly agree with this approach. Automated functional end-to-end tests may not pinpoint what line of code has an issue but they will identify there’s a problem somewhere which is better than not knowing. Your product code and tests can be setup to help isolate where the source of the issue may be. Have developers continue to write unit tests but if you’re big enough to it have one developer focused on writing end-to-end automated functional tests it will give everyone a sense of assurance at each release that nothing is fundamentally broken. If you’re not big enough, like the parent comment said, try to do the major ones eg. sign up flow, core functionality. Someone else made the point about using data effectively and I agree with that too. I’ve been involved with automated testing and tooling for a few years now and solid test reporting and test case management is something I advocate. I’m the lead dev and founder at Tesults (http://www.tesults.com http://www.tesults.com) - try it and push your results data to it and house your manual test cases. There’s a free tier and if you contact me I’m happy to expand it. Basically if you set up your CI system to run your tests regularly, or even if you schedule them to run regularly (E.g. nightly/daily) then having something like this boosts your ROI because you can track history easily, link bugs, and do things like get indications about which tests may be flaky. We don’t know exactly how small your team is, it may be you have to rely on manual testing if there’s no one to spare but I’d try to move some of your end-to-end to be automated sooner rather later if you end up growing otherwise you’ll make testing a bottleneck to releasing, full regression testing of even a moderately complex application can end up taking days.

tcbasche 7y ago
Why would I want to link tests to bugs? Or track the history of tests? Or assign someone to a failing test? I've found they either fail or they pass, and you find why they failed and you fix the failure. Just genuinely curious about the purpose of this software, as it's quite expensive, and most CI tools provide enough information for history / scheduling etc.
- ajeet_dhaliwal 7y ago
  If you don’t see it as something you need, you probably don’t ... at least not yet. Ive been an Engineering Manager and Head of Quality Engineering and when you end up with hundreds or thousands of test cases, either in one product or service or across many, managing the output of testing becomes a challenge. There may be low priority regressions that you want to fix later and create bugs for and link to failing tests so you know you’re waiting on those fixes before you expect the tests to pass. If there is a problem with a test itself then assigning it to a team member to fix is useful. Tracking history helps identify flaky test cases - they’re automatically highlighted as being flaky. Consolidating tests from several sources helps understand the overall health of a project. If you’re maintaining a product or service that goes through a lot of change and there’s say 10 people or more on a team managing all of this becomes increasingly difficult. With respect to pricing it’s free to try and $25/month per target so not particularly expensive.