14 ms·

Software testing, and why I'm unhappy about it

78 points by drothlis 4y ago

ranting-moth 4y ago
> The above development practice works well when the SUT and TB are both defined by the same code repository and are developed together. I once witnessed a team creating an app, specs and tests in three respective repositories. For no other reason than "each project should be in it's own repository". The added work/maintenance around that is crazy, for absolutely no gain in that case.
nurettin 4y ago
If I am given the time and resources I do this: Phase 1. Code and test basic functions concerning any kind of arithmetic, mathematical distribution, state machines, file operations and datetimes. This documents any assumptions and makes a solid foundation. Phase 2. Write a simulation for generating randomized inputs to test the whole system. Run it for hours. If I can't generate the inputs, find as big a variety of inputs as possible. Collect any bugs, fix, repeat. This reduces the chances of finding real time bugs by three orders of magnitude. This has worked really well in the past whether I'm working on games, parsers or financial software. I don't conform to corporate whatever driven testing patterns because they are usually missing the crucial part 2 and time part 1 incorrectly.
t00 4y ago
Have I misunderstood the article or it is just a matter of separating feature branches and putting relevant tests in a feature branch while keeping regression in a master branch?
drothlis 4y ago
Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).
- Gigachad 4y ago
  Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.
  - jeroenhd 4y ago
    I can't think of any project I've worked on where external test suites even make sense. I suppose it would work when you have a very clear spec or compliance document you could write independent tests, or if you're rewriting a system and need the public API to be bug-for-bug compatible with the old one, but other than those niche use cases I wouldn't want to keep those tests external at all. Even if you do have external tests, you still need internal ones for the surface area your external tests don't check for. Unit tests and such don't make sense at all combined with a separate test repo.
  - adamwk 4y ago
    It also makes it impossible to test outside of the public API
  - cstoner 4y ago
    The only time "tests in a separate repo" makes sense to me is if they are truly cross-functional end to end tests that exercise several systems. Those tests should be as small as possible to verify that everything is still wired together correctly. Everything else should be either unit tests or narrow integration tests between a small handful of components. And as you said, they should live in the repository of the software they test.
  - drothlis 4y ago
    Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.
    - Gigachad 4y ago
mixedCase 4y ago
> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based development and see if you like the waits and constant rebasing in large projects/teams. But don't eff with the bloody tests!
- emptysea 4y ago
  when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket will be created for the test owner to fix their test (and it will be disabled for future test runs) I think for small to medium projects, you can have all tests succeed but once the repo is large enough / has frequent enough changes, flakey tests are bound to slip in.
  - wpietri 4y ago
    That sounds like it works, but shouldn't flaky tests be detected by frequent automatic test suite runs?
    - emptysea 4y ago
      yeah I think the system is a combination, although I'm not sure the intricacies tbh I've heard google does something fancier where they take a test, run it a bunch after it fails to check if it's failed I think the system at work only runs each test a couple times before giving up and marking it failed
  - idontwantthis 4y ago
    That sounds fantastic. What tool are you describing?
drewcoo 4y ago
Doctor, it hurts when I punch myself in the head! If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them. Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff. Test failures can be thought of as "things developers should investigate." Make sure the tests are focused on telling you about those things as fast as possible. Also, take the human out of the "wait for green, then submit PR" steps. Open a PR but don't alert everyone else about it until you run green, maybe?
- kubanczyk 4y ago
  > take the human out of the "wait for green, then submit PR" It'd be great if GitHub could open a PR for reviews (aka un-draft) automatically after CI succeeds. (If not in the core product, is there a bot that does that?)
  - mike_hearn 4y ago
    My company uses a workflow where we don't use PRs for code reviews. Instead we each have our own git repo that's a fork of the tech lead's, with some git rules in place to impose a branch namespace. To open a review request you push a branch into the reviewer's repository. Our CI system detects the new branch and starts running it. Once CI passes that updates the bug tracker which triggers a notification to the reviewer. The reviewer then does a git fetch, and then checks out the newly created rr/ branch. They make any small changes that aren't worth a roundtrip and push them to the rr branch. They add FIXME comments for bigger changes. They then either assign the ticket back to the developer, or go ahead and merge straight into their own dev branch. Once an rr branch is merged it's simply deleted. The dev branch is then pushed and CI will merge it to that user's master when it's green. IntelliJ will show branches in each origin organized by "folder" if you use backslashes in branch names, and gitolite (which is what we use to run our repos) can impose ACLs by branch name too. So for example only user alice can push to a branch named rr/alice/whatever in each persons repo. That ensures it's always clear where a PR/RR is coming from. Because each user gets their own git repo and cloned set of individual CI builds, you can push experimental or WIP branches to your personal area and iterate there without bothering other people. This workflow gets rid of things like draft PRs (which are a contradiction), it ensures each reviewer has a personal review queue, it means work and progress is tracked via the bug tracker (which understands commands in commit messages so you can mark bugs as fixed when they clear CI automatically) and it eliminates the practice of requesting dozens of tiny changes that'd be faster for the reviewer to apply themselves, because reviewer and task owner can trade commits on the rr branch using git's features to keep it all organized and mergeable.
theamk 4y ago
The author's problem is pretty simple: the test repo is required for pre-merge tests to pass, but it can be updated independently, without having pre-merge tests pass. And the answer is pretty simple: pin the specific test repo version! Use lockfiles, or git submodules, or put "cd tests && git checkout 3e524575cc61" in your CI config file _and keep it in the same repo as source code_ (that part is very important!). This solves all of author problems: > new test case is added to the conformance test suite, but that test happens to fail. Suddenly nobody can submit any changes anymore. Conformance test suite is pinned, so new test is not used. A separate PR has to update conformance test suite version/revision, and it must go through regular driver PR process and therefore must pass. Practically, this is a PR with 2 changes: update pin and disable new test. > are you going to remember to update that exclusion list? That's why you use "expect fail" list (not exclusion) and keep it in driver's dir. Ad you submit your PR you might see a failure saying: "congrats, test X which was expect-fail is now passing! Please remove it from the list". You'll need to make one more PR revision but then you get working tests. > allowing tests to be marked as "expected to fail". But they typically also assume that the TB can be changed in lockstep with the SUT and fall on their face when that isn't the case. And if your TB cannot be changed in lockstep with SUT, you are going to have truly miserable time. You cannot even reproduce the problems of the past! So make sure your kernel is known or at least recorded, repos are pinned. Ideally the whole machine image, with packages and all is archived somehow -- maybe via docker or raw disk image or some sort of ostree system. > Problem #2 is that good test coverage means that tests take a very long time to run. The described system sounds very nice, and I would love to have something like this. I suspect it will be non-trivial to get working, however. But meanwhile, there is a manual solution: have more than one test suite. "Pre-merge" tests run before each merge and contain small subset of testing. A bigger "continuous" test suite (if you use physical machines) or "every X hours" (if you use some sort of auto-scaling cloud) will run a bigger set of tests, and can be triggered manually on PRs if a developer suspects the PR is especially risky. You can even have multiple levels (pre-merge, once per hour, 4 times per day) but this is often more trouble than it worth. And of course it is absolutely critical to have reproducible tests first -- if you come up to work and find a bunch of continuous failures, you want to be able to re-run with extra debugging or bisect what happened.
- 4y ago
andreareina 4y ago
The continuous testing is something I’ve thought about and it’s a tricky one. We use property tests[1] so here’s a quick stab at how I’d like it to look like: Test starts failing, immediately send a report with the failing input, then continue with the test case minimisation and send another report when that finishes. Concurrently, start up another long running process to look for other failures, skipping the input that caused the previous failure. We do want new inputs for the same failure though. This is the tricky one. We could probably make it work by having the prop test framework not reuse previously-failing inputs, but that’s one of the big strategies it uses to catch regressions. [1] specifically, hypothesis on python
13of40 4y ago
I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in some code, you're only giving the poor bastard investigating it two puzzles to debug instead of one.
- dgb23 4y ago
  What about data driven tests where you lay out several variants, including edge cases, for function arguments? Seems pretty clear to me.
- midasz 4y ago
  I always use (if the scenario is simple enough, which most are): @Test public void myTestMethod_Scenario_ShouldReturnThis() {....
  - geraldwhen 4y ago
    Jest makes this far more straightforward. It(“throws when the object belongs to another user”) It(“does a business thing when thing is in state BLAH”)
- sirsinsalot 4y ago
  To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention. I don't think it quite does it right, but it is of note.
  - hgomersall 4y ago
    We have a policy of making each test a spec. That is, a test requires a plain text spec to be attached to it in its doc string. It's kind of like BDD but without all the weird DSLs.
- 4y ago
gampleman 4y ago
Seems to me like you're underinvesting in tooling. It's a mistake a lot of development shops make - you focus on your product, so you can't spend time building something completely orthogonal, but in the process you suddenly waste man-years wasting time on a broken PR process, instead of spending a month early on building some tooling that would have removed the pain in the first place.

78 points by drothlis 4y ago

> The above development practice works well when the SUT and TB are both defined by the same code repository and are developed together. I once witnessed a team creating an app, specs and tests in three respective repositories. For no other reason than "each project should be in it's own repository". The added work/maintenance around that is crazy, for absolutely no gain in that case.

nurettin 4y ago

If I am given the time and resources I do this: Phase 1. Code and test basic functions concerning any kind of arithmetic, mathematical distribution, state machines, file operations and datetimes. This documents any assumptions and makes a solid foundation. Phase 2. Write a simulation for generating randomized inputs to test the whole system. Run it for hours. If I can't generate the inputs, find as big a variety of inputs as possible. Collect any bugs, fix, repeat. This reduces the chances of finding real time bugs by three orders of magnitude. This has worked really well in the past whether I'm working on games, parsers or financial software. I don't conform to corporate whatever driven testing patterns because they are usually missing the crucial part 2 and time part 1 incorrectly.

t00 4y ago

Have I misunderstood the article or it is just a matter of separating feature branches and putting relevant tests in a feature branch while keeping regression in a master branch?

drothlis 4y ago

Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).

Gigachad 4y ago
Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.
- jeroenhd 4y ago
  I can't think of any project I've worked on where external test suites even make sense. I suppose it would work when you have a very clear spec or compliance document you could write independent tests, or if you're rewriting a system and need the public API to be bug-for-bug compatible with the old one, but other than those niche use cases I wouldn't want to keep those tests external at all. Even if you do have external tests, you still need internal ones for the surface area your external tests don't check for. Unit tests and such don't make sense at all combined with a separate test repo.
- adamwk 4y ago
  It also makes it impossible to test outside of the public API
- cstoner 4y ago
  The only time "tests in a separate repo" makes sense to me is if they are truly cross-functional end to end tests that exercise several systems. Those tests should be as small as possible to verify that everything is still wired together correctly. Everything else should be either unit tests or narrow integration tests between a small handful of components. And as you said, they should live in the repository of the software they test.
- drothlis 4y ago
  Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.
  - Gigachad 4y ago

mixedCase 4y ago

> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based development and see if you like the waits and constant rebasing in large projects/teams. But don't eff with the bloody tests!

emptysea 4y ago
when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket will be created for the test owner to fix their test (and it will be disabled for future test runs) I think for small to medium projects, you can have all tests succeed but once the repo is large enough / has frequent enough changes, flakey tests are bound to slip in.
- wpietri 4y ago
  That sounds like it works, but shouldn't flaky tests be detected by frequent automatic test suite runs?
  - emptysea 4y ago
    yeah I think the system is a combination, although I'm not sure the intricacies tbh I've heard google does something fancier where they take a test, run it a bunch after it fails to check if it's failed I think the system at work only runs each test a couple times before giving up and marking it failed
- idontwantthis 4y ago
  That sounds fantastic. What tool are you describing?

drewcoo 4y ago

Doctor, it hurts when I punch myself in the head! If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them. Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff. Test failures can be thought of as "things developers should investigate." Make sure the tests are focused on telling you about those things as fast as possible. Also, take the human out of the "wait for green, then submit PR" steps. Open a PR but don't alert everyone else about it until you run green, maybe?

kubanczyk 4y ago
> take the human out of the "wait for green, then submit PR" It'd be great if GitHub could open a PR for reviews (aka un-draft) automatically after CI succeeds. (If not in the core product, is there a bot that does that?)
- mike_hearn 4y ago
  My company uses a workflow where we don't use PRs for code reviews. Instead we each have our own git repo that's a fork of the tech lead's, with some git rules in place to impose a branch namespace. To open a review request you push a branch into the reviewer's repository. Our CI system detects the new branch and starts running it. Once CI passes that updates the bug tracker which triggers a notification to the reviewer. The reviewer then does a git fetch, and then checks out the newly created rr/ branch. They make any small changes that aren't worth a roundtrip and push them to the rr branch. They add FIXME comments for bigger changes. They then either assign the ticket back to the developer, or go ahead and merge straight into their own dev branch. Once an rr branch is merged it's simply deleted. The dev branch is then pushed and CI will merge it to that user's master when it's green. IntelliJ will show branches in each origin organized by "folder" if you use backslashes in branch names, and gitolite (which is what we use to run our repos) can impose ACLs by branch name too. So for example only user alice can push to a branch named rr/alice/whatever in each persons repo. That ensures it's always clear where a PR/RR is coming from. Because each user gets their own git repo and cloned set of individual CI builds, you can push experimental or WIP branches to your personal area and iterate there without bothering other people. This workflow gets rid of things like draft PRs (which are a contradiction), it ensures each reviewer has a personal review queue, it means work and progress is tracked via the bug tracker (which understands commands in commit messages so you can mark bugs as fixed when they clear CI automatically) and it eliminates the practice of requesting dozens of tiny changes that'd be faster for the reviewer to apply themselves, because reviewer and task owner can trade commits on the rr branch using git's features to keep it all organized and mergeable.

theamk 4y ago

The author's problem is pretty simple: the test repo is required for pre-merge tests to pass, but it can be updated independently, without having pre-merge tests pass. And the answer is pretty simple: pin the specific test repo version! Use lockfiles, or git submodules, or put "cd tests && git checkout 3e524575cc61" in your CI config file _and keep it in the same repo as source code_ (that part is very important!). This solves all of author problems: > new test case is added to the conformance test suite, but that test happens to fail. Suddenly nobody can submit any changes anymore. Conformance test suite is pinned, so new test is not used. A separate PR has to update conformance test suite version/revision, and it must go through regular driver PR process and therefore must pass. Practically, this is a PR with 2 changes: update pin and disable new test. > are you going to remember to update that exclusion list? That's why you use "expect fail" list (not exclusion) and keep it in driver's dir. Ad you submit your PR you might see a failure saying: "congrats, test X which was expect-fail is now passing! Please remove it from the list". You'll need to make one more PR revision but then you get working tests. > allowing tests to be marked as "expected to fail". But they typically also assume that the TB can be changed in lockstep with the SUT and fall on their face when that isn't the case. And if your TB cannot be changed in lockstep with SUT, you are going to have truly miserable time. You cannot even reproduce the problems of the past! So make sure your kernel is known or at least recorded, repos are pinned. Ideally the whole machine image, with packages and all is archived somehow -- maybe via docker or raw disk image or some sort of ostree system. > Problem #2 is that good test coverage means that tests take a very long time to run. The described system sounds very nice, and I would love to have something like this. I suspect it will be non-trivial to get working, however. But meanwhile, there is a manual solution: have more than one test suite. "Pre-merge" tests run before each merge and contain small subset of testing. A bigger "continuous" test suite (if you use physical machines) or "every X hours" (if you use some sort of auto-scaling cloud) will run a bigger set of tests, and can be triggered manually on PRs if a developer suspects the PR is especially risky. You can even have multiple levels (pre-merge, once per hour, 4 times per day) but this is often more trouble than it worth. And of course it is absolutely critical to have reproducible tests first -- if you come up to work and find a bunch of continuous failures, you want to be able to re-run with extra debugging or bisect what happened.

4y ago

andreareina 4y ago

The continuous testing is something I’ve thought about and it’s a tricky one. We use property tests[1] so here’s a quick stab at how I’d like it to look like: Test starts failing, immediately send a report with the failing input, then continue with the test case minimisation and send another report when that finishes. Concurrently, start up another long running process to look for other failures, skipping the input that caused the previous failure. We do want new inputs for the same failure though. This is the tricky one. We could probably make it work by having the prop test framework not reuse previously-failing inputs, but that’s one of the big strategies it uses to catch regressions. [1] specifically, hypothesis on python

13of40 4y ago

I have a lot of bitter things to say about automated testing, having spent 14 years of my life trying to knead it into a legitimate profession, but here's the most significant: You test case is more useless than a turd in the middle of the dining room table unless you put a comment in front of it that explains what it assumes, what it attempts, and what you expect to happen as a result. Because if you just throw in some code, you're only giving the poor bastard investigating it two puzzles to debug instead of one.

dgb23 4y ago
What about data driven tests where you lay out several variants, including edge cases, for function arguments? Seems pretty clear to me.
midasz 4y ago
I always use (if the scenario is simple enough, which most are): @Test public void myTestMethod_Scenario_ShouldReturnThis() {....
- geraldwhen 4y ago
  Jest makes this far more straightforward. It(“throws when the object belongs to another user”) It(“does a business thing when thing is in state BLAH”)
sirsinsalot 4y ago
To some degree, this is what BDD attempts to solve, separation of test mechanics and documentation of the test's intention. I don't think it quite does it right, but it is of note.
- hgomersall 4y ago
  We have a policy of making each test a spec. That is, a test requires a plain text spec to be attached to it in its doc string. It's kind of like BDD but without all the weird DSLs.
4y ago

gampleman 4y ago

Seems to me like you're underinvesting in tooling. It's a mistake a lot of development shops make - you focus on your product, so you can't spend time building something completely orthogonal, but in the process you suddenly waste man-years wasting time on a broken PR process, instead of spending a month early on building some tooling that would have removed the pain in the first place.