3 ms·
I never clear database, use fuzzy tests, and combined its VERY hard in practice. The reason you do it is mostly DEVELOPER HAPPINESS and QUUICK TEST WRITTING/DEB
by majkinetor 5y ago
I never clear database, use fuzzy tests, and combined its VERY hard in practice. The reason you do it is mostly DEVELOPER HAPPINESS and QUUICK TEST WRITTING/DEBUGGING which I can't state enough how important it is.
So, integration tests. I have REST API which i test via PowerShell and pester so no transactions are possible. There are 2 options:
1. The database is dropped before each run and recreated via migrations and seed
2. The database is never dropped
3. This one possible to: run thousands of times then drop the database. It helps with randomization exhaustion of certain limited set of objects.
First step lasts from 20s to several minutes depending on the size of the db so its not that bad during CI/CD run, but its BAD during development because you can't just run single test 10 times quickly but you need to wait for DB cleaning in between. That clearly sux. Oracle db is the worst here.
Another thing that sux is once you have it running longer then 5 minutes, you again block development AND deployment. That sux big time. Now you want to parallelize which is mega problem since its ONE THING to run single test on database, and ANOTHER THING to run multiple tests on the single database in parallel. It can be done, but its better to forget this IMO and just make it in parallel on different databases (I did this one tests run on 10 servers each using its own db and taking set of suites of tests).
Now, WRITING TESTS LIKE THIS REQUIRES LOTS OF THOUGHT, REFACTORING OF BOTH TESTS AND BACKENDS, particularly combined with FUZZY tests (where in essence you randomize test input on each run)
Fuzzy tests with the non-dropping database are absolute astonishing both in its testing ability and its potential to produce flaky tests. Out of 10 failed tests, in my experience 1 is due to bug, 9 are due to flakiness of randomization and database state.
And all could be fixed. But it requires infrastrucutre, tools, and methodology.
1. I log each test run on InfluxDb and dashboard them on Grafana so I can compare months or runs and see what tests are flaky and if there are any other patterns.
2. I require from developers to NOT drop the database and use randomization for any input of almost ALL attributes it has.
3. I ask for them to run serially any test thousands times without ANY error (to prevent flaky tests). Then I ask them to run them in combination with all other tests (so they don't get dependent).
4. I have tools like complete db export when tests fail, verbose communication details with any touched REST endpoints, option to quickly run single or set of test in a loop, option to change tests before run without any particular toolset (i.e. notepad/vim).
This is VERY hard to do, honestly. But it totally rocks if you happen to do it right. We almost never had a bug found by a user.