4 ms·

> Dirty databases make the tests non-deterministic. If this is the case, is the test really testing something useful? Tests should verify that the code works a

> Dirty databases make the tests non-deterministic. If this is the case, is the test really testing something useful? Tests should verify that the code works as it's intended to work. If pre-existing (but presumably valid) data can break the test, then surely there's states where whatever behaviour is being tested won't work in production? Provided the test is creating and verifying its own data, it's not really non-deterministic except in a very technical sense.

bsaul 5y ago
I personnally use tests for two things : - in tdd, to make sure my code works in expected scenarios - in CI to make sure new code don't break old features. In both case, i need to control my testing environment. Randomizing inputs could be used in a third scenario for extensive testing, a bit like fuzzing. But it's yet another different usage.
- ryanbrunner 5y ago
  I guess the disconnect is that I don't think of the state of the database as inputs, per se, but more the environment the test is running in. I wouldn't for instance necessarily need to set my clock to a fixed time (provided I'm not specifically testing code where that is relevant), or wipe out the hard drive that the code is running on for a test.
  - deleted 5y ago
    [deleted]
michaelt 5y ago
> If pre-existing (but presumably valid) data can break the test, then surely there's states where whatever behaviour is being tested won't work in production? Imagine I'm testing a web page that shows a list of all customers who have placed at least one order. If I set up the test by wiping the database and adding ten customers, I can assert that the list of customers should contain those ten and no others, giving it ten rows. If I don't wipe the database, the web page may show more than ten customers - and I can't tell if the extra entries indicate a fault in my code or not.
- ryanbrunner 5y ago
  Sure, but in that case I wouldn't test that the list contains exactly the 10 customers I expect, since that's not what the code is supposed to do. That's a shortcut you introduced while testing. I usually approach things like this with two assertions - that the size of the list under test increased by X, and that the newly created record(s) exist in the list. After all, the code doesn't show the 10 most recent customers that purchased an item, it shows ALL customers. You could hide a pagination bug where only the first page of results functions if you wipe the database first.
  - sseagull 5y ago
    But then how do you deal with unique constraints or things like upserts? If I want to add 10 entries, then I need to know that all have been added and not pre-existing, or else the code for adding new entries doesn't actually get tested. Or you end up with spurious test errors if insertion fails due to duplicates.
    - dragonwriter 5y ago
      > But then how do you deal with unique constraints or things like upserts? If I want to add 10 entries, then I need to know that all have been added and not pre-existing, or else the code for adding new entries doesn't actually get tested. Then your test code needs to take into account the current state of the database when creating input data for the code under test. It is more complicated than with a know start state where you can pick fixed data, sure, but it's doable.
      - atom_arranger 5y ago
        If you want to test that “it adds X entries” then write the test to add X entries from a clean state, then add another X entries and check it is done correctly. You don’t need to rely on a dirty start state to do that.
      - kazinator 5y ago
        > Then your test code needs to take into account the current state ... [ etc ] ... and you're on your way to writing an application, not a test! Tests should be as simple as possible, and a clean slate goes a long way toward that. A test should be riddled with as few "needs to do" responsibilities as possible while still achieving the intended coverage.
kazinator 5y ago
What if the test thinks it is creating its own data, but that has failed, and the old data is mistaken for new data?

bsaul 5y ago

I personnally use tests for two things : - in tdd, to make sure my code works in expected scenarios - in CI to make sure new code don't break old features. In both case, i need to control my testing environment. Randomizing inputs could be used in a third scenario for extensive testing, a bit like fuzzing. But it's yet another different usage.

ryanbrunner 5y ago
I guess the disconnect is that I don't think of the state of the database as inputs, per se, but more the environment the test is running in. I wouldn't for instance necessarily need to set my clock to a fixed time (provided I'm not specifically testing code where that is relevant), or wipe out the hard drive that the code is running on for a test.
- deleted 5y ago
  [deleted]

michaelt 5y ago

> If pre-existing (but presumably valid) data can break the test, then surely there's states where whatever behaviour is being tested won't work in production? Imagine I'm testing a web page that shows a list of all customers who have placed at least one order. If I set up the test by wiping the database and adding ten customers, I can assert that the list of customers should contain those ten and no others, giving it ten rows. If I don't wipe the database, the web page may show more than ten customers - and I can't tell if the extra entries indicate a fault in my code or not.

ryanbrunner 5y ago
Sure, but in that case I wouldn't test that the list contains exactly the 10 customers I expect, since that's not what the code is supposed to do. That's a shortcut you introduced while testing. I usually approach things like this with two assertions - that the size of the list under test increased by X, and that the newly created record(s) exist in the list. After all, the code doesn't show the 10 most recent customers that purchased an item, it shows ALL customers. You could hide a pagination bug where only the first page of results functions if you wipe the database first.
- sseagull 5y ago
  But then how do you deal with unique constraints or things like upserts? If I want to add 10 entries, then I need to know that all have been added and not pre-existing, or else the code for adding new entries doesn't actually get tested. Or you end up with spurious test errors if insertion fails due to duplicates.
  - dragonwriter 5y ago
    > But then how do you deal with unique constraints or things like upserts? If I want to add 10 entries, then I need to know that all have been added and not pre-existing, or else the code for adding new entries doesn't actually get tested. Then your test code needs to take into account the current state of the database when creating input data for the code under test. It is more complicated than with a know start state where you can pick fixed data, sure, but it's doable.
    - atom_arranger 5y ago
      If you want to test that “it adds X entries” then write the test to add X entries from a clean state, then add another X entries and check it is done correctly. You don’t need to rely on a dirty start state to do that.
    - kazinator 5y ago
      > Then your test code needs to take into account the current state ... [ etc ] ... and you're on your way to writing an application, not a test! Tests should be as simple as possible, and a clean slate goes a long way toward that. A test should be riddled with as few "needs to do" responsibilities as possible while still achieving the intended coverage.

kazinator 5y ago

What if the test thinks it is creating its own data, but that has failed, and the old data is mistaken for new data?