5 ms·

Unit tests should be small, specific enough tests that they should not ever require using the database. End-to-end tests, behavioral tests, integration tests h

by hk1337 5y ago

Unit tests should be small, specific enough tests that they should not ever require using the database. End-to-end tests, behavioral tests, integration tests however can involve the database as you can actually test how the application responds to requests. I don’t know if it’s laziness, IDGAF, ignorance, stubbornness, or what but the above tests seemed to have merged recently where we now have “unit tests” that are actually behavior tests.

dragonwriter 5y ago
> Unit tests should be small, specific enough tests that they should not ever require using the database. That's somewhat architecture dependent; in a microservice architecture, it can be quite reasonable to see the service boundary as the unit boundary, in which case a DB-involved “unit test” makes some sense. You probably still also want fairly complete DB-independent tests that you can run automatically as you code, but the idea that the latter fully encompass unit tests is, I think, an artifact of predominantly monolithic architecture of the time the language developed, where you’d have multiple logical units in monolithic app that would then connect to a database (which even the monolith might not fully own).
- goblin89 5y ago
  The “unit” in “unit testing” is unambiguous across the industry, referring to a function or a class that you want to verify the implementation of. Trying to impose another meaning onto the word is a sure way to complicate knowledge exchange with unnecessary misunderstanding (I certainly would think twice about joining a team that invents its own bespoke language to reference common concepts). Testing a unit of any complexity is already a can of worms; with multitudes of factors like hardware conditions, memory state, or phases of the moon causing Heisenbugs what we need is more predictability, not less. So if some runtime quirk at the database server, unavailable connection, etc. has the power to fail your tests, call those tests “integration”, “component”, “service”, “behavior”, “e2e”, anything else—the available vocabulary is rich enough that it doesn’t really call for overloading existing designations.
cwbrandsma 5y ago
Unit Test is the common generic term anymore. So Unit test is most commonly used to refer to any type of test, integration, end to end, or isolation. Most people will view you as being pedantic and old school for insisting on using an older definition. Which is exactly the feedback I kept running into so I changed my stance on the subject to fit what people expected. At least I got them to make more tests.
- hk1337 5y ago
  Yeah, I wondered about that too. Lumping it under one term is imprecise. I don’t like that there’s 3 different types of non unit tests, personally I would say pick one and run with it so you’d say we have unit tests and behavior tests for example.
- Supermancho 5y ago
  > So Unit test is most commonly used to refer to any type of test, integration, end to end, or isolation. Most people will view you as being pedantic and old school for insisting on using an older definition. I understand that anything can be categorized as a unit, even a functional test. That's fine. Transactions for integration tests. I'll suffer the distain of people who rather produce problematic (eventually buggy) software in lieu of understanding categoric differences and purpose, leading to these kinds of "questions". If that's "most people", so be it.
- monksy 5y ago
  Calling a mix of tests into the category of unit tests is incredibly wrong and misguided. It is a formal term that has been around for a long time and talks about a test that is extremely limited in scope and is tested in isolation.
- morpheuskafka 5y ago
  The old definition is still how we learned it in undergrad software engineering (taught by the youngest professor in the department, no less) in 2021. Integrating into a database, much less one with simulated real-world data, would be integration or end-to-end testing, definitely not unit. The name unit implies testing a single unit of code, not how everything runs in real world conditions.
icedchai 5y ago
As usual, "it depends." For every person complaining about unit tests using databases, there's another complaining about all the annoying mocks they need to maintain, when they could just use the real thing.
101011 5y ago
I respectfully disagree. I have found great value out of database tests that did not require the overhead of end to end tests, which I've found can be brittle and slow - especially if your service depends in some way on another application or outside piece of infrastructure. As an aside, anytime I hear "not ever" from somebody wrt design patterns, my doubt sensors jump to alert.
vlovich123 5y ago
Because people hear “unit tests” and “integration test” and “behavior test” and bend over backwards to start DI and mock every single little thing and be very careful about organizing things in that way, which is a huge anti pattern in my experience. You should DI/mock only when you have no other choice (eg you need to corrupt the data returned by an I/O call to test some corner condition). Generally, the less DI/mocking/supporting code just for testing purposes you have, the simpler and more testable (ironically) your codebase actually is. The reason is that you’re actually testing the behavior of each function in a wider variety of test cases than just the ones written for that one function. You use fuzzers or property tests to generate a wide variety of test cases. The purists will argue that this makes tests brittle because changing a core routing can suddenly cause many tests to fail/ need updating. In practice I’ve observed that such updates are typically trivial and mechanical while with mocks you end up doing the same thing anyway (change an interface and suddenly you have a bunch of mocks that need updating anyway). Mocks and DI can be useful. Don’t get me wrong. But use them sparingly and judiciously when they’re the right tool for the job. Recognize that they’re usually not.
- the_gipsy 5y ago
  Very insightful. I've found the same about "using DI only to force errors / corrupt data on IO".
- throwaway858 5y ago
  Agreed. If you have a function that executes an SQL query then your unit test(s) of this function should also execute the query. Otherwise, what exactly are you testing? Just because the database runs as a separate process does not mean that this is an "integration" test. Imagine if you had a function that did some floating point math and then you wanted to test it and I told you "hang on, this function actually utilizes the FPU which is a separate piece of hardware and therefore if you test it then you're doing an integration test.. if you want a proper unit test then you must mock the FPU and use dependency injection". Madness. Executing a database query is no different from executing a CPU instruction or calling a library function.
cabalamat 5y ago
> Unit tests should be small, specific enough tests that they should not ever require using the database. What if the code you're testing is an ORM, or something else that intrinsically implies a database? It seems to me that tests that're good enough that they'll catch a decent proportion of errors would involve a database here. Of course, you might designate such tests "integration tests" rather than "unit tests", but I think the important thing about tests is not what you call them, it's whether they catch bugs.
- robertlagrant 5y ago
  What you call them is also important. Otherwise people won't know what you're talking about. You might say "it's better to do integration tests with SQLite than unit tests", but that's a different question.
  - cabalamat 5y ago
    > Otherwise people won't know what you're talking about. That's a good point!
patrec 5y ago
> Unit tests should be small, specific enough tests that they should not ever require using the database. > I don’t know if it’s laziness, IDGAF, ignorance, stubbornness, [...] My impression is that most people who think along these lines just lack the technical ability to make tests that involve the database fast (or maybe even robust), or just parrot something they heard. If you want to test a service that requires DB access for most things it does as part of its core logic, why either exclusively rely on more difficult to debug end-to-end tests, or put in extra effort and bloat for worse tests with nonsense like mocking when you can just give it a database in a way that adds basically no discernible overhead to your test run?

by hk1337 5y ago

dragonwriter 5y ago

> Unit tests should be small, specific enough tests that they should not ever require using the database. That's somewhat architecture dependent; in a microservice architecture, it can be quite reasonable to see the service boundary as the unit boundary, in which case a DB-involved “unit test” makes some sense. You probably still also want fairly complete DB-independent tests that you can run automatically as you code, but the idea that the latter fully encompass unit tests is, I think, an artifact of predominantly monolithic architecture of the time the language developed, where you’d have multiple logical units in monolithic app that would then connect to a database (which even the monolith might not fully own).

goblin89 5y ago
The “unit” in “unit testing” is unambiguous across the industry, referring to a function or a class that you want to verify the implementation of. Trying to impose another meaning onto the word is a sure way to complicate knowledge exchange with unnecessary misunderstanding (I certainly would think twice about joining a team that invents its own bespoke language to reference common concepts). Testing a unit of any complexity is already a can of worms; with multitudes of factors like hardware conditions, memory state, or phases of the moon causing Heisenbugs what we need is more predictability, not less. So if some runtime quirk at the database server, unavailable connection, etc. has the power to fail your tests, call those tests “integration”, “component”, “service”, “behavior”, “e2e”, anything else—the available vocabulary is rich enough that it doesn’t really call for overloading existing designations.

cwbrandsma 5y ago

Unit Test is the common generic term anymore. So Unit test is most commonly used to refer to any type of test, integration, end to end, or isolation. Most people will view you as being pedantic and old school for insisting on using an older definition. Which is exactly the feedback I kept running into so I changed my stance on the subject to fit what people expected. At least I got them to make more tests.

hk1337 5y ago
Yeah, I wondered about that too. Lumping it under one term is imprecise. I don’t like that there’s 3 different types of non unit tests, personally I would say pick one and run with it so you’d say we have unit tests and behavior tests for example.
Supermancho 5y ago
> So Unit test is most commonly used to refer to any type of test, integration, end to end, or isolation. Most people will view you as being pedantic and old school for insisting on using an older definition. I understand that anything can be categorized as a unit, even a functional test. That's fine. Transactions for integration tests. I'll suffer the distain of people who rather produce problematic (eventually buggy) software in lieu of understanding categoric differences and purpose, leading to these kinds of "questions". If that's "most people", so be it.
monksy 5y ago
Calling a mix of tests into the category of unit tests is incredibly wrong and misguided. It is a formal term that has been around for a long time and talks about a test that is extremely limited in scope and is tested in isolation.
morpheuskafka 5y ago
The old definition is still how we learned it in undergrad software engineering (taught by the youngest professor in the department, no less) in 2021. Integrating into a database, much less one with simulated real-world data, would be integration or end-to-end testing, definitely not unit. The name unit implies testing a single unit of code, not how everything runs in real world conditions.

icedchai 5y ago

As usual, "it depends." For every person complaining about unit tests using databases, there's another complaining about all the annoying mocks they need to maintain, when they could just use the real thing.

101011 5y ago

I respectfully disagree. I have found great value out of database tests that did not require the overhead of end to end tests, which I've found can be brittle and slow - especially if your service depends in some way on another application or outside piece of infrastructure. As an aside, anytime I hear "not ever" from somebody wrt design patterns, my doubt sensors jump to alert.

vlovich123 5y ago

Because people hear “unit tests” and “integration test” and “behavior test” and bend over backwards to start DI and mock every single little thing and be very careful about organizing things in that way, which is a huge anti pattern in my experience. You should DI/mock only when you have no other choice (eg you need to corrupt the data returned by an I/O call to test some corner condition). Generally, the less DI/mocking/supporting code just for testing purposes you have, the simpler and more testable (ironically) your codebase actually is. The reason is that you’re actually testing the behavior of each function in a wider variety of test cases than just the ones written for that one function. You use fuzzers or property tests to generate a wide variety of test cases. The purists will argue that this makes tests brittle because changing a core routing can suddenly cause many tests to fail/ need updating. In practice I’ve observed that such updates are typically trivial and mechanical while with mocks you end up doing the same thing anyway (change an interface and suddenly you have a bunch of mocks that need updating anyway). Mocks and DI can be useful. Don’t get me wrong. But use them sparingly and judiciously when they’re the right tool for the job. Recognize that they’re usually not.

the_gipsy 5y ago
Very insightful. I've found the same about "using DI only to force errors / corrupt data on IO".
throwaway858 5y ago
Agreed. If you have a function that executes an SQL query then your unit test(s) of this function should also execute the query. Otherwise, what exactly are you testing? Just because the database runs as a separate process does not mean that this is an "integration" test. Imagine if you had a function that did some floating point math and then you wanted to test it and I told you "hang on, this function actually utilizes the FPU which is a separate piece of hardware and therefore if you test it then you're doing an integration test.. if you want a proper unit test then you must mock the FPU and use dependency injection". Madness. Executing a database query is no different from executing a CPU instruction or calling a library function.

cabalamat 5y ago

> Unit tests should be small, specific enough tests that they should not ever require using the database. What if the code you're testing is an ORM, or something else that intrinsically implies a database? It seems to me that tests that're good enough that they'll catch a decent proportion of errors would involve a database here. Of course, you might designate such tests "integration tests" rather than "unit tests", but I think the important thing about tests is not what you call them, it's whether they catch bugs.

robertlagrant 5y ago
What you call them is also important. Otherwise people won't know what you're talking about. You might say "it's better to do integration tests with SQLite than unit tests", but that's a different question.
- cabalamat 5y ago
  > Otherwise people won't know what you're talking about. That's a good point!

patrec 5y ago

> Unit tests should be small, specific enough tests that they should not ever require using the database. > I don’t know if it’s laziness, IDGAF, ignorance, stubbornness, [...] My impression is that most people who think along these lines just lack the technical ability to make tests that involve the database fast (or maybe even robust), or just parrot something they heard. If you want to test a service that requires DB access for most things it does as part of its core logic, why either exclusively rely on more difficult to debug end-to-end tests, or put in extra effort and bloat for worse tests with nonsense like mocking when you can just give it a database in a way that adds basically no discernible overhead to your test run?