3 ms·

I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. I also don'

by vmenge 3y ago

I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. I also don't think unit tests bring as much value as integration tests. In fact, a lot of times unit tests are IMO useless or just make your code harder to change. The more towards testing implementation the worse it gets IMO, unless I really really care that something is done in a very peculiar way, which is not very often. My opinion will be of course biased by my past experiences, but this has worked well for me so far with both monoliths and microservices, from e-shops and real estate marketplaces to IoTs.

solraph 3y ago
> I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. I've come around on this. I used to mock the DB, especially when it was being used as a dumb store, and now I just recreate the DB in SQLite, and see the DB as part of the system being tested, rather than something to mock. However, I think it's important to note that it wasn't until improved SQLite capabilities, SSDs (and sometimes docker if I really need postgres) all came together that this actually practical. Previously using an actual DB would have blown out my test runtimes by a factor of 10x. > I also don't think unit tests bring as much value as integration tests. In fact, a lot of times unit tests are IMO useless or just make your code harder to change. The more towards testing implementation the worse it gets IMO, unless I really really care that something is done in a very peculiar way, which is not very often. I see this a slightly different way. My concept of a unit (ignoring the article) has expanded to be what makes sense for a given test. This may be a class or set of classes where there's a well crafted set of inputs and outputs, but where there's a tricky set of inputs and outputs (anything involving date calculation for example) I'll often write a set of tests for just that function. I'd probably call all of these "unit tests" however. To me, an integration test involves testing that disparate vertical parts of a SUT work together. I haven't seen many of these in the wild.
icedchai 3y ago
I once worked at a place that demanded we write unit tests for every new method. Something that was simply a getter or setter? New unit test. I'd argue that the code was covered by tests on other code, where it was actually used. This would result in more and more useless arguments. Eventually, I just moved on. The company is no longer in business anyway.
- monksy 3y ago
  I follow the rule that if you have code that was written, it should be tested. > Something that was simply a getter or setter? New unit test. Bring in a framework to generate those. You shouldn't be writing them, and if you are it's because they're sepecial.
- sidlls 3y ago
  I'd argue if you have "getters and setters" you're already doing it wrong, because you're employing OOP. Just use a raw data structure and write functions that accept it as a parameter. Other things being equal, the code will be less complex, cleaner, and easier to test.
imiric 3y ago
> I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. They might not be slow individually, but if you have thousands of them, even a runtime of a couple of seconds adds up considerably. Especially if they're not parallelized, or parallelizable. Also, since they depend on an external service, they're tedious to execute, so now Docker becomes a requirement for every environment they run in, including slow CI machines. Then there is external state you need to think about, to ensure tests are isolated and don't clobber each other, expensive setup/teardown, ensuring that they cleanup after they're done, etc. It's all complexity that you don't have, or shouldn't have, with low-level tests. That's not to say that such tests shouldn't exist, of course, but that they shouldn't be the primary test type a project relies on. > I also don't think unit tests bring as much value as integration tests. In fact, a lot of times unit tests are IMO useless or just make your code harder to change. You're repeating the same argument as TFA, which is what I disagree with. IME I _much_ preferred working on codebases with a high coverage from low-level tests, than on those that mostly rely on higher level ones. This is because with more lower level tests there is a higher degree of confidence that a change won't inadvertently break something a higher level test is not accounting for. Yes, this means larger refactorings also means having to update your tests, but this is a trade-off worth making. Besides, nowadays it's becoming easier to just have an AI maintain tests for you, so this argument is quickly losing ground. > My opinion will be of course biased by my past experiences Sure, as all our opinions are, and this is fine. There is no golden rule that should be strictly followed about this, regardless of what some authority claims. I've also worked on codebases that use your approach, and it has worked "well" to some extent, but with more code coverage I always had more confidence in my work, and quicker and convenient test suites ensured that I would rely on this safety net more often.
- PH95VuimJjqBqy 3y ago
  unit vs integration is not low vs high level, that's a mischaracterization.
layer8 3y ago
Do you set up a new database schema for each unit test? If yes, that tends to be slow if you have many tests (hundreds or thousands), and if no, then you risk getting stateful dependencies between tests and they aren’t really unit tests anymore.
- solraph 3y ago
  Not OP, but I do this with SQLite in memory DB. If each unit only adds the tables the test needs, this is super fast, probably around 2ms per table.
  - imiric 3y ago
    Unless you use SQLite in production as well, this can give you a false sense of security, as there can be subtle behavior differences between DBMSs. Ideally you should always test with the same DBMS and same version as used in production. This is the same risk with mocking/stubbing, but for integration tests it's important that you're testing with the actual system used in production.
    - solraph 3y ago
      Yep. I have very much been bitten by these differences. Particularly for some of the funkier SQL statements where SQLite and PG handle arrays very differently. In our case, we decided it was a worthy trade off in terms of developer feedback times on our python microservices, because most of the SQL there is INSERT only. In our golang services which do 90% of the interesting SQL, we spin up PG in a docker container.
- TylerE 3y ago
  Don't need to do that, just run each test in it's own transaction, no?
  - layer8 3y ago
    How do you guarantee the database contents is in the right state for the given test? Dropping all tables and recreating them with the respective test data is the only sure way. And if you run tests in parallel, you need parallel database instances.

by vmenge 3y ago

solraph 3y ago

> I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. I've come around on this. I used to mock the DB, especially when it was being used as a dumb store, and now I just recreate the DB in SQLite, and see the DB as part of the system being tested, rather than something to mock. However, I think it's important to note that it wasn't until improved SQLite capabilities, SSDs (and sometimes docker if I really need postgres) all came together that this actually practical. Previously using an actual DB would have blown out my test runtimes by a factor of 10x. > I also don't think unit tests bring as much value as integration tests. In fact, a lot of times unit tests are IMO useless or just make your code harder to change. The more towards testing implementation the worse it gets IMO, unless I really really care that something is done in a very peculiar way, which is not very often. I see this a slightly different way. My concept of a unit (ignoring the article) has expanded to be what makes sense for a given test. This may be a class or set of classes where there's a well crafted set of inputs and outputs, but where there's a tricky set of inputs and outputs (anything involving date calculation for example) I'll often write a set of tests for just that function. I'd probably call all of these "unit tests" however. To me, an integration test involves testing that disparate vertical parts of a SUT work together. I haven't seen many of these in the wild.

icedchai 3y ago

I once worked at a place that demanded we write unit tests for every new method. Something that was simply a getter or setter? New unit test. I'd argue that the code was covered by tests on other code, where it was actually used. This would result in more and more useless arguments. Eventually, I just moved on. The company is no longer in business anyway.

monksy 3y ago
I follow the rule that if you have code that was written, it should be tested. > Something that was simply a getter or setter? New unit test. Bring in a framework to generate those. You shouldn't be writing them, and if you are it's because they're sepecial.
sidlls 3y ago
I'd argue if you have "getters and setters" you're already doing it wrong, because you're employing OOP. Just use a raw data structure and write functions that accept it as a parameter. Other things being equal, the code will be less complex, cleaner, and easier to test.

imiric 3y ago

> I've never had issues with integration tests running with real databases -- they never felt slow or incurred any significant amount of time for me. They might not be slow individually, but if you have thousands of them, even a runtime of a couple of seconds adds up considerably. Especially if they're not parallelized, or parallelizable. Also, since they depend on an external service, they're tedious to execute, so now Docker becomes a requirement for every environment they run in, including slow CI machines. Then there is external state you need to think about, to ensure tests are isolated and don't clobber each other, expensive setup/teardown, ensuring that they cleanup after they're done, etc. It's all complexity that you don't have, or shouldn't have, with low-level tests. That's not to say that such tests shouldn't exist, of course, but that they shouldn't be the primary test type a project relies on. > I also don't think unit tests bring as much value as integration tests. In fact, a lot of times unit tests are IMO useless or just make your code harder to change. You're repeating the same argument as TFA, which is what I disagree with. IME I _much_ preferred working on codebases with a high coverage from low-level tests, than on those that mostly rely on higher level ones. This is because with more lower level tests there is a higher degree of confidence that a change won't inadvertently break something a higher level test is not accounting for. Yes, this means larger refactorings also means having to update your tests, but this is a trade-off worth making. Besides, nowadays it's becoming easier to just have an AI maintain tests for you, so this argument is quickly losing ground. > My opinion will be of course biased by my past experiences Sure, as all our opinions are, and this is fine. There is no golden rule that should be strictly followed about this, regardless of what some authority claims. I've also worked on codebases that use your approach, and it has worked "well" to some extent, but with more code coverage I always had more confidence in my work, and quicker and convenient test suites ensured that I would rely on this safety net more often.

PH95VuimJjqBqy 3y ago
unit vs integration is not low vs high level, that's a mischaracterization.

layer8 3y ago

Do you set up a new database schema for each unit test? If yes, that tends to be slow if you have many tests (hundreds or thousands), and if no, then you risk getting stateful dependencies between tests and they aren’t really unit tests anymore.

solraph 3y ago
Not OP, but I do this with SQLite in memory DB. If each unit only adds the tables the test needs, this is super fast, probably around 2ms per table.
- imiric 3y ago
  Unless you use SQLite in production as well, this can give you a false sense of security, as there can be subtle behavior differences between DBMSs. Ideally you should always test with the same DBMS and same version as used in production. This is the same risk with mocking/stubbing, but for integration tests it's important that you're testing with the actual system used in production.
  - solraph 3y ago
    Yep. I have very much been bitten by these differences. Particularly for some of the funkier SQL statements where SQLite and PG handle arrays very differently. In our case, we decided it was a worthy trade off in terms of developer feedback times on our python microservices, because most of the SQL there is INSERT only. In our golang services which do 90% of the interesting SQL, we spin up PG in a docker container.
TylerE 3y ago
Don't need to do that, just run each test in it's own transaction, no?
- layer8 3y ago
  How do you guarantee the database contents is in the right state for the given test? Dropping all tables and recreating them with the respective test data is the only sure way. And if you run tests in parallel, you need parallel database instances.