6 ms·

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real ins

by mabbo 7y ago

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code may depend on an large, complex database. I could have a continually running instance just for my unit tests, but that's expensive (time, money, hosts, etc) and means I can't run my unit tests if the wifi goes down. If we're talking about large, complete systems running very large suites of system tests as part of a CI/CD pipeline, then sure, I'm on board. Let's use a real database. But there's lots of other small, simple tests that absolutely don't need anything more than a mock while I work on business logic bugs. And no, I'd not like to get into a debate of "you shouldn't bother to write small unit tests". I find them very useful.

amw-zero 7y ago
What I got out of the article was that, in unit tests, you should stub the response of a database query. However, you shouldn't directly assert that some database API method is called, because that is one additional level of coupling. What I don't understand is, a stub to me is almost the same level of coupling as a mock. Your test setup needs to know about the dependency on what you're stubbing. So when you want to change the interface of the dependency or use a different one, or remove the dependency, your tests have to change. It's not like you can add a query to an operation that didn't have one previously and the result will magically appear without connecting to a real database. So you have to modify the tests to provide the stub. Seems like a logical contradiction to me.
jaitsu 7y ago
Writing unit tests is debatable?
- ashtonkem 7y ago
  That argument gets floated occasionally. The utility of a unit test suite is roughly proportional to the amount of in-process computation you do. “Glue” systems which mostly transform from one protocol to another usually need more integration systems than unit, making skipping unit tests not disastrous.
- jerf 7y ago
  Oh, yes. We've still got a bunch of Real Programmers [1] out there. Real Programmers have moved on in my lifetime from insisting that everything should be written in assembly to insisting that they're Real Programmers enough to handle C directly and anyone who writes undefined behavior or buffer overflows or memory management issues just aren't Real Programmers enough and should put down their keyboards and walk away in shame, perhaps taking up goat farming or plumbing. In all seriousness, their numbers seem to be diminishing. But you certainly can't expect the Real Programmers to write test code. Why should a programmer write test code when they know they didn't write any bugs? [1]: http://catb.org/~esr/jargon/html/R/Real-Programmer.html http://catb.org/~esr/jargon/html/R/Real-Programmer.html - I'd say as time goes on, this term is relative.
- commandlinefan 7y ago
  On HN, it's not debatable: you should definitely do it. Everywhere I've ever worked, it's also not debatable: no time or effort should be "wasted" writing unit tests, that's what QA is for.
  - swat535 7y ago
    > no time or effort should be "wasted" writing unit tests, that's what QA is for. This is not my experience at all, everywhere I worked unit tests are required. I don't agree at all, writing unit tests is not a waste of time, this sentiment comes from people who don't know how to avoid brittle tests. It's impossible for QA to test every single path in your code. Unless you wanna cover all of those in your slow end to end tests?
    - commandlinefan 7y ago
      Oh, you're preaching to the choir here, pal. It's incredibly frustrating. I have reason to hope, though... when I first started programming in the 90's, I would fight with my co-workers about using version control (I insisted on it, they said it was a waste of time). Now, people who oppose version control are off selling life insurance or whatever happens to people who probably shouldn't be writing computer programs.
- Tomis02 7y ago
  Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.
  - allendoerfer 7y ago
    Writing getters and setters is harmf... - okay, I will stop before things escalate in here!
    - Tomis02 7y ago
      If you live in OOP land and have a natural aversion to directly accessing the data then by all means, write all the getters and setters you want. But testing that "int getX() { return X;}" actually returns X is pathological behaviour. It's harmful because 1) it costs to write 2) it costs to read 3) it takes up space, distracting from more important tests 4) it costs to run 5) prevents you from easily making changes (it's a brittle test). Can't really see any benefits; if you manage to screw up writing a dumb getter, how do you trust yourself that the test code isn't wrong as well?
  - packetlost 7y ago
    That's why I write large sweeping tests that make coverage metric to up, but don't take too much time to write or maintain /s
- uk_programmer 7y ago
  Yes. For a lot of smaller (web) applications where it is obvious if it isn't working correctly it just become a time sink. Everything in engineering is a set of trade-offs. With regards to TDD specifically. It isn't for everyone and many consider it to be a bit of a cult. While I don't consider it a cult, It doesn't work with how I personally solve problems. I normally for example get something extremely rough working and then iterate until I consider it to be perfect and then write my tests to define how it should behave.
- squiggleblaz 7y ago
  On legacy code? Alas. The amount of changes you might have to make to make the legacy code unit testable versus the benefit of unit tests can be exceedingly significant. For instance, if the implementation involves calling up a bunch of value objects from a database, each of which do the same, and all of the code is inhouse so there is no standard mock or stub libraries available, adding unit tests is tantamount to rewriting the whole system without tests. Existing codebases can also be too complicated for a few people to formalise into unit tests. The algorithm itself might be simple, but again, to discover that, you need to rewrite the whole system without tests. (You can add tests, of course, but when you're adding tests, you're making an assertion you cannot prove. Since you don't know what the code is meant to do, you don't know whether the tests are complete or even accurate.) Once you're coding in a world where tests passing or tests failing has almost no predictive value on success or failure in release, you're working in a self-fulfilling prophecy where the code will never be tested because the tests literally make things worse. Many codebases clearly do not have any automated tests at all, but unit tests can be the hardest to add onto a system after the fact.
bluejekyll 7y ago
Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.
- sanderjd 7y ago
  Do people usually mock the database at the integration / functional level? I haven't seen that... I interpreted this to mean the unit test level, which similarly to the thread starter, seems crazy to me.
  - bluejekyll 7y ago
    If you think of the DB like a service, then yes, people sometimes mock it. IMO, the danger comes when you start trying to mock things and act like a DB, i.e. you create some in-memory store to act like a DB. That's dangerous because then you're potentially introducing issues that are completely unrelated to the DB operations, and therefor not testing anything of value. But it can be far cheaper to generate data through a mocked interface, than say fill a DB with data, and test against that data-set. Obviously there are ways to structure your code such that the DB isn't part of the data flow at all, but sometimes existing code structure isn't perfect.
- pydry 7y ago
  However, the further down the pyramid you go: * The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all). * The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the database and shut it down is an investment cost that pays dividends. On the whole I think the pyramid idea enforces a wrong view that there is a "right" mix of "test levels" across any project. The best mix is determined by the kind of bugs and code you have (integration vs logical) and the kinds of abstractions you need or already have (in general, the worse your abstractions, the higher level you need your tests to be). A lot of projects are best done with 100% integration tests while others can be done with 100% unit (especially small, self contained, simple-to-interact-with code bases that are 99% about calculations/logical decision making).
grey-area 7y ago
When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate. Also, most unit tests don't need a db, it's more useful for integration tests.
- pc86 7y ago
  If you're using a Repository pattern or similar you'll have a lot of unit tests where a database or db provider is an injected dependency.
- amw-zero 7y ago
  Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.
  - grey-area 7y ago
    I use the db for some tests, it's totally fine, because it's limited to those tests which actually need db access (most don't). It's also closer to the actual application behaviour than mocks and doesn't need to be maintained.
davedx 7y ago
> Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? So what we do where I work is for unit tests, we mock services and repositories, unit tests don't go anywhere near the database. For integration tests, we use an in memory database. BUT! Be careful before you embark down the path of running integration tests against a different database than the one you use when running the application. There are SO many pitfalls, nasty bugs, and other warts along that road. EntityFramework alone has quite some weirdness there. Expect these kinds of integration tests to cost a lot of developer effort to build and maintain. For us, it took months of effort to get these kinds of tests working usefully. Personally after working on applications that have a solid test pyramid, I would recommend: * Write unit tests as per the standard advice (bottom of pyramid kind of quantity), but try and keep it sane (don't go for 90% coverage just for the sake of hitting an arbitrary number; don't pointlessly unit test your framework/libraries/other dependencies) * Write some integration tests where they really add value (interaction between 2 or a few complicated components in your system, for example places where the state of a component changes a lot depending on input from another component). Make sure when you start out writing an integration test that it doesn't turn into an "almost" end-to-end test along the way. They have a habit of doing this and it can really cost you later. Integration tests should still be focused. * Write end-to-end tests that test as much of your system as possible, including a database (preferably the same vendor; one approach is to truncate all tables before each run). IME it is very good here to have one e2e test that covers a big chunk of functionality in one run, than lots of separate tests covering different things. Why? Because no matter how "different" the things being tested by the e2e test there will still be LOTS of overlap by its very nature, and changing anything where that overlap is will tend to break ALL your e2e tests. Not a fun workflow. One last comment on E2E tests. This is more opinionated. But try and limit E2E tests to things that are really business critical (e.g. your "sign up" process and your "renew subscription" process, but not every single form in your dashboard). This is just a cost-benefit thing. E2E tests help, but they also slow you down, and the more you have, the more slowly you will be able to change your software. Sometimes minor bugs slipping through is an acceptable cost if it means you release 1 week sooner.
- matwood 7y ago
  > For integration tests, we use an in memory database. I think this can work for running the tests locally when speed is what you want, but you should still have the tests run on the actual database on commits. It requires keeping a database running just for tests to run (it can be a smaller instance and startup/shutdown on demand), but it will save so much pain later.
tybit 7y ago
A big part of the authors unit testing philosophy is that you should separate IO from business logic. That way you can unit test without mocks and without heavy real dependencies either, and leave that for integration tests.
je42 7y ago
yep. the test pyramid helps figuring when to use what test setup. https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html
BenoitP 7y ago
> should spin up an entire instance of Oracle I'm sort of doing that at the moment, with postgres and series of tests. But it is still useful for unit testing too: pull up the right CREATE tables, INSERT your test data, execute your tests, then drop tables (lots of safeguards here), and repeat. The container loads up in 1.5 sec, and all my tests (~100) are done in 10 seconds. It's been great in my use case. I'm on postgres, Java, and using testcontainers. They have handy containers available, not limited to RDBMS and not even limited to databases; and here is Oracle Express edition, which should be enough for most tests: https://www.testcontainers.org/modules/databases/oraclexe/ https://www.testcontainers.org/modules/databases/oraclexe/
stickfigure 7y ago
IMO, the classic test pyramid with integration tests at the narrow top and unit tests at the wide bottom is inverted. We test software that way because integration tests are hard and unit tests are easy, but integration tests provide more value than unit tests. It's the streetlight effect: "Did you lose your car keys here?" "No, but the light is much better here." I built some framework code so that integration tests as easy to write as unit tests. Almost all of my tests are "integration tests". I never mock the database. My test harness clones a template database at the start of each test; that database is maintained with migrations just like every other database. I run against test accounts at all the 3rd party services I use. It's not super fast - a full CI run takes about 15 minutes. And occasionally a 3rd party service will cause a test to flake. But it's fast enough and most importantly it's thorough. I still have some unit tests that run without the expensive setup harness, but they're for components that have a lot of algorithmic complexity. Anything that touches the database gets a real instance. This works pretty well with Postgres, which is free and I can run locally and has a fast db clone operation. YMMV with other databases.
- Tainnor 7y ago
  That's not the sole reason for avoiding too many integration tests. The main reason for that IMHO is combinatorial explosion of code paths which you'd have to test for if you wanted to be thorough.
regularfry 7y ago
I find it utterly bizarre that this should still be a current topic of contention. Running unit tests against a local database was a norm in Rails a decade ago, because the framework made it trivial to set up. Yes, there were knock-on performance questions that wanted answering - how do you avoid the database setup costs for those tests that don't care about it, for instance, the "fast Rails test" movement was a big thing - but by and large those were solved problems by the time I stopped writing Rails code professionally around the 5.1 era. The answer is, of course, no, you don't spin up an entire instance of Oracle to run a single unit test. You run against a local instance, and you use whatever tricks your-RDBMS-of-choice gives you to make resetting the test tables to a known-good state extremely fast. That way you can have your tests running continuously, giving you fast feedback as you develop, only suffering db overheads when you actually need to run tests that hit it. If your chosen stack makes it difficult to do this, it's worth asking why.
lifeisstillgood 7y ago
If I am honest, at this point I would rather not use a mock, but a stub / injection. I know this gets horrbily pedantic but its easier to see in code def difficult_function(dbconn, a,b,c): dbconn.execute(Select * from tbl) <comlicated stuff invlvoing results set and a,b,c> I would not want to mock the dbase at this point. Please can we instead do this def difficult_function(dbconn, a,b,c): resultset_as_dict = dbconn.execute(select * from tbl) insider_function(a,b,c,results_set_as_dict) def insider_function(a,b,c,results_set_as_dict): This can now be tested without mocks quite easily. I think if you are doing 'difficult' stuff with a exernal database you are de facto, writing integration tests. In fact i would say anything involving a database MUST be treated as a integration test. If it takes ten minutes thats fine - its an integration test. If you want fast and external connections, use sqllite as part of your testing CI suite. But dont moan. and do not use Mocks.

by mabbo 7y ago

amw-zero 7y ago

What I got out of the article was that, in unit tests, you should stub the response of a database query. However, you shouldn't directly assert that some database API method is called, because that is one additional level of coupling. What I don't understand is, a stub to me is almost the same level of coupling as a mock. Your test setup needs to know about the dependency on what you're stubbing. So when you want to change the interface of the dependency or use a different one, or remove the dependency, your tests have to change. It's not like you can add a query to an operation that didn't have one previously and the result will magically appear without connecting to a real database. So you have to modify the tests to provide the stub. Seems like a logical contradiction to me.

jaitsu 7y ago

Writing unit tests is debatable?

ashtonkem 7y ago
That argument gets floated occasionally. The utility of a unit test suite is roughly proportional to the amount of in-process computation you do. “Glue” systems which mostly transform from one protocol to another usually need more integration systems than unit, making skipping unit tests not disastrous.
jerf 7y ago
Oh, yes. We've still got a bunch of Real Programmers [1] out there. Real Programmers have moved on in my lifetime from insisting that everything should be written in assembly to insisting that they're Real Programmers enough to handle C directly and anyone who writes undefined behavior or buffer overflows or memory management issues just aren't Real Programmers enough and should put down their keyboards and walk away in shame, perhaps taking up goat farming or plumbing. In all seriousness, their numbers seem to be diminishing. But you certainly can't expect the Real Programmers to write test code. Why should a programmer write test code when they know they didn't write any bugs? [1]: http://catb.org/~esr/jargon/html/R/Real-Programmer.html http://catb.org/~esr/jargon/html/R/Real-Programmer.html - I'd say as time goes on, this term is relative.
commandlinefan 7y ago
On HN, it's not debatable: you should definitely do it. Everywhere I've ever worked, it's also not debatable: no time or effort should be "wasted" writing unit tests, that's what QA is for.
- swat535 7y ago
  > no time or effort should be "wasted" writing unit tests, that's what QA is for. This is not my experience at all, everywhere I worked unit tests are required. I don't agree at all, writing unit tests is not a waste of time, this sentiment comes from people who don't know how to avoid brittle tests. It's impossible for QA to test every single path in your code. Unless you wanna cover all of those in your slow end to end tests?
  - commandlinefan 7y ago
    Oh, you're preaching to the choir here, pal. It's incredibly frustrating. I have reason to hope, though... when I first started programming in the 90's, I would fight with my co-workers about using version control (I insisted on it, they said it was a waste of time). Now, people who oppose version control are off selling life insurance or whatever happens to people who probably shouldn't be writing computer programs.
Tomis02 7y ago
Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.
- allendoerfer 7y ago
  Writing getters and setters is harmf... - okay, I will stop before things escalate in here!
  - Tomis02 7y ago
    If you live in OOP land and have a natural aversion to directly accessing the data then by all means, write all the getters and setters you want. But testing that "int getX() { return X;}" actually returns X is pathological behaviour. It's harmful because 1) it costs to write 2) it costs to read 3) it takes up space, distracting from more important tests 4) it costs to run 5) prevents you from easily making changes (it's a brittle test). Can't really see any benefits; if you manage to screw up writing a dumb getter, how do you trust yourself that the test code isn't wrong as well?
- packetlost 7y ago
  That's why I write large sweeping tests that make coverage metric to up, but don't take too much time to write or maintain /s
uk_programmer 7y ago
Yes. For a lot of smaller (web) applications where it is obvious if it isn't working correctly it just become a time sink. Everything in engineering is a set of trade-offs. With regards to TDD specifically. It isn't for everyone and many consider it to be a bit of a cult. While I don't consider it a cult, It doesn't work with how I personally solve problems. I normally for example get something extremely rough working and then iterate until I consider it to be perfect and then write my tests to define how it should behave.
squiggleblaz 7y ago
On legacy code? Alas. The amount of changes you might have to make to make the legacy code unit testable versus the benefit of unit tests can be exceedingly significant. For instance, if the implementation involves calling up a bunch of value objects from a database, each of which do the same, and all of the code is inhouse so there is no standard mock or stub libraries available, adding unit tests is tantamount to rewriting the whole system without tests. Existing codebases can also be too complicated for a few people to formalise into unit tests. The algorithm itself might be simple, but again, to discover that, you need to rewrite the whole system without tests. (You can add tests, of course, but when you're adding tests, you're making an assertion you cannot prove. Since you don't know what the code is meant to do, you don't know whether the tests are complete or even accurate.) Once you're coding in a world where tests passing or tests failing has almost no predictive value on success or failure in release, you're working in a self-fulfilling prophecy where the code will never be tested because the tests literally make things worse. Many codebases clearly do not have any automated tests at all, but unit tests can be the hardest to add onto a system after the fact.

bluejekyll 7y ago

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.

sanderjd 7y ago
Do people usually mock the database at the integration / functional level? I haven't seen that... I interpreted this to mean the unit test level, which similarly to the thread starter, seems crazy to me.
- bluejekyll 7y ago
  If you think of the DB like a service, then yes, people sometimes mock it. IMO, the danger comes when you start trying to mock things and act like a DB, i.e. you create some in-memory store to act like a DB. That's dangerous because then you're potentially introducing issues that are completely unrelated to the DB operations, and therefor not testing anything of value. But it can be far cheaper to generate data through a mocked interface, than say fill a DB with data, and test against that data-set. Obviously there are ways to structure your code such that the DB isn't part of the data flow at all, but sometimes existing code structure isn't perfect.
pydry 7y ago
However, the further down the pyramid you go: * The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all). * The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the database and shut it down is an investment cost that pays dividends. On the whole I think the pyramid idea enforces a wrong view that there is a "right" mix of "test levels" across any project. The best mix is determined by the kind of bugs and code you have (integration vs logical) and the kinds of abstractions you need or already have (in general, the worse your abstractions, the higher level you need your tests to be). A lot of projects are best done with 100% integration tests while others can be done with 100% unit (especially small, self contained, simple-to-interact-with code bases that are 99% about calculations/logical decision making).

grey-area 7y ago

When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate. Also, most unit tests don't need a db, it's more useful for integration tests.

pc86 7y ago
If you're using a Repository pattern or similar you'll have a lot of unit tests where a database or db provider is an injected dependency.
amw-zero 7y ago
Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.
- grey-area 7y ago
  I use the db for some tests, it's totally fine, because it's limited to those tests which actually need db access (most don't). It's also closer to the actual application behaviour than mocks and doesn't need to be maintained.

davedx 7y ago

> Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? So what we do where I work is for unit tests, we mock services and repositories, unit tests don't go anywhere near the database. For integration tests, we use an in memory database. BUT! Be careful before you embark down the path of running integration tests against a different database than the one you use when running the application. There are SO many pitfalls, nasty bugs, and other warts along that road. EntityFramework alone has quite some weirdness there. Expect these kinds of integration tests to cost a lot of developer effort to build and maintain. For us, it took months of effort to get these kinds of tests working usefully. Personally after working on applications that have a solid test pyramid, I would recommend: * Write unit tests as per the standard advice (bottom of pyramid kind of quantity), but try and keep it sane (don't go for 90% coverage just for the sake of hitting an arbitrary number; don't pointlessly unit test your framework/libraries/other dependencies) * Write some integration tests where they really add value (interaction between 2 or a few complicated components in your system, for example places where the state of a component changes a lot depending on input from another component). Make sure when you start out writing an integration test that it doesn't turn into an "almost" end-to-end test along the way. They have a habit of doing this and it can really cost you later. Integration tests should still be focused. * Write end-to-end tests that test as much of your system as possible, including a database (preferably the same vendor; one approach is to truncate all tables before each run). IME it is very good here to have one e2e test that covers a big chunk of functionality in one run, than lots of separate tests covering different things. Why? Because no matter how "different" the things being tested by the e2e test there will still be LOTS of overlap by its very nature, and changing anything where that overlap is will tend to break ALL your e2e tests. Not a fun workflow. One last comment on E2E tests. This is more opinionated. But try and limit E2E tests to things that are really business critical (e.g. your "sign up" process and your "renew subscription" process, but not every single form in your dashboard). This is just a cost-benefit thing. E2E tests help, but they also slow you down, and the more you have, the more slowly you will be able to change your software. Sometimes minor bugs slipping through is an acceptable cost if it means you release 1 week sooner.

matwood 7y ago
> For integration tests, we use an in memory database. I think this can work for running the tests locally when speed is what you want, but you should still have the tests run on the actual database on commits. It requires keeping a database running just for tests to run (it can be a smaller instance and startup/shutdown on demand), but it will save so much pain later.

tybit 7y ago

A big part of the authors unit testing philosophy is that you should separate IO from business logic. That way you can unit test without mocks and without heavy real dependencies either, and leave that for integration tests.

je42 7y ago

yep. the test pyramid helps figuring when to use what test setup. https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html

BenoitP 7y ago

> should spin up an entire instance of Oracle I'm sort of doing that at the moment, with postgres and series of tests. But it is still useful for unit testing too: pull up the right CREATE tables, INSERT your test data, execute your tests, then drop tables (lots of safeguards here), and repeat. The container loads up in 1.5 sec, and all my tests (~100) are done in 10 seconds. It's been great in my use case. I'm on postgres, Java, and using testcontainers. They have handy containers available, not limited to RDBMS and not even limited to databases; and here is Oracle Express edition, which should be enough for most tests: https://www.testcontainers.org/modules/databases/oraclexe/ https://www.testcontainers.org/modules/databases/oraclexe/

stickfigure 7y ago

IMO, the classic test pyramid with integration tests at the narrow top and unit tests at the wide bottom is inverted. We test software that way because integration tests are hard and unit tests are easy, but integration tests provide more value than unit tests. It's the streetlight effect: "Did you lose your car keys here?" "No, but the light is much better here." I built some framework code so that integration tests as easy to write as unit tests. Almost all of my tests are "integration tests". I never mock the database. My test harness clones a template database at the start of each test; that database is maintained with migrations just like every other database. I run against test accounts at all the 3rd party services I use. It's not super fast - a full CI run takes about 15 minutes. And occasionally a 3rd party service will cause a test to flake. But it's fast enough and most importantly it's thorough. I still have some unit tests that run without the expensive setup harness, but they're for components that have a lot of algorithmic complexity. Anything that touches the database gets a real instance. This works pretty well with Postgres, which is free and I can run locally and has a fast db clone operation. YMMV with other databases.

Tainnor 7y ago
That's not the sole reason for avoiding too many integration tests. The main reason for that IMHO is combinatorial explosion of code paths which you'd have to test for if you wanted to be thorough.

regularfry 7y ago

I find it utterly bizarre that this should still be a current topic of contention. Running unit tests against a local database was a norm in Rails a decade ago, because the framework made it trivial to set up. Yes, there were knock-on performance questions that wanted answering - how do you avoid the database setup costs for those tests that don't care about it, for instance, the "fast Rails test" movement was a big thing - but by and large those were solved problems by the time I stopped writing Rails code professionally around the 5.1 era. The answer is, of course, no, you don't spin up an entire instance of Oracle to run a single unit test. You run against a local instance, and you use whatever tricks your-RDBMS-of-choice gives you to make resetting the test tables to a known-good state extremely fast. That way you can have your tests running continuously, giving you fast feedback as you develop, only suffering db overheads when you actually need to run tests that hit it. If your chosen stack makes it difficult to do this, it's worth asking why.

lifeisstillgood 7y ago

If I am honest, at this point I would rather not use a mock, but a stub / injection. I know this gets horrbily pedantic but its easier to see in code def difficult_function(dbconn, a,b,c): dbconn.execute(Select * from tbl) <comlicated stuff invlvoing results set and a,b,c> I would not want to mock the dbase at this point. Please can we instead do this def difficult_function(dbconn, a,b,c): resultset_as_dict = dbconn.execute(select * from tbl) insider_function(a,b,c,results_set_as_dict) def insider_function(a,b,c,results_set_as_dict): This can now be tested without mocks quite easily. I think if you are doing 'difficult' stuff with a exernal database you are de facto, writing integration tests. In fact i would say anything involving a database MUST be treated as a integration test. If it takes ten minutes thats fine - its an integration test. If you want fast and external connections, use sqllite as part of your testing CI suite. But dont moan. and do not use Mocks.