6 ms·

With the advent of easy to use docker containers, this is really easy to do for most backing services, right on your machine during tests (and in CI). If all y

• okamiueru 6y ago
  I agree with most of what you write, and I use test fixtures that spin up real services with docker for running integration tests. However, I'd argue that mocking will always have a usefulness, as it goes hand in hand with unit tests. Integration tests confirms the behaviour for the code branching you are able to reach that way, and more closely reflects reality and normal operations. Mocking allows you to more easily test the edge cases that the software unit might encounter that are hard to create conditions for otherwise. In fact, from also being tempted by what you write (ie. why mock, when you can automatically spin up a docker service). I've done this too, and my conclusion is that it is more trouble than it's worth, although that can differ for other systems, I suppose. The only realistic approach is that a test fixture spins services up once, and not per test/unit, so it just introduced a concern for correct clean and concurrency limitations that wouldn't exist. It also has unexpected issues with wildly varying startup times whenever or not the container image is cached. All in all, it works well enough now, but when I do it all over somewhere, I'm tempted to keep a clean distinction of deploying temporary services for running integration tests, and regular unit tests, without the in between. TL;DR: mocking isn't just for "i need service A in my test", but also "I need A to behave like X", and don't get that with external services.
  • hardwaresofton 6y ago
    Just want to make sure, but it seems like I can sum up your reasons for mocking are: - spinning up the service is hard - startup times can be inconsistent I may have not read deep enough but I don't see much explanation around the "I need A to behave like X" -- could you give a more concrete example? Lets say you had two queue implementations and one used Redis and one used RabbitMQ (i.e `GenericQueue<Redis|Rabbit>`), it seems ridiculous to need one of them to act like the other -- I'd love an example if you have one in mind. Optimizing spinning up the service and making startup times consistent are basically solved in half a day of work (that pays off forever), in my experience. Here's an excerpt from a Gitlab CI config I carry around from project-to-project: e2e: stage: test only: - merge_requests services: - docker:dind variables: DOCKER_HOST: tcp://docker:2376 # TLS enabled (2375 for disabled) DOCKER_TLS_CERTDIR: '/certs' DOCKER_TLS_VERIFY: 1 DOCKER_CERT_PATH: '/certs/client' script: - make ensure-images setup build - make test-e2e The "ensure-images" make target actually just runs a `docker pull` before the tests: ensure-images: $(DOCKER) pull $(DB_IMAGE):$(DB_IMAGE_TAG) $(DOCKER) pull $(MAILER_IMAGE):$(MAILER_IMAGE_TAG) $(DOCKER) pull $(STRIPE_IMAGE):$(STRIPE_IMAGE_TAG) Which actually reminds me, another cool thing you can mock if you do this -- it's not 100% parity but there's a cool stripe replacement called localstripe[0]. There are bugs/unimplemented features (expecting a random F/OSS project without a ton of members to replicate stripe's API is a lot) but it exists and isn't too hard to use, so I use it in test/local/dev environments, use stripe test API keys in staging, and the real keys in prod. Another thing worth thinking is that how you optimize each piece is different -- it's often pretty fast to make new exchanges/databases and you can even just TRUNCATE tables if you're really worried about it. Here's an example from some code I wrote: public resetForTest(): Promise<this> { if (!this.conn) { return Promise.reject(new DatabaseDisconnectedError()); } // Do not truncate if database is <production db name> // for example when process.env.USE_LOCAL_DB_FOR_TEST is set if (this.getDatabaseName() === "<production db name>") { return Promise.resolve(this); } return this.getConnection() .then(c => c.query(` TRUNCATE TABLE <some table> CASCADE; TRUNCATE TABLE <another table> CASCADE; ... more ... `)) .then(() => this); } And this pattern extends itself to anything that is testable, so I've enshrined it: export interface TestableComponent extends Component { resetForTest(): Promise<this>; } Completely separate from all this -- some of the time you can literally just.. run these dependencies in memory[1]. Doesn't get much faster than that. I haven't run into a mock that didn't take time and effort to maintain/alter whenever code or the underlying systems changed. Time spent maintaining mocks is time wasted. Anyway, I won't go as far as saying that mocks are never useful, sometimes a mock is the easiest and best way -- but I can say that I spend most of my time writing E2E tests, and I trust them a lot and they give me trust in my codebase much more than mocks or unit tests do. I barely write unit tests anymore to check edge cases, because if I really cared I'd just generate them -- and I don't really have to care because I use strong type systems where possible (Typescript for JS, I avoid PHP/Python/Ruby, Rust, Haskell). Here comes the hot-take -- unit tests (and their rise in popularity/necessity) is actually a direct reflection of the adoption of non-compile-time-type-checked dynamically interpreted languages, and burdensome class-based "type systems". Correct usage of good, expressive and concise (where possible) type systems encourages you to make invalid/nonsensical states impossible, and people got excited about how fast you could churn out code (compared to Java most things feel pretty productive) and they started having numbers where they thought they had strings, and negative numbers where they thought they had natural numbers. Good type systems make it easy to make these cases impossible. As for the business stuff (never store decimals for currency, make sure your amounts are natural numbers, etc), you have to learn that with time/intuition built over time -- you don't know to write that unit test unless it's bit you before (though sitting down to write unit tests might tease it out of you). I write unit tests as regression tests basically now, but I find I rarely have to do that, since most of the time when a weird case has gone through it's an indicator that I was too lose with the types. [0]: https://github.com/adrienverge/localstripe https://github.com/adrienverge/localstripe [1]: https://github.com/nodkz/mongodb-memory-server https://github.com/nodkz/mongodb-memory-server
• ex_amazon_sde 6y ago
  Complexity never comes for free. Running a bunch of services creates all sort of failure modes unrelated to the tests you are running and requires maintenance in the long term. Adding docker to it only increases the overall complexity. There are good reasons for doing unit and functional testing with mocks.
  • hardwaresofton 6y ago
    > Complexity never comes for free. A platitude, thanks. > Running a bunch of services creates all sort of failure modes unrelated to the tests you are running and requires maintenance in the long term. Yes, and mocks are not free either -- the more complex the backend system you're dealing with, the more complex your mock of it must be, and that code will be brittle. I specifically said in the case where backing software is simple/easy to spin up, then it makes sense to spin it up. How much of Redis are you going to re-implement to get a mock with good parity that is in the end still missing the quirks of real redis? > Adding docker to it only increases the overall complexity. We are literally discussing a project that runs temporary isolated postgres with an extremely simple single executable experience. It's case-dependent but all you need is a different query string. It's not that hard -- if you think it's that hard to spin up a redis/postgres container on your machine in 2021 then I don't know what to tell you, a lot of things are going to be hard for you. And again, what you want to do instead is write an approximation of postgres in your codebase that will never run anywhere else -- this kind of thing is only easy if it's done for you. As soon as you have a non-trivial interface (let's say one that would do a JOIN in production but you've mocked locally) then you're going to have a hard time writing that mock and maintaining it. > There are good reasons for doing unit and functional testing with mocks. Did I say there weren't? You've eviscerated the strawman you made. Let me restate -- if you're mocking function calls/point-in-time calls with what they're supposed to return then that's fine. However, if you're maintaining a fake approximation of a redis server in your code that isn't a library (i.e. written by someone else), then that is stupid. Just run redis (even without a container). If you find it difficult to run a single extremely easy to administer (in the case of redis) external process (dockerized or not) in your tests, then I don't know what to tell you -- I just hope we never work together.