9 ms·

Amen to the idea. - Prefer real objects, or fakes over mocks. It will make your tests usually more robust. - Use mocks when you must: to avoid networking, or

Amen to the idea. - Prefer real objects, or fakes over mocks. It will make your tests usually more robust. - Use mocks when you must: to avoid networking, or other flaky things such as storage. - Use mocks for “output only objects”, for example listeners, or when verifying the output for some logging. (But, prefer a good fake) - Use mocks when you “need to get shit done”, it’s the easiest way to add tests in an area that has almost none, and the code is not designed to be easily testable. But remember this is tech debt, and try to migrate towards real objects over time. That’s my short advice I told many times. So might as well comment with it here.

tomcam 5y ago
It all sounds great. I agree totally in principle! I am finding that testing my fairly small Go project (static site generator, because the world definitely needs another one) chews up massive amounts of time. So I tend to avoid the testing pass for longer than I should. Any thoughts on that issue?
- chii 5y ago
  if you don't mind spending the effort, have a quick profile to see where the tests are taking a long time - is it initializing the real components that could be mocked? Or are the tests themselves taking a long time because of other factors, such as IO etc? For IO, may be there should be an abstraction over these IO api and you use an inmemory option instead for testing.
  - bluGill 5y ago
    Of io is the problem I general solve that by testing smaller data sets. I have not found io to the local disk is slow. Of course if it is network io that is bad, but local disks are fine for the size of data in my tests
- vlovich123 5y ago
  Make it CI’s problem. You should only really be running tests regularly for the component you’re currently working on with CI making sure you don’t have accidental regressions.
- edoceo 5y ago
  I try to write the test first, or at least stub in the test how I think it should work. Or at least enough notes to pick it up "tomorrow". Then I have a reference for usage i can build with that in mind
- jerf 5y ago
  It isn't immediately obvious to me why a small Go static site generator would require "massive amounts of time" to run its tests, so it's hard to answer what you're doing wrong. Are your tests perhaps just too darned big? You don't, in general, need to render 5000 pages of something to test your template doesn't crash or something. It could also just be disk access. Consider trying an in-memory file system, or if you're on linux, look at using /dev/shm which is a RAM disk. It is also possible you've snuck in a quadratic or worse time algorithm. There's nothing fundamental to the problem of a static site generator that would require such algorithms, but speaking from experience it is an environment where it's easy to loop over the return value of one function, which itself loops over something other function (quite likely the same data), which itself loops over the same structure, and it's easy to end up with O(n^3) or O(n^4) without realizing it. It's especially easy to end up with that being a "for each page" type of loop. Static site generation should be O(n), give or take small factors (maybe O(n log n) for some things technically, but at a scale where O(n log n) is practically O(n) anyhow).
- tomcam 5y ago
  Thanks all for super helpful feedback. This was a generous gift of your time and brainpower.
bluGill 5y ago
I have not found storage to be flaky and so I don't mock it. Tmpfile always gives me a unique file, and that is all the fake I need. I don't even look up the various forms of temp file to see which ones don't have a race condition as in practice they never do (if I was writing encryption or other such production code I would, but for a unit test the odds of. A race causing a failure are low enough to ignore)
- epage 5y ago
  At minimum, you need to be able to choose where the files are stored to use temp, I find thtre is still the problem of complicated setup. Also, if you are going for particular semaitics (e.g. cross process interactions), isolating that for testing it more specifically can be helpful. For me, I look for higher level abstractions and mock when possible and don't sweat testing against temp files otherwise. I've had several TDD people jump to wanting to mock each filesystem call. One was for a cross process storage API. I was trying to get them to just have a Backend interface for reading and writing instead, with tests just using an InMemory implementation as a Double.
Tainnor 5y ago
I think the original ideas of mocks (if you go and read Growing Object-Oriented Software, Guided by Tests) had some merit: In that style of TDD, mocks are used to discover (hopefully somewhat stable) interfaces between components, and in theory, it fits with the idea that OOP is about "objects sending messages to each other". I can believe that it's possible to write good systems with this kind of approach. Unfortunately, in practice, mocks are rarely used like that and most "OOP" designs have horrible boundaries and are really not much about message passing anymore. That leads to brittle mocks where you constantly have to change tests when you change implementation details. I have also gravitated away from classic OOP and much more towards the "functional core, imperative shell" concept as outlined in the article (although it's difficult to keep this pattern throughout a codebase, especially if you have team members). In such a system you really rarely need mocks. Agreed that fakes, when you have them, are nicer than mocks, especially when the system to be faked has a large API (i.e. use a redis fake, instead of checking the exact commands you send to redis). However, for some outside systems, writing a fake can be a lot of effort. In such a case, I think it's totally valid to write a "gateway class" that isn't unit tested (you can cover it with integration tests instead) which exposes a nice API (e.g. "storeFile(...)") and then to use mocks of that class in other tests.
- arcticbull 5y ago
  In test cases where you extensively involve mocks, more often than not in my experience, you end up testing that your mocks do the thing you told them to.
  - mikepurvis 5y ago
    Yeah, particularly the case with a lot of "glue" type code that's really just passing stuff back and forth and not really making any decisions with it. I've always struggled to feel that mock-based testing in this scenario was anything other than busy-work.
  - pstuart 5y ago
    Yes, but the coverage numbers look great!
mirekrusin 5y ago
- Prefer simulators over mocks We're running business critical system for several years now, creating simulators has been one of our "secrets" that contributed to the success. Not only our tests are running in as close to production environment as possible, we're also using them for local development where developers can spin up functioning system on own dev machine.
- valenterry 5y ago
  I agree - but it can be hard to get good ones, e.g. if you need to simulate sql query execution.
  - bluGill 5y ago
    Sqlite is a great simulation for SQL. You need to limit your SQL to the subset that us supported by both it and your target database though, which might be a problem.
    - jon-wood 5y ago
      In a world of Docker existing there’s rarely a reason not to just use your target database - I can have a non-production grade Postgres instance up and running in less time than it took me to write this message.
  - withinboredom 5y ago
    We spin up a temporary and local MySQL instance per test run with helpers to generate necessary data on-demand. All our tests therefore use real queries on a real db. Due to speed constraints, this db is shared between all the tests you run in that session, so it’s possible to influence tests after yours runs in CI. The reality is that that is pretty easy to detect and it’s caused us to write less brittle code.
    - mirekrusin 5y ago
      We're using mssql, on bootstrap we're running all migrations, seeding database, then backing it up; each test file restores its own database from backup which is much faster than alternatives; e2e tests run against simulated environment that has single database; e2e are split into concurrent runs with distinct set of tests to speed things up.
collyw 5y ago
Generally the more difficult a test is to write and run, the more useful it is.

tomcam 5y ago

It all sounds great. I agree totally in principle! I am finding that testing my fairly small Go project (static site generator, because the world definitely needs another one) chews up massive amounts of time. So I tend to avoid the testing pass for longer than I should. Any thoughts on that issue?

chii 5y ago
if you don't mind spending the effort, have a quick profile to see where the tests are taking a long time - is it initializing the real components that could be mocked? Or are the tests themselves taking a long time because of other factors, such as IO etc? For IO, may be there should be an abstraction over these IO api and you use an inmemory option instead for testing.
- bluGill 5y ago
  Of io is the problem I general solve that by testing smaller data sets. I have not found io to the local disk is slow. Of course if it is network io that is bad, but local disks are fine for the size of data in my tests
vlovich123 5y ago
Make it CI’s problem. You should only really be running tests regularly for the component you’re currently working on with CI making sure you don’t have accidental regressions.
edoceo 5y ago
I try to write the test first, or at least stub in the test how I think it should work. Or at least enough notes to pick it up "tomorrow". Then I have a reference for usage i can build with that in mind
jerf 5y ago
It isn't immediately obvious to me why a small Go static site generator would require "massive amounts of time" to run its tests, so it's hard to answer what you're doing wrong. Are your tests perhaps just too darned big? You don't, in general, need to render 5000 pages of something to test your template doesn't crash or something. It could also just be disk access. Consider trying an in-memory file system, or if you're on linux, look at using /dev/shm which is a RAM disk. It is also possible you've snuck in a quadratic or worse time algorithm. There's nothing fundamental to the problem of a static site generator that would require such algorithms, but speaking from experience it is an environment where it's easy to loop over the return value of one function, which itself loops over something other function (quite likely the same data), which itself loops over the same structure, and it's easy to end up with O(n^3) or O(n^4) without realizing it. It's especially easy to end up with that being a "for each page" type of loop. Static site generation should be O(n), give or take small factors (maybe O(n log n) for some things technically, but at a scale where O(n log n) is practically O(n) anyhow).
tomcam 5y ago
Thanks all for super helpful feedback. This was a generous gift of your time and brainpower.

bluGill 5y ago

I have not found storage to be flaky and so I don't mock it. Tmpfile always gives me a unique file, and that is all the fake I need. I don't even look up the various forms of temp file to see which ones don't have a race condition as in practice they never do (if I was writing encryption or other such production code I would, but for a unit test the odds of. A race causing a failure are low enough to ignore)

epage 5y ago
At minimum, you need to be able to choose where the files are stored to use temp, I find thtre is still the problem of complicated setup. Also, if you are going for particular semaitics (e.g. cross process interactions), isolating that for testing it more specifically can be helpful. For me, I look for higher level abstractions and mock when possible and don't sweat testing against temp files otherwise. I've had several TDD people jump to wanting to mock each filesystem call. One was for a cross process storage API. I was trying to get them to just have a Backend interface for reading and writing instead, with tests just using an InMemory implementation as a Double.

Tainnor 5y ago

I think the original ideas of mocks (if you go and read Growing Object-Oriented Software, Guided by Tests) had some merit: In that style of TDD, mocks are used to discover (hopefully somewhat stable) interfaces between components, and in theory, it fits with the idea that OOP is about "objects sending messages to each other". I can believe that it's possible to write good systems with this kind of approach. Unfortunately, in practice, mocks are rarely used like that and most "OOP" designs have horrible boundaries and are really not much about message passing anymore. That leads to brittle mocks where you constantly have to change tests when you change implementation details. I have also gravitated away from classic OOP and much more towards the "functional core, imperative shell" concept as outlined in the article (although it's difficult to keep this pattern throughout a codebase, especially if you have team members). In such a system you really rarely need mocks. Agreed that fakes, when you have them, are nicer than mocks, especially when the system to be faked has a large API (i.e. use a redis fake, instead of checking the exact commands you send to redis). However, for some outside systems, writing a fake can be a lot of effort. In such a case, I think it's totally valid to write a "gateway class" that isn't unit tested (you can cover it with integration tests instead) which exposes a nice API (e.g. "storeFile(...)") and then to use mocks of that class in other tests.

arcticbull 5y ago
In test cases where you extensively involve mocks, more often than not in my experience, you end up testing that your mocks do the thing you told them to.
- mikepurvis 5y ago
  Yeah, particularly the case with a lot of "glue" type code that's really just passing stuff back and forth and not really making any decisions with it. I've always struggled to feel that mock-based testing in this scenario was anything other than busy-work.
- pstuart 5y ago
  Yes, but the coverage numbers look great!

mirekrusin 5y ago

- Prefer simulators over mocks We're running business critical system for several years now, creating simulators has been one of our "secrets" that contributed to the success. Not only our tests are running in as close to production environment as possible, we're also using them for local development where developers can spin up functioning system on own dev machine.

valenterry 5y ago
I agree - but it can be hard to get good ones, e.g. if you need to simulate sql query execution.
- bluGill 5y ago
  Sqlite is a great simulation for SQL. You need to limit your SQL to the subset that us supported by both it and your target database though, which might be a problem.
  - jon-wood 5y ago
    In a world of Docker existing there’s rarely a reason not to just use your target database - I can have a non-production grade Postgres instance up and running in less time than it took me to write this message.
- withinboredom 5y ago
  We spin up a temporary and local MySQL instance per test run with helpers to generate necessary data on-demand. All our tests therefore use real queries on a real db. Due to speed constraints, this db is shared between all the tests you run in that session, so it’s possible to influence tests after yours runs in CI. The reality is that that is pretty easy to detect and it’s caused us to write less brittle code.
  - mirekrusin 5y ago
    We're using mssql, on bootstrap we're running all migrations, seeding database, then backing it up; each test file restores its own database from backup which is much faster than alternatives; e2e tests run against simulated environment that has single database; e2e are split into concurrent runs with distinct set of tests to speed things up.

collyw 5y ago

Generally the more difficult a test is to write and run, the more useful it is.