3 ms·

Lots of people have different opinions on this. You're entitled to yours, but I think it's hasty to call it bad advice. I'm not so worried about http.Client be

Lots of people have different opinions on this. You're entitled to yours, but I think it's hasty to call it bad advice. I'm not so worried about http.Client behaving incorrectly, but I am worried about the server I'm targeting behaving in a way that's different than my expectations. I'm also worried about getting cookie, post-data, url, etc. formats wrong. Even if they match what I put in my mock, they won't necessarily match what's out there in production. I'm worried that I'll handle download resumption wrong, for example. If I have the capability to bring up the target server locally, that's what I will do. If I can't, but it's straightforward to create a work-alike, I'll bring that up in process. Only in the most dire of circumstances will I use mocks to test my code. Think about it this way. What harm is done by using the real object, under the circumstances I described? Sure, if the dependency code breaks, my tests will too. Is that bad? Is there any point in my tests passing when my code won't actually work in production? OK, so maybe the tests will run slower? This matters in the limit, but often times you can use the production dependency without materially increasing test time. OK but what about the CI system using more resources? That will only be a problem of significant economic import if you're running at a scale like Google or Facebook. My opinion is there are a few main valid objections to my approach. One is an argument about the size of the test code. In some cases, bringing up the production dependency will require more code than a mock. Sometimes a lot more code. My response is that we should make it easier to bring up the production dependency! But if we can't, I can see the case for a mock or a fake. The other counterargument is for when we really are interested in testing the sequence of interactions the code under test has with the production system. In that event, I guess I'm ok with using a mock, but I think most of the time we're more interested in testing state. We can also use things like testvalue injection to test behavior sequences, which is normally a lighter-weight approach, and in some cases can allow you to reproduce paths that you'd otherwise have trouble hitting, even with a mock.

sifoo 9y ago
This is how I do it as well these days, testing as much of the final product as possible in the normal test run. It's not that I didn't try unit testing and mocking, or don't understand; my experience tells me that it's not a constructive approach. This means that I always test mostly the same code that other people run, which prevents all sorts of mistakes; and I don't waste precious time writing and maintaining mocks. It will creep up on you, but if you keep mocking everything, sooner or later the effort required to change anything will start hurting the project. It's easy to loose track of the goal; which should be preventing as many production errors as possible as effectively as possible, not checking all the boxes of whatever religion.
sagichmal 9y ago
> I am worried about the server I'm targeting behaving in a way that's different than my expectations This is the responsibility of system or integration tests, not unit tests. (Unit tests are what's under discussion in the article.) > What harm is done by using the real object, under the circumstances I described? Time, energy, and effort, all of which inevitably lead to longer (slower) dev/test cycles.
- sdhgaiojfsa 9y ago
  > Time, energy, and effort My contention is that using real objects normally does not take materially more of any of these, and thus that most code is served best by being tested only by integration and regression tests. Where my contention is untrue, apply your best judgment, of course. :)
convolvatron 9y ago
i wish i could give you 1 thousand points. isolated unit tests can have value in that the bugs they expose are more easily tracked down. full-system testing, with an honest attempt to do fault injection, by definition exposes all the issues you might have in production. if you have a decent test suite you can honestly claim you can expect it to take client code correctly. i don't know how thats true if you've only tested A against some stubbed out version of B that always returns success. integrated testing is harder and requires more framework. its also kinda necessary, and if you only have the resources to do one or the other, the choice seems pretty clear. i also heartily agree with your comment that since we're supposed to have frictionless and scalable deployment. there is no reason not to test this as part of integration testing and also exploit this function to make integration testing easier.
tigershark 9y ago
It seems a bad advice also in my eyes. I guess that you have never worked on big systems if for you it is normal to bring up all the production dependencies for an integration test. For small projects it may make sense, although I would still prefer to write unit tests to catch exactly were the bug happens and to be independent from breakages in the external systems. The correct approach is to rely heavily on unit tests that run continuously at every build and run the full integration testing suite at regular intervals or overnight. In a quite big project on which I worked many years ago the full integration test suite took 11-12 hours to run, I would be curious how you would proceed in that case given that from what you write it seems that unit tests are pretty useless to you..
- sdhgaiojfsa 9y ago
  > I guess that you have never worked on big systems I work on search and indexing systems at Google, and have done for more than a decade. I'm by no means the best, nor an authority. But I have spent some time living in this problem space, and this is my conclusion from my years of experience. FWIW a single run of the tests for the system I work on now takes . . . I don't know. I would guess hours or maybe even a day of computer time? But, computers are cheap, and defects are expensive. So you split the test suite into bits small enough that the whole thing runs in ten minutes, and run them all in parallel. And, yes, there have been cases where I have had to debug the slow startup of a dependency so that I can make it fast enough to run in my test. But, that's what they pay me for, I enjoy that type of work, and it has ancillary benefits to the dependency when it needs to start up in production.
  - tigershark 9y ago
    In the real world you don’t have pretty much unlimited resources. If I remember correctly those tests I was speaking about were already running in parallel on 2 or 3 machines otherwise the total time would have been 2 or 3 times more. Obviously we couldn’t use hundreds of machines to make them run in 10 minutes. In most of the places were I worked even getting a couple of new machines for production use would take months, I think if I requested even a couple of machines dedicated only for the integration testing I would have been laughed in the face. So if you can run a full suite of integration tests in minutes on hundreds/thousands of machines then good for you, but don’t expect that every company has the resources for doing it. And in those cases unit tests that run continuously are more useful than integration tests that run once a day.

sifoo 9y ago

This is how I do it as well these days, testing as much of the final product as possible in the normal test run. It's not that I didn't try unit testing and mocking, or don't understand; my experience tells me that it's not a constructive approach. This means that I always test mostly the same code that other people run, which prevents all sorts of mistakes; and I don't waste precious time writing and maintaining mocks. It will creep up on you, but if you keep mocking everything, sooner or later the effort required to change anything will start hurting the project. It's easy to loose track of the goal; which should be preventing as many production errors as possible as effectively as possible, not checking all the boxes of whatever religion.

sagichmal 9y ago

> I am worried about the server I'm targeting behaving in a way that's different than my expectations This is the responsibility of system or integration tests, not unit tests. (Unit tests are what's under discussion in the article.) > What harm is done by using the real object, under the circumstances I described? Time, energy, and effort, all of which inevitably lead to longer (slower) dev/test cycles.

sdhgaiojfsa 9y ago
> Time, energy, and effort My contention is that using real objects normally does not take materially more of any of these, and thus that most code is served best by being tested only by integration and regression tests. Where my contention is untrue, apply your best judgment, of course. :)

convolvatron 9y ago

i wish i could give you 1 thousand points. isolated unit tests can have value in that the bugs they expose are more easily tracked down. full-system testing, with an honest attempt to do fault injection, by definition exposes all the issues you might have in production. if you have a decent test suite you can honestly claim you can expect it to take client code correctly. i don't know how thats true if you've only tested A against some stubbed out version of B that always returns success. integrated testing is harder and requires more framework. its also kinda necessary, and if you only have the resources to do one or the other, the choice seems pretty clear. i also heartily agree with your comment that since we're supposed to have frictionless and scalable deployment. there is no reason not to test this as part of integration testing and also exploit this function to make integration testing easier.

tigershark 9y ago

It seems a bad advice also in my eyes. I guess that you have never worked on big systems if for you it is normal to bring up all the production dependencies for an integration test. For small projects it may make sense, although I would still prefer to write unit tests to catch exactly were the bug happens and to be independent from breakages in the external systems. The correct approach is to rely heavily on unit tests that run continuously at every build and run the full integration testing suite at regular intervals or overnight. In a quite big project on which I worked many years ago the full integration test suite took 11-12 hours to run, I would be curious how you would proceed in that case given that from what you write it seems that unit tests are pretty useless to you..

sdhgaiojfsa 9y ago
> I guess that you have never worked on big systems I work on search and indexing systems at Google, and have done for more than a decade. I'm by no means the best, nor an authority. But I have spent some time living in this problem space, and this is my conclusion from my years of experience. FWIW a single run of the tests for the system I work on now takes . . . I don't know. I would guess hours or maybe even a day of computer time? But, computers are cheap, and defects are expensive. So you split the test suite into bits small enough that the whole thing runs in ten minutes, and run them all in parallel. And, yes, there have been cases where I have had to debug the slow startup of a dependency so that I can make it fast enough to run in my test. But, that's what they pay me for, I enjoy that type of work, and it has ancillary benefits to the dependency when it needs to start up in production.
- tigershark 9y ago
  In the real world you don’t have pretty much unlimited resources. If I remember correctly those tests I was speaking about were already running in parallel on 2 or 3 machines otherwise the total time would have been 2 or 3 times more. Obviously we couldn’t use hundreds of machines to make them run in 10 minutes. In most of the places were I worked even getting a couple of new machines for production use would take months, I think if I requested even a couple of machines dedicated only for the integration testing I would have been laughed in the face. So if you can run a full suite of integration tests in minutes on hundreds/thousands of machines then good for you, but don’t expect that every company has the resources for doing it. And in those cases unit tests that run continuously are more useful than integration tests that run once a day.