4 ms·

> A big smell here is if the large majority of your tests are mocked. We fell into this hard for a few years at my work. Going back to any of that code is a n

> A big smell here is if the large majority of your tests are mocked. We fell into this hard for a few years at my work. Going back to any of that code is a nightmare because the test suites are so fragile. Testing behavior and setting up as much of the system as possible leads to much better results in my experience.

hinkley 6y ago
Bertrand Meyer had the right of it, but I had to figure this out myself before I ever saw him quoted on the subject. Me: Code that makes decisions has branches. Branches require combinatoric tests. Code with external actions requires mocks. Therefore: Code that makes decisions and calls external systems requires combinatorics for mocks. Bertrand, more (too?) concisely: Separate code that makes decisions from code that acts on them. Follow this pattern to its logical conclusions, and most of you mocks become fixtures instead. You are passing in a blob of text as an argument instead of mocking the code that reads it from the file system. You are looking at a request body instead of mocking the PUT function in the HTTP library. The tests of external systems are much fewer, and tend to be testing the plumbing and transportation of data. If I give you a response body do you actually propagate it to the http library? And even here, spies and stubs are simpler than full mocks.
- bmh100 6y ago
  I used this strategy when developing a client library for a web socket API. It was hugely helpful. I could just include the string of the response in my tests, instead of needing a live server or even a mock server for testing. Tests were much simpler to write and faster to execute.
  - danenania 6y ago
    This is great until the API response changes and you have to painstakingly update all your string fixtures to match.
    - hinkley 6y ago
      The mocks have the same fixtures, plus a bunch of plumbing you have to sort out with every major and some minor version number upgrades. You pay a bigger tax on the mocks down the road.
    - jschwartzi 6y ago
      One would argue that you should change your string fixtures to match and verify that the new API response doesn't break anything with your existing API client. Then you change the API client and verify that all the old tests still work as expected. Better yet is if you keep the old fixtures and the new fixtures and ensure that your API client doesn't suddenly throw errors if the API server downgrades to before the new field was added.

hinkley 6y ago

Bertrand Meyer had the right of it, but I had to figure this out myself before I ever saw him quoted on the subject. Me: Code that makes decisions has branches. Branches require combinatoric tests. Code with external actions requires mocks. Therefore: Code that makes decisions and calls external systems requires combinatorics for mocks. Bertrand, more (too?) concisely: Separate code that makes decisions from code that acts on them. Follow this pattern to its logical conclusions, and most of you mocks become fixtures instead. You are passing in a blob of text as an argument instead of mocking the code that reads it from the file system. You are looking at a request body instead of mocking the PUT function in the HTTP library. The tests of external systems are much fewer, and tend to be testing the plumbing and transportation of data. If I give you a response body do you actually propagate it to the http library? And even here, spies and stubs are simpler than full mocks.

bmh100 6y ago
I used this strategy when developing a client library for a web socket API. It was hugely helpful. I could just include the string of the response in my tests, instead of needing a live server or even a mock server for testing. Tests were much simpler to write and faster to execute.
- danenania 6y ago
  This is great until the API response changes and you have to painstakingly update all your string fixtures to match.
  - hinkley 6y ago
    The mocks have the same fixtures, plus a bunch of plumbing you have to sort out with every major and some minor version number upgrades. You pay a bigger tax on the mocks down the road.
  - jschwartzi 6y ago
    One would argue that you should change your string fixtures to match and verify that the new API response doesn't break anything with your existing API client. Then you change the API client and verify that all the old tests still work as expected. Better yet is if you keep the old fixtures and the new fixtures and ensure that your API client doesn't suddenly throw errors if the API server downgrades to before the new field was added.