6 ms·

There's actually some good ideas in here. I wish it wasn't written in such a buzzword-heavy, consultant-y style. When advice includes so many Novel Proper Nouns

by djur 4y ago

There's actually some good ideas in here. I wish it wasn't written in such a buzzword-heavy, consultant-y style. When advice includes so many Novel Proper Nouns I find myself checking my wallet. My main quibbles are that these "Nullables" are still test doubles, just implemented in a way with different tradeoffs than typical mocking frameworks. Also, spies are still helpful when you're checking for side effects. I'm not really seeing any alternative offered here for testing "does this code send 0, 1, or 500 emails".

readthenotes1 4y ago
otoh, I worked with a guy who was so mock heavy that he actually wrote real mocks for hash table for each test. I worked with another guy who mocked out so much external behavior that he couldn't figure out why his stuff never worked, not doing the necessary work to ensure that his expectations of external behavior were actually still met ( if you're going to mock out external behavior, you need to add in tests that ensures your expectations are valid). There is a place for Mocks, but I have found the other varieties of test doubles to more useful...
- convolvatron 4y ago
  what is the place for mocks?
  - Tainnor 4y ago
    External systems, unexpected error cases (which by nature of being unexpected can't really be triggered in a "natural" way), very generic code (e.g. instrumentation frameworks), major architectural boundaries, ... I agree that most people over-mock, though.
    - readthenotes1 4y ago
      External systems dependencies are better handled with stubs and fakes than mocks, in my experience.
      - Tainnor 4y ago
        That really depends on the case in question and on the particulars of the codebase. I wouldn't recommend mocking http libraries, database calls etc. - that usually leads to madness and impenetrable test setups. I would rather recommend to either use fake implementations (e.g. in-memory DB) or to only integration-test the boundaries of your interaction with the external system (that might or might not be hard to achieve). But I do like the pattern of wrapping calls to external APIs through a layer that I control - so that, for example, I may just write mailer.sendMessage(message) - and then that mailer interface seems like a good candidate for mocking.
  - readthenotes1 4y ago
    In my experience, they are the test double of last resort. I used them to verify behavior, not outcome (as that's what expectations do), e.g., did I really call this method only once?
- what-no-tests 4y ago
  If I'm writing unit tests for #some_method, I'll mock anything that method calls outside of that method (so I can check how that method will respond to error conditions from calling those other methods). Unit tests != Integration tests, and the two must both exist otherwise I'll run into that situation where my unit tests work but the software doesn't.
  - UglyToad 4y ago
    But do you not find the problem is you now end up with tests that assume the existence of #some_method with the tests tightly coupled to the internal logic and flow of that method? If you decide to refactor, every part of the system now has one or more tests that break because they mimic or clone each method 1:1 rather than testing input and output.
    - readthenotes1 4y ago
      Exactly the problem the 2nd guy had
    - what-no-tests 4y ago
      Perhaps, but if I refactor #some_method I only have to look in my unit tests where I describe #some_method and focus there. The integration tests should not have to be updated, since they only describe expected behavior -- unless the expected behavior must change as well. The coupling between unit tests and the methods they test is correct, IMO, because really how else are you supposed to test what #some_method would do when it tries to call #another_method but #another_method raises various different exceptions, or returns no results, or returns successfully? If the answer is "I only test happy paths" then good luck - you're not really testing anything.
jdlshore 4y ago
Author here. The way to test "does this code send 0, 1, or 500 emails" is the Output Tracking pattern: https://www.jamesshore.com/v2/projects/testing-without-mocks/testing-without-mocks#output-tracking https://www.jamesshore.com/v2/projects/testing-without-mocks... Thanks for the wallet!
- djur 4y ago
  Thanks for the link and the original article. To me, OutputTracker looks a lot like a spy, except instead of verifying that the unit under test took a particular action you're instead verifying that it logged that it took an action. That would seem to create the risk of missing cases where the events emitted by code don't match its actual behavior.
  - jdlshore 4y ago
    Output tracking and spies are solving the same problem, but they do it in different ways. Spies record which methods are called. Output tracking records behavior that's otherwise invisible to callers (such as inserting something into a database). There's no risk of missing cases. The output tracking happens at the same semantic level as the rest of the code and is a binary "tracked / not tracked" type of thing. There's no behavior to match, and the code is tested anyway. Edit: By "no behavior to match," I mean that the thing doing the behavior is the thing tracking the behavior. The tracker is driven by events you emit when you perform a behavior.
    - djur 4y ago
      I'm saying that if I have code like this, with payload = prepare_payload if verify_payload?(payload) mailer.deliver(payload) emitter.emit(:sent_the_email) end there is a risk that a later change to this code will make it so that the `mailer` and `emitter` are not guaranteed to be called together. I have seen this bug in production and I don't see how your approach catches it. I'm also not sure how I'm supposed to test for different desired values in `payload` here.
      - 4y ago
chociej 4y ago
I feel that the writing style reinforced the well-thought-out consistency of the system. The information of value was all right there, so I didn't feel worried about getting a bill.

by djur 4y ago

readthenotes1 4y ago

otoh, I worked with a guy who was so mock heavy that he actually wrote real mocks for hash table for each test. I worked with another guy who mocked out so much external behavior that he couldn't figure out why his stuff never worked, not doing the necessary work to ensure that his expectations of external behavior were actually still met ( if you're going to mock out external behavior, you need to add in tests that ensures your expectations are valid). There is a place for Mocks, but I have found the other varieties of test doubles to more useful...

convolvatron 4y ago
what is the place for mocks?
- Tainnor 4y ago
  External systems, unexpected error cases (which by nature of being unexpected can't really be triggered in a "natural" way), very generic code (e.g. instrumentation frameworks), major architectural boundaries, ... I agree that most people over-mock, though.
  - readthenotes1 4y ago
    External systems dependencies are better handled with stubs and fakes than mocks, in my experience.
    - Tainnor 4y ago
      That really depends on the case in question and on the particulars of the codebase. I wouldn't recommend mocking http libraries, database calls etc. - that usually leads to madness and impenetrable test setups. I would rather recommend to either use fake implementations (e.g. in-memory DB) or to only integration-test the boundaries of your interaction with the external system (that might or might not be hard to achieve). But I do like the pattern of wrapping calls to external APIs through a layer that I control - so that, for example, I may just write mailer.sendMessage(message) - and then that mailer interface seems like a good candidate for mocking.
- readthenotes1 4y ago
  In my experience, they are the test double of last resort. I used them to verify behavior, not outcome (as that's what expectations do), e.g., did I really call this method only once?
what-no-tests 4y ago
If I'm writing unit tests for #some_method, I'll mock anything that method calls outside of that method (so I can check how that method will respond to error conditions from calling those other methods). Unit tests != Integration tests, and the two must both exist otherwise I'll run into that situation where my unit tests work but the software doesn't.
- UglyToad 4y ago
  But do you not find the problem is you now end up with tests that assume the existence of #some_method with the tests tightly coupled to the internal logic and flow of that method? If you decide to refactor, every part of the system now has one or more tests that break because they mimic or clone each method 1:1 rather than testing input and output.
  - readthenotes1 4y ago
    Exactly the problem the 2nd guy had
  - what-no-tests 4y ago
    Perhaps, but if I refactor #some_method I only have to look in my unit tests where I describe #some_method and focus there. The integration tests should not have to be updated, since they only describe expected behavior -- unless the expected behavior must change as well. The coupling between unit tests and the methods they test is correct, IMO, because really how else are you supposed to test what #some_method would do when it tries to call #another_method but #another_method raises various different exceptions, or returns no results, or returns successfully? If the answer is "I only test happy paths" then good luck - you're not really testing anything.

jdlshore 4y ago

Author here. The way to test "does this code send 0, 1, or 500 emails" is the Output Tracking pattern: https://www.jamesshore.com/v2/projects/testing-without-mocks/testing-without-mocks#output-tracking https://www.jamesshore.com/v2/projects/testing-without-mocks... Thanks for the wallet!

djur 4y ago
Thanks for the link and the original article. To me, OutputTracker looks a lot like a spy, except instead of verifying that the unit under test took a particular action you're instead verifying that it logged that it took an action. That would seem to create the risk of missing cases where the events emitted by code don't match its actual behavior.
- jdlshore 4y ago
  Output tracking and spies are solving the same problem, but they do it in different ways. Spies record which methods are called. Output tracking records behavior that's otherwise invisible to callers (such as inserting something into a database). There's no risk of missing cases. The output tracking happens at the same semantic level as the rest of the code and is a binary "tracked / not tracked" type of thing. There's no behavior to match, and the code is tested anyway. Edit: By "no behavior to match," I mean that the thing doing the behavior is the thing tracking the behavior. The tracker is driven by events you emit when you perform a behavior.
  - djur 4y ago
    I'm saying that if I have code like this, with payload = prepare_payload if verify_payload?(payload) mailer.deliver(payload) emitter.emit(:sent_the_email) end there is a risk that a later change to this code will make it so that the `mailer` and `emitter` are not guaranteed to be called together. I have seen this bug in production and I don't see how your approach catches it. I'm also not sure how I'm supposed to test for different desired values in `payload` here.
    - 4y ago

chociej 4y ago

I feel that the writing style reinforced the well-thought-out consistency of the system. The information of value was all right there, so I didn't feel worried about getting a bill.