30 ms·

Author here. The way to test "does this code send 0, 1, or 500 emails" is the Output Tracking pattern: https://www.jamesshore.com/v2/projects/testing-without-mo

Author here. The way to test "does this code send 0, 1, or 500 emails" is the Output Tracking pattern: https://www.jamesshore.com/v2/projects/testing-without-mocks/testing-without-mocks#output-tracking https://www.jamesshore.com/v2/projects/testing-without-mocks... Thanks for the wallet!

djur 4y ago
Thanks for the link and the original article. To me, OutputTracker looks a lot like a spy, except instead of verifying that the unit under test took a particular action you're instead verifying that it logged that it took an action. That would seem to create the risk of missing cases where the events emitted by code don't match its actual behavior.
- jdlshore 4y ago
  Output tracking and spies are solving the same problem, but they do it in different ways. Spies record which methods are called. Output tracking records behavior that's otherwise invisible to callers (such as inserting something into a database). There's no risk of missing cases. The output tracking happens at the same semantic level as the rest of the code and is a binary "tracked / not tracked" type of thing. There's no behavior to match, and the code is tested anyway. Edit: By "no behavior to match," I mean that the thing doing the behavior is the thing tracking the behavior. The tracker is driven by events you emit when you perform a behavior.
  - djur 4y ago
    I'm saying that if I have code like this, with payload = prepare_payload if verify_payload?(payload) mailer.deliver(payload) emitter.emit(:sent_the_email) end there is a risk that a later change to this code will make it so that the `mailer` and `emitter` are not guaranteed to be called together. I have seen this bug in production and I don't see how your approach catches it. I'm also not sure how I'm supposed to test for different desired values in `payload` here.
    - jdlshore 4y ago
      I mean, sure, if you program the output tracker incorrectly, it won't work. Not sure what else you expect. You're expected to have tests, of the output tracking code itself. They catch changes that breaks the output tracker, just like you have tests to catch any other regressions. There's an example of that kind of test here: https://github.com/jamesshore/livestream/blob/2020-09-22-end/src/node_modules/infrastructure/_command_line_test.js#L33-L42 https://github.com/jamesshore/livestream/blob/2020-09-22-end... Regarding testing different values, I think what you're missing is that you don't just emit an event; you emit an event with data. Typically it's whatever data you're delivering. emitter.emit(:sent_the_email, payload) Then later, you assert on that data. assert.equal(delivered, { 'my_expected' => 'data_goes_here' })
      - djur 4y ago
        Looking at your example, I'm still not seeing how this isn't just implementing ad hoc mocking for each component. The reason I'm interested is because the overall approach is very similar to what I've settled on over the years, other than the aversion to using labor-saving DI and mocking frameworks. I'm not sure why I should prefer to write more code (that needs to be tested itself) rather than relying on a well-tested and well-understood library.
        jdlshore 4y ago
        I don’t know what else to say, man. Maybe try it for yourself so you can see how it works? Mocks lead to solitary, interaction-based tests. My approach leads to sociable, state-based tests. These are polar opposite testing approaches, with different tradeoffs. I don’t care which approach you use, but saying they’re the same thing means you don’t understand it. Other people haven’t had the same problem understanding the fundamentals that you are. You’re asking very basic questions, which makes me think you haven’t taken the time to read the article carefully. I’m happy to help, but your dismissive attitude makes me think you’re less interested in understanding the material and more interested in proving that you don’t need to understand it. Your shallow comments about Capitalization and wallets didn’t exactly endear you to me, either. I’ve provided a lot of material online. An article with tons of details and examples. Links to additional full-fledged examples. Multiple video series. Now it’s on you to take advantage of these resources. Or not; no skin off my nose either way.
valenterry 4y ago
That stops working quickly - namely as soon as you want to test a function A that uses two other functions B and C both of which have some output that is being used. For example, a function B that sends an email to a user through a 3rd party system and returns an indiciation whether the request to send the email was successfull, a function C that stores in the database that a notification was sent successfully and now a function A that calls B and, if it fails, repeats it a few times, then calls C and, if it fails, repeats it a few times, otherwise fails itself. This "do X, then depending on the ouput do Y or Z and dependin on their or ..." can't be tested in the way you describe. You WILL end up using a form of "mocking", for example passing the functions B and C as arguments to A and then, under test, don't really pass B and C but different functions that allow you to make assertions in test. That is still mocking.
- jdlshore 4y ago
  There's nothing difficult about the scenario you're describing at all. I don't have example code for that specific scenario, but I do have an example of the following scenario: A calls B, which calls an external service. B returns function D, which can be used to cancel the request. When B fails to return within five seconds, A calls D to cancel the request, then calls E to write an error message to stdout. The test for this scenario checks that the request was made, the request was cancelled, and the error was written to stdout. You can see that test here: https://github.com/jamesshore/livestream/blob/2020-09-22-end/src/rot13-cli/_rot13_cli_test.js#L44-L67 https://github.com/jamesshore/livestream/blob/2020-09-22-end...
  - djur 4y ago
    That `run` function looks to me like setting up test doubles. What makes `stderr` in this code different from a spy?
    - jdlshore 4y ago
      Not being a spy. :-) It's an array that's populated by an event listener. CommandLine is the actual production code that writes to stdout and stderr (and reads command-line arguments). CommandLine.createNull() creates an instance of CommandLine that's "turned off" and doesn't actually write to stdout or stderr. CommandLine.trackStderr() returns a reference to an array that is updated whenever something is written to stderr (or not, in the case of a nulled CommandLine). I'm off to bed, but I'm happy to answer further questions in the morning. For free, even.

djur 4y ago

Thanks for the link and the original article. To me, OutputTracker looks a lot like a spy, except instead of verifying that the unit under test took a particular action you're instead verifying that it logged that it took an action. That would seem to create the risk of missing cases where the events emitted by code don't match its actual behavior.

jdlshore 4y ago
Output tracking and spies are solving the same problem, but they do it in different ways. Spies record which methods are called. Output tracking records behavior that's otherwise invisible to callers (such as inserting something into a database). There's no risk of missing cases. The output tracking happens at the same semantic level as the rest of the code and is a binary "tracked / not tracked" type of thing. There's no behavior to match, and the code is tested anyway. Edit: By "no behavior to match," I mean that the thing doing the behavior is the thing tracking the behavior. The tracker is driven by events you emit when you perform a behavior.
- djur 4y ago
  I'm saying that if I have code like this, with payload = prepare_payload if verify_payload?(payload) mailer.deliver(payload) emitter.emit(:sent_the_email) end there is a risk that a later change to this code will make it so that the `mailer` and `emitter` are not guaranteed to be called together. I have seen this bug in production and I don't see how your approach catches it. I'm also not sure how I'm supposed to test for different desired values in `payload` here.
  - jdlshore 4y ago
    I mean, sure, if you program the output tracker incorrectly, it won't work. Not sure what else you expect. You're expected to have tests, of the output tracking code itself. They catch changes that breaks the output tracker, just like you have tests to catch any other regressions. There's an example of that kind of test here: https://github.com/jamesshore/livestream/blob/2020-09-22-end/src/node_modules/infrastructure/_command_line_test.js#L33-L42 https://github.com/jamesshore/livestream/blob/2020-09-22-end... Regarding testing different values, I think what you're missing is that you don't just emit an event; you emit an event with data. Typically it's whatever data you're delivering. emitter.emit(:sent_the_email, payload) Then later, you assert on that data. assert.equal(delivered, { 'my_expected' => 'data_goes_here' })
    - djur 4y ago
      Looking at your example, I'm still not seeing how this isn't just implementing ad hoc mocking for each component. The reason I'm interested is because the overall approach is very similar to what I've settled on over the years, other than the aversion to using labor-saving DI and mocking frameworks. I'm not sure why I should prefer to write more code (that needs to be tested itself) rather than relying on a well-tested and well-understood library.
      - jdlshore 4y ago
        I don’t know what else to say, man. Maybe try it for yourself so you can see how it works? Mocks lead to solitary, interaction-based tests. My approach leads to sociable, state-based tests. These are polar opposite testing approaches, with different tradeoffs. I don’t care which approach you use, but saying they’re the same thing means you don’t understand it. Other people haven’t had the same problem understanding the fundamentals that you are. You’re asking very basic questions, which makes me think you haven’t taken the time to read the article carefully. I’m happy to help, but your dismissive attitude makes me think you’re less interested in understanding the material and more interested in proving that you don’t need to understand it. Your shallow comments about Capitalization and wallets didn’t exactly endear you to me, either. I’ve provided a lot of material online. An article with tons of details and examples. Links to additional full-fledged examples. Multiple video series. Now it’s on you to take advantage of these resources. Or not; no skin off my nose either way.

valenterry 4y ago

That stops working quickly - namely as soon as you want to test a function A that uses two other functions B and C both of which have some output that is being used. For example, a function B that sends an email to a user through a 3rd party system and returns an indiciation whether the request to send the email was successfull, a function C that stores in the database that a notification was sent successfully and now a function A that calls B and, if it fails, repeats it a few times, then calls C and, if it fails, repeats it a few times, otherwise fails itself. This "do X, then depending on the ouput do Y or Z and dependin on their or ..." can't be tested in the way you describe. You WILL end up using a form of "mocking", for example passing the functions B and C as arguments to A and then, under test, don't really pass B and C but different functions that allow you to make assertions in test. That is still mocking.

jdlshore 4y ago
There's nothing difficult about the scenario you're describing at all. I don't have example code for that specific scenario, but I do have an example of the following scenario: A calls B, which calls an external service. B returns function D, which can be used to cancel the request. When B fails to return within five seconds, A calls D to cancel the request, then calls E to write an error message to stdout. The test for this scenario checks that the request was made, the request was cancelled, and the error was written to stdout. You can see that test here: https://github.com/jamesshore/livestream/blob/2020-09-22-end/src/rot13-cli/_rot13_cli_test.js#L44-L67 https://github.com/jamesshore/livestream/blob/2020-09-22-end...
- djur 4y ago
  That `run` function looks to me like setting up test doubles. What makes `stderr` in this code different from a spy?
  - jdlshore 4y ago
    Not being a spy. :-) It's an array that's populated by an event listener. CommandLine is the actual production code that writes to stdout and stderr (and reads command-line arguments). CommandLine.createNull() creates an instance of CommandLine that's "turned off" and doesn't actually write to stdout or stderr. CommandLine.trackStderr() returns a reference to an array that is updated whenever something is written to stderr (or not, in the case of a nulled CommandLine). I'm off to bed, but I'm happy to answer further questions in the morning. For free, even.