4 ms·

>That's a terrible idea. You've now coupled your tests to someone else's code. It's a terrible idea not to. If your code isn't coupled to someone else's code y

>That's a terrible idea. You've now coupled your tests to someone else's code. It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Moreover, you should test against the code your code is coupled to as realistically as possible, which ideally means testing against the real thing.

WalterSear 9y ago
>It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite. You also need the modularity in your tests and code, so you can fix and replace 3rd party endpoints without being afraid of polluting your code's inputs, or having to rewrite your integration suite. This is where the data contracts come in: they make sure that each module of the system and its partners are speaking the same language, even after they have been replaced or updated to handle new 3rd party behaviours.
- cjalmeida 9y ago
  I usually have this concept of integration tests covering the interaction of modules you are in control. 3rd party services are tested using something I call "edge tests". This leads to few mocks, yet I know the expected behavior of the external services.
- skybrian 9y ago
  You're missing a fundamental difference in philosophy. Why should we only care about bugs in our code? They aren't the only bugs our users care about. A browser bug (for example) may not be our fault, but it is our problem. Therefore, we need to run our tests in real browsers to make sure it actually works, and keep running those tests on new browser releases to find any new bugs. Similarly, mobile apps need to be tested with a variety of different phones. If you upgrade a dependency and it changes behavior, you're still responsible for fixing or working around it, hopefully before users find out. Lots of interesting bugs happen in the gaps between modules, especially when they're maintained by different teams. A good reason to test code in isolation is that it makes diagnostics easier. However, if you only do that you're blind to a lot of other bugs.
  - WalterSear 9y ago
    I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated. That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge and the integration test suites to ensure end-to-end integrity (and do the same for every other part of your code that interacts with another part of your code, fwiw). > Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. So, test in the browser, or on the app. Just test the endpoints separate from your integration test, and use the explicitly stated output of integration test as the input parameters for your browser tests. I'd go far as to argue that browsers and hardware - where you have the greatest concerns about the stability of your 3rd party APIs, are actually the circumstances where separating edge testing from integration makes the most sense. Perhaps a better way to explain my process is to think of my set of edge and integration tests as the equivalent of your 'integration' test suite, with the addition of specific contracts (the shared data) to ensure that interaction between modules is also controlled for. You are still testing end-to-end and you are still testing with real-world data, since that's what the edge tests will be returning. You are just explicitly stating what you expect at each boundary of the code.
    - crdoconnor 9y ago
      >So, test in the browser, or on the app. Just test the endpoints separate from your integration test Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code. That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you change the code. It's not that you shouldn't sometimes drill down to test code at a lower level, it's that when you do, you should be drilling down to a reusable abstraction that is independent of the higher level code.
      - WalterSear 9y ago
- crdoconnor 9y ago
  What you are arguing for is that code should be loosely coupled. I generally agree with this (although, like DRY, its validity is context sensitive and one can take this approach too far). This is an orthogonal question to whether code should be tested realistically or not, however. You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible to isolate the external inputs and remove a potential source of test instability.
  - bonesss 9y ago
    > ...You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible... Ding ding: it's not either/or, it's both/and across a spectrum from dumb to smart to dumb again. If you use a google rest API in something serious you want to be able to model the error conditions of that API, you also need to verify the current status and content of that API, you also need to verify your apps behaviour separately for both cases. Loose coupling and DI are important to strike the right balance, but fundamentally you're either testing your solution or you're not.
    - crdoconnor 9y ago
      >Ding ding: it's not either/or, it's both Yes, that was the point I was making. >Loose coupling and DI are important to strike the right balance DI is useful sometimes (e.g. when you have a set of modules you want to hotswap with one another), but it's often just an overused crutch to deal with the inability of unit tests to couple to real things - like an actual REST API endpoints over a loopback interface. The whole idea that unit tests drive "good code" by making you DI all the things is a pile of shit. Unit tests just make it painful to not do DI because unit tests are themselves a form of tight coupling.

WalterSear 9y ago

>It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite. You also need the modularity in your tests and code, so you can fix and replace 3rd party endpoints without being afraid of polluting your code's inputs, or having to rewrite your integration suite. This is where the data contracts come in: they make sure that each module of the system and its partners are speaking the same language, even after they have been replaced or updated to handle new 3rd party behaviours.

cjalmeida 9y ago
I usually have this concept of integration tests covering the interaction of modules you are in control. 3rd party services are tested using something I call "edge tests". This leads to few mocks, yet I know the expected behavior of the external services.
skybrian 9y ago
You're missing a fundamental difference in philosophy. Why should we only care about bugs in our code? They aren't the only bugs our users care about. A browser bug (for example) may not be our fault, but it is our problem. Therefore, we need to run our tests in real browsers to make sure it actually works, and keep running those tests on new browser releases to find any new bugs. Similarly, mobile apps need to be tested with a variety of different phones. If you upgrade a dependency and it changes behavior, you're still responsible for fixing or working around it, hopefully before users find out. Lots of interesting bugs happen in the gaps between modules, especially when they're maintained by different teams. A good reason to test code in isolation is that it makes diagnostics easier. However, if you only do that you're blind to a lot of other bugs.
- WalterSear 9y ago
  I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated. That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge and the integration test suites to ensure end-to-end integrity (and do the same for every other part of your code that interacts with another part of your code, fwiw). > Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. So, test in the browser, or on the app. Just test the endpoints separate from your integration test, and use the explicitly stated output of integration test as the input parameters for your browser tests. I'd go far as to argue that browsers and hardware - where you have the greatest concerns about the stability of your 3rd party APIs, are actually the circumstances where separating edge testing from integration makes the most sense. Perhaps a better way to explain my process is to think of my set of edge and integration tests as the equivalent of your 'integration' test suite, with the addition of specific contracts (the shared data) to ensure that interaction between modules is also controlled for. You are still testing end-to-end and you are still testing with real-world data, since that's what the edge tests will be returning. You are just explicitly stating what you expect at each boundary of the code.
  - crdoconnor 9y ago
    >So, test in the browser, or on the app. Just test the endpoints separate from your integration test Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code. That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you change the code. It's not that you shouldn't sometimes drill down to test code at a lower level, it's that when you do, you should be drilling down to a reusable abstraction that is independent of the higher level code.
    - WalterSear 9y ago
crdoconnor 9y ago
What you are arguing for is that code should be loosely coupled. I generally agree with this (although, like DRY, its validity is context sensitive and one can take this approach too far). This is an orthogonal question to whether code should be tested realistically or not, however. You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible to isolate the external inputs and remove a potential source of test instability.
- bonesss 9y ago
  > ...You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible... Ding ding: it's not either/or, it's both/and across a spectrum from dumb to smart to dumb again. If you use a google rest API in something serious you want to be able to model the error conditions of that API, you also need to verify the current status and content of that API, you also need to verify your apps behaviour separately for both cases. Loose coupling and DI are important to strike the right balance, but fundamentally you're either testing your solution or you're not.
  - crdoconnor 9y ago
    >Ding ding: it's not either/or, it's both Yes, that was the point I was making. >Loose coupling and DI are important to strike the right balance DI is useful sometimes (e.g. when you have a set of modules you want to hotswap with one another), but it's often just an overused crutch to deal with the inability of unit tests to couple to real things - like an actual REST API endpoints over a loopback interface. The whole idea that unit tests drive "good code" by making you DI all the things is a pile of shit. Unit tests just make it painful to not do DI because unit tests are themselves a form of tight coupling.