7 ms·

> The problem with this approach is it’s completely tautological. What are you asserting? The tests themselves. Every time I've heard this kind of rationality

> The problem with this approach is it’s completely tautological. What are you asserting? The tests themselves. Every time I've heard this kind of rationality it's from someone who doesn't understand what a unit test is and is pushing for integration tests. It's extremely useful to have very rapid tests that boil down to making sure `add(2, 2) == 4` because you can have them running constantly and catch inadvertent logic changes. Should you also have more heavy tests using local/fake services? Obviously - but it's not one or the other.

zabzonk 3y ago
he was talking about mocking, not tests such as CHECK(add(2, 2) == 4). and BTW i would challenge anyone working with accessing complex real-world systems (for example, an stock-exchange, or a robotic warehouse, to write a worthwhile mock for them.
- lexicality 3y ago
  If I have a function that hits a JSON API and returns a path from the resultant object and I mock that API to return 4, I'm doing exactly that. I can write a mock for every data type the API could return and one for every error the API can return and test all of them near-instantly without having to do a HTTP round trip to the fake server (which presumably also now needs configuring before each test)
  - zabzonk 3y ago
    i think you either over-estimate the complexity of your json api call, or under-estimate how complex accessing a stock exchange is
    - lexicality 3y ago
      anything is mockable if your code is broken into smaller easy to reason about easy to test chunks and you have a complete understanding of what the thing you're mocking is able to do. If you don't have a complete understanding of what it's able to do then I think how you write your tests is the least of your worries.
      - zabzonk 3y ago
        > anything is mockable true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange (and mocked other clients than your client app). and then you are going to have to test that mock. this could all take a while. i really don't think that people who suggest mocking realise how complex applications work and how they are always changing their interfaces - for SEs due to regulatory pressures, for example.
        pyrale 3y ago
        > true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange All your mock has to do is to provide the specific answer you're expecting to receive from that service in the specific case you're testing. In most situations, re-imlementing logic in a mock is a mistake that should be avoided.
        zabzonk 3y ago
        with a stock exchange, it's a dialog - don't expect any particular response. for example - using sell and buy just to keep things simple, and i am the seller and i could not be bothered looking up real prices or tickers: sell 10000 BT at 50GBP // i put this on the exchange buy 500 // somebody bought some buy 2000 // somebody else (or maybe the same) bought some crash // exchange crashed (not rare) we recover cancel 500 // they can do this in a limited window buy 8000 // we have filled you see, it isn't simple, but that is about as simple as you get
        hiatus 3y ago
        Wouldn't each of these be a test case?
        zabzonk 3y ago
        only if you have a stateful mock server with multiple mock clients attached to it. for example, when you put a sell on the server, you are basically going to be listening on a socket for multiple responses and what the responses are will depend on the buyers (which you must also mock). and don't get me started on buying, ticker conversions between different exchanges, currency, time, etc. it is too hard to mock. to mitigate this some exchanges, such as london and paris, do provide test exchanges, which are crap, and available every other fridays, or whatever. when i worked in this area, my boss used to test my code by putting a trade on the exchange and then immediatly cancelling it. this was illegal, as he wasn't a licensed trader, and the exchanges got enraged. i refused to have anything to do with it. but my point is - you cannot do meaningful unit tests on very complex systems. even integration tests can be very hard, but are worthwhile.
  - tsimionescu 3y ago
    Reimplementing a whole complex API with mocks is not exactly quick or easy or less error prone than the code that should interact with that API.
    - lexicality 3y ago
      if your test implements a whole complex API then it's not a unit test
    - MoreQARespect 3y ago
      It can be fairly quick with tools like mitmproxy.
    - pyrale 3y ago
      The point of mocking is not to reimplement a service you depend on, it is to provide the answer that service would give in the specific example you're testing.
      - zabzonk 3y ago
        but suppose that service could give all sorts of different responses, depending on (say) the time of day and the country originating the request.
        pyrale 3y ago
        What's relevant to you is your element under test, not the dependency. Does the component you're testing have a different behaviour for each different response? If yes, you're out of luck (but also, maybe you should split it in multiple components). If not, then you probably don't need to test every possible response the dependency returns.
        8n4vidtmkvmk 3y ago
        You're not out of luck. Your testing framework should allow you to hardcode different responses to different requests. Include the time in the response if you need to, it shouldn't be hard. I've started adding random elements to my mock responses to force other developers to not assert things that aren't relevant to the test.
ipython 3y ago
I’ve found those “obvious” checks to be useful to catch unexpected hardware issues as well. I had a customer upset that my code was crashing- I sent a debug version with asserts around some arithmetic core to the algorithm and lo and behold those asserts failed His machines memory was bad and corrupting the internal state of the application. Until I could prove that his machine was calculating the equivalent of 2+2=5, he wouldn’t believe us.
sanderjd 3y ago
I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I have a weakly-held preference (that I will not fight for in code reviews) for not unit testing imperative code at all because I think it has very limited usefulness and is just more code to maintain. Instead, I prefer to rely on integration or end-to-end testing to validate the behavior of those parts of the system. But I think unit tests are extremely useful for functions. It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. I think it is always useful to have edge case tests - when is it an error, what happens right on the boundary between error and non-error, etc. - and to have at least one illustrative happy-path example. (I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification.) Where this breaks down is when the vast majority of code is imperative, which is very common in practice. My usual approach there is to lean heavily on higher-level testing, and refactor out little chunks that can be more reasonably unit tested, when practical. And I also write and maintain lots of these not-very-useful imperative tests, because people get itchy about low unit test coverage and I'd rather write the tests than spend time convincing people that they aren't very useful.
- PartiallyTyped 3y ago
  > I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification. I am that person! I am a big fan of property based testing and try to add such tests whenever possible. They serve as "edge-case hunters" and are usually part of my programming loop. As soon as something has been modified, a test suite executes. Writing property tests first and the code later allows you think about the invariants and behaviour of what you are building in a way that regular unit tests just don't.
steveBK123 3y ago
My problem with these is that they are generally written by the developer who wrote the function. So devs code the happy path in their function, and test the happy path in their unit test. Yes, in fact, 2+2 is in fact 4. But, do they test add(2,-2) or add(-2,-2) or add(null,2) or add (0,null) .. etc. Whatever cases they forgot exist and never coded for, never get tested for either. If this is not caught in the initial commit by another dev, then it makes it through to prod happily despite having "unit tests". This is why I think tests where the application is forced to interact with real world data are better because if your app will be broken by negative/0/null, and that state exists in the real world data, it will break in testing. Of course this is all a long dissertation on how to code&test the addition of 2 integers. One of the simplest possible functions, now do this on more complex software. The unit test I think biases later developers to believe the original developer actually coded the software correctly, when to me, in a large number of cases.. it was wrong to begin with, but uncaught.

zabzonk 3y ago

he was talking about mocking, not tests such as CHECK(add(2, 2) == 4). and BTW i would challenge anyone working with accessing complex real-world systems (for example, an stock-exchange, or a robotic warehouse, to write a worthwhile mock for them.

lexicality 3y ago
If I have a function that hits a JSON API and returns a path from the resultant object and I mock that API to return 4, I'm doing exactly that. I can write a mock for every data type the API could return and one for every error the API can return and test all of them near-instantly without having to do a HTTP round trip to the fake server (which presumably also now needs configuring before each test)
- zabzonk 3y ago
  i think you either over-estimate the complexity of your json api call, or under-estimate how complex accessing a stock exchange is
  - lexicality 3y ago
    anything is mockable if your code is broken into smaller easy to reason about easy to test chunks and you have a complete understanding of what the thing you're mocking is able to do. If you don't have a complete understanding of what it's able to do then I think how you write your tests is the least of your worries.
    - zabzonk 3y ago
      > anything is mockable true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange (and mocked other clients than your client app). and then you are going to have to test that mock. this could all take a while. i really don't think that people who suggest mocking realise how complex applications work and how they are always changing their interfaces - for SEs due to regulatory pressures, for example.
      - pyrale 3y ago
        > true. but to mock something like a stock exchange, you are basically going to have to write a stock exchange All your mock has to do is to provide the specific answer you're expecting to receive from that service in the specific case you're testing. In most situations, re-imlementing logic in a mock is a mistake that should be avoided.
        zabzonk 3y ago
        with a stock exchange, it's a dialog - don't expect any particular response. for example - using sell and buy just to keep things simple, and i am the seller and i could not be bothered looking up real prices or tickers: sell 10000 BT at 50GBP // i put this on the exchange buy 500 // somebody bought some buy 2000 // somebody else (or maybe the same) bought some crash // exchange crashed (not rare) we recover cancel 500 // they can do this in a limited window buy 8000 // we have filled you see, it isn't simple, but that is about as simple as you get
        hiatus 3y ago
        Wouldn't each of these be a test case?
        zabzonk 3y ago
        only if you have a stateful mock server with multiple mock clients attached to it. for example, when you put a sell on the server, you are basically going to be listening on a socket for multiple responses and what the responses are will depend on the buyers (which you must also mock). and don't get me started on buying, ticker conversions between different exchanges, currency, time, etc. it is too hard to mock. to mitigate this some exchanges, such as london and paris, do provide test exchanges, which are crap, and available every other fridays, or whatever. when i worked in this area, my boss used to test my code by putting a trade on the exchange and then immediatly cancelling it. this was illegal, as he wasn't a licensed trader, and the exchanges got enraged. i refused to have anything to do with it. but my point is - you cannot do meaningful unit tests on very complex systems. even integration tests can be very hard, but are worthwhile.
- tsimionescu 3y ago
  Reimplementing a whole complex API with mocks is not exactly quick or easy or less error prone than the code that should interact with that API.
  - lexicality 3y ago
    if your test implements a whole complex API then it's not a unit test
  - MoreQARespect 3y ago
    It can be fairly quick with tools like mitmproxy.
  - pyrale 3y ago
    The point of mocking is not to reimplement a service you depend on, it is to provide the answer that service would give in the specific example you're testing.
    - zabzonk 3y ago
      but suppose that service could give all sorts of different responses, depending on (say) the time of day and the country originating the request.
      - pyrale 3y ago
        What's relevant to you is your element under test, not the dependency. Does the component you're testing have a different behaviour for each different response? If yes, you're out of luck (but also, maybe you should split it in multiple components). If not, then you probably don't need to test every possible response the dependency returns.
        8n4vidtmkvmk 3y ago
        You're not out of luck. Your testing framework should allow you to hardcode different responses to different requests. Include the time in the response if you need to, it shouldn't be hard. I've started adding random elements to my mock responses to force other developers to not assert things that aren't relevant to the test.

ipython 3y ago

I’ve found those “obvious” checks to be useful to catch unexpected hardware issues as well. I had a customer upset that my code was crashing- I sent a debug version with asserts around some arithmetic core to the algorithm and lo and behold those asserts failed His machines memory was bad and corrupting the internal state of the application. Until I could prove that his machine was calculating the equivalent of 2+2=5, he wouldn’t believe us.

sanderjd 3y ago

I think this is a natural conclusion from working with a lot of imperative code. The usual test you see for an imperative code block is like "mock this out, assert it was called N times with these arguments". That approach is usually redundant and tautological. I have a weakly-held preference (that I will not fight for in code reviews) for not unit testing imperative code at all because I think it has very limited usefulness and is just more code to maintain. Instead, I prefer to rely on integration or end-to-end testing to validate the behavior of those parts of the system. But I think unit tests are extremely useful for functions. It looks silly when the example function is `add(2, 2)` but most functions aren't just redefining mathematical operators, but rather are encoding non-trivial domain logic. I think it is always useful to have edge case tests - when is it an error, what happens right on the boundary between error and non-error, etc. - and to have at least one illustrative happy-path example. (I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification.) Where this breaks down is when the vast majority of code is imperative, which is very common in practice. My usual approach there is to lean heavily on higher-level testing, and refactor out little chunks that can be more reasonably unit tested, when practical. And I also write and maintain lots of these not-very-useful imperative tests, because people get itchy about low unit test coverage and I'd rather write the tests than spend time convincing people that they aren't very useful.

PartiallyTyped 3y ago
> I think this can be frustrating for more mathematically-inclined people, who want to be able to test the function not just for a couple examples, but for the entire domain of inputs, which then leads to a lot of interesting but less pragmatic fancy ideas approaching formal verification. I am that person! I am a big fan of property based testing and try to add such tests whenever possible. They serve as "edge-case hunters" and are usually part of my programming loop. As soon as something has been modified, a test suite executes. Writing property tests first and the code later allows you think about the invariants and behaviour of what you are building in a way that regular unit tests just don't.

steveBK123 3y ago

My problem with these is that they are generally written by the developer who wrote the function. So devs code the happy path in their function, and test the happy path in their unit test. Yes, in fact, 2+2 is in fact 4. But, do they test add(2,-2) or add(-2,-2) or add(null,2) or add (0,null) .. etc. Whatever cases they forgot exist and never coded for, never get tested for either. If this is not caught in the initial commit by another dev, then it makes it through to prod happily despite having "unit tests". This is why I think tests where the application is forced to interact with real world data are better because if your app will be broken by negative/0/null, and that state exists in the real world data, it will break in testing. Of course this is all a long dissertation on how to code&test the addition of 2 integers. One of the simplest possible functions, now do this on more complex software. The unit test I think biases later developers to believe the original developer actually coded the software correctly, when to me, in a large number of cases.. it was wrong to begin with, but uncaught.