4 ms·

I don't see how a classical unit test would have catched this, as all other components are mocked away. Regression tests on the other hand can only catch error

by wolfi1 5y ago

I don't see how a classical unit test would have catched this, as all other components are mocked away. Regression tests on the other hand can only catch errors that already occurred. I#m not sure that was the case.

Izkata 5y ago
The idea is to write a regression test now, to catch it if it's accidentally reintroduced by a future developer who knows nothing about this bug.
- pydry 5y ago
  Still not something a unit test can really catch. An end to end test maybe (run multiple times coz it's a race condition). I've seen unit tests that try to reproduce race conditions. They're naive mirrors of the code that break as soon as you even think about changing the implementation. They're kind of pointless.
  - mnahkies 5y ago
    Personally I've found that the value in the tests you are describing is in proving that you've fixed the issue. When you have a race condition that is difficult to reason about or reproduce, if you can distil it down to a test case that consistently fails until you fix it then you can be comfortable that your solution holds - yes it may be of little value going forward but it can be a very valuable part of the process. It can also serve as documentation going forward, as whilst it may be brittle, when does fail hopefully the person will check the description/blame and instantly find the context as to why it exists, before using their judgement as to whether it is still valid or not. (Rather than changing something that looks odd and not realising that they've re-introduced the edge case)
    - aetherspawn 5y ago
      Yes, I think the most value comes from reminding someone that the problem exists in say 2 years time. At work the other day I refactored a major system and discovered several completely left of field legacy behaviours because of regression tests that didn’t seem to make sense (“Why would a sane person want THIS to happen? Eh? Ooooh”)
    - pydry 5y ago
      I don't find that this type of test provides any value. It's a hollow reflection of the code. At best it provides some sort of confidence that the code you intended to write was indeed the code you wrote. End to end tests do provide a lot of value, though, both in validating that your fix was correct and preventing regressions. They require a lot of up front investment in tooling, however.
      - mnahkies 5y ago
        I prefer to lean heavily on e2e/integration tests as much as possible, because I've found most bugs arise from the boundaries between discreet systems, particularly when they are developed by different teams. However the point I was trying to make was more around super specific scenarios where a somewhat artificial test can highlight the possibility and prove that it's correctly implemented. As well as showing future readers that the code wasn't making an arbitrary decision but it was desired behaviour. For example it's difficult to simulate rare/transitive failure states in e2e testing, but unit tests you can easily force these, and verify that they are handled correctly. One thing that is tangential but springs to mind is when there are log statements that a monitoring system is observing - in these cases I'd add a unit test for that log line being emitted, not because I'm worried that the code doesn't work, but because I'm worried someone will remove it in future not realising that it was the backbone of some monitor
        nitrogen 5y ago
        I'm worried someone will remove it in future not realising that it was the backbone of some monitor It's for this reason that defining a separate method for monitors might be a good idea. Then it's clear this isn't just someone's overeager noise, and the method could be changed to send data directly to a monitoring system in the future if the logging and monitoring stack changes.
londons_explore 5y ago
The ideal tests are not strictly unit or integration tests, but have a suitable mixture of both. They also run with sanitisers which can detect races and undefined behaviour. And really good tooling also does formal verification to try and find any codepaths or inputs that could lead to an assertion failure. I think any of the above 3 things would have had a reasonable chance of catching this issue.

by wolfi1 5y ago

Izkata 5y ago

The idea is to write a regression test now, to catch it if it's accidentally reintroduced by a future developer who knows nothing about this bug.

pydry 5y ago
Still not something a unit test can really catch. An end to end test maybe (run multiple times coz it's a race condition). I've seen unit tests that try to reproduce race conditions. They're naive mirrors of the code that break as soon as you even think about changing the implementation. They're kind of pointless.
- mnahkies 5y ago
  Personally I've found that the value in the tests you are describing is in proving that you've fixed the issue. When you have a race condition that is difficult to reason about or reproduce, if you can distil it down to a test case that consistently fails until you fix it then you can be comfortable that your solution holds - yes it may be of little value going forward but it can be a very valuable part of the process. It can also serve as documentation going forward, as whilst it may be brittle, when does fail hopefully the person will check the description/blame and instantly find the context as to why it exists, before using their judgement as to whether it is still valid or not. (Rather than changing something that looks odd and not realising that they've re-introduced the edge case)
  - aetherspawn 5y ago
    Yes, I think the most value comes from reminding someone that the problem exists in say 2 years time. At work the other day I refactored a major system and discovered several completely left of field legacy behaviours because of regression tests that didn’t seem to make sense (“Why would a sane person want THIS to happen? Eh? Ooooh”)
  - pydry 5y ago
    I don't find that this type of test provides any value. It's a hollow reflection of the code. At best it provides some sort of confidence that the code you intended to write was indeed the code you wrote. End to end tests do provide a lot of value, though, both in validating that your fix was correct and preventing regressions. They require a lot of up front investment in tooling, however.
    - mnahkies 5y ago
      I prefer to lean heavily on e2e/integration tests as much as possible, because I've found most bugs arise from the boundaries between discreet systems, particularly when they are developed by different teams. However the point I was trying to make was more around super specific scenarios where a somewhat artificial test can highlight the possibility and prove that it's correctly implemented. As well as showing future readers that the code wasn't making an arbitrary decision but it was desired behaviour. For example it's difficult to simulate rare/transitive failure states in e2e testing, but unit tests you can easily force these, and verify that they are handled correctly. One thing that is tangential but springs to mind is when there are log statements that a monitoring system is observing - in these cases I'd add a unit test for that log line being emitted, not because I'm worried that the code doesn't work, but because I'm worried someone will remove it in future not realising that it was the backbone of some monitor
      - nitrogen 5y ago
        I'm worried someone will remove it in future not realising that it was the backbone of some monitor It's for this reason that defining a separate method for monitors might be a good idea. Then it's clear this isn't just someone's overeager noise, and the method could be changed to send data directly to a monitoring system in the future if the logging and monitoring stack changes.

londons_explore 5y ago

The ideal tests are not strictly unit or integration tests, but have a suitable mixture of both. They also run with sanitisers which can detect races and undefined behaviour. And really good tooling also does formal verification to try and find any codepaths or inputs that could lead to an assertion failure. I think any of the above 3 things would have had a reasonable chance of catching this issue.