3 ms·

> How do I know whether my tests are passing because they're properly testing correct code or because they're failing to test incorrect code? One mechanism to

> How do I know whether my tests are passing because they're properly testing correct code or because they're failing to test incorrect code? One mechanism to verify that is by running a mutation testing [0] tool. They are available for many languages; mutmut [1] is a great example for Python. [0] https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing [1] https://mutmut.readthedocs.io https://mutmut.readthedocs.io

deleted 2y ago
[deleted]
layer8 2y ago
That’s basically the approach mentioned in the article’s paragraph starting with “A broader technique I follow is make it work, make it break.”
- bunderbunder 2y ago
  If there's one thing that engineer engineers have considered standard practice for ever and ever, but software engineers seem to still not entirely grok, it's destructive testing. I see this a lot with performance measurement, for example. A team will run small-scale benchmarks, and then try to estimate how a system will scale by linearly extrapolating those results. I don't think I've ever seen it work out well in practice. Nothing scales linearly forever, and there's no reliable way to know when and how it will break down unless you actually push it to the point of breaking down.
  - erik_seaberg 2y ago
    One of the things I like about cloud is that it's relatively easy to spin up an isolated full-scale environment and find out where prod's redline probably is. On-prem hardware might have different bottlenecks.
  - johnnyanmac 2y ago
    I think it's because companies tend to segregate engineering from testing/QA. There's more features to work on, stress testing is Qa's job and those tickets will come later. Engineers will do some basic common tests to make sure it functions as expected, but aren't given the time nor tools to really dig in and ensure it's truly robust. It also reflects the domain. For mission critical code there better be 10 different layers of red lines between development and shipping. For web code, care for stuff like performance and even correctness can fall by the wayside.
    - deleted 2y ago
      [deleted]
pron 2y ago
Yep. And for Java: https://pitest.org https://pitest.org

deleted 2y ago

[deleted]

layer8 2y ago

That’s basically the approach mentioned in the article’s paragraph starting with “A broader technique I follow is make it work, make it break.”

bunderbunder 2y ago
If there's one thing that engineer engineers have considered standard practice for ever and ever, but software engineers seem to still not entirely grok, it's destructive testing. I see this a lot with performance measurement, for example. A team will run small-scale benchmarks, and then try to estimate how a system will scale by linearly extrapolating those results. I don't think I've ever seen it work out well in practice. Nothing scales linearly forever, and there's no reliable way to know when and how it will break down unless you actually push it to the point of breaking down.
- erik_seaberg 2y ago
  One of the things I like about cloud is that it's relatively easy to spin up an isolated full-scale environment and find out where prod's redline probably is. On-prem hardware might have different bottlenecks.
- johnnyanmac 2y ago
  I think it's because companies tend to segregate engineering from testing/QA. There's more features to work on, stress testing is Qa's job and those tickets will come later. Engineers will do some basic common tests to make sure it functions as expected, but aren't given the time nor tools to really dig in and ensure it's truly robust. It also reflects the domain. For mission critical code there better be 10 different layers of red lines between development and shipping. For web code, care for stuff like performance and even correctness can fall by the wayside.
  - deleted 2y ago
    [deleted]

pron 2y ago

Yep. And for Java: https://pitest.org https://pitest.org