3 ms·

Tests should be optimized to find bugs. Tests that you have already run have a lower chance of doing that (they only find regressions); tests with novel inputs

Tests should be optimized to find bugs. Tests that you have already run have a lower chance of doing that (they only find regressions); tests with novel inputs are preferred. And since writing tests manually is so expensive, this means automatic test input generation. How do you determine if such tests pass? Properties. In practice, property based testing fails because the organization is not actually interested in delivering correct code. "This bug will never happen in practice so we won't fix it." "If we fix this, we may change some incorrect behavior some customer is depending on." And once that happens, PBT is useless, because it will keep finding that "don't fix" bug over and over.

chriswarbo 1y ago
Your last paragraph is using terms like "correct", "fix" and "bug" as if they're absolute, when they're actually relative to some sort of spec (whether formal or informal, written or vibes-based, etc.). If the organisation controls the spec, then it can be perfectly reasonable for them to update that spec to e.g. allow certain behaviours that previously would have been considered bugs. In that case, we update the properties to reflect the new spec.
- pfdietz 1y ago
  The spec becomes "don't break important customers code". How could one possibly formalize that?
  - chriswarbo 1y ago
    The same way one formalises the implementation (AKA writing code): iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc.
    - pfdietz 1y ago
      That makes no sense. How do you know what bugs the customer depends on, especially if you don't have access to their code? What's more PBT doesn't depend on having a spec, just on having some properties that hold. So you very possibly didn't have a spec to start with.
      - chriswarbo 1y ago
        > How do you know what bugs the customer depends on, especially if you don't have access to their code? You don't know, but you can ask, you can make educated guesses, etc. As I said, we do this stuff iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc. That's what most programming is. > What's more PBT doesn't depend on having a spec, just on having some properties that hold I'd say that "having some properties that hold" certainly counts as "some sort of spec (whether formal or informal, written or vibes-based, etc.)". > So you very possibly didn't have a spec to start with. There's always "some sort of spec"; even if it starts as vague as "let's try to make some money using computers".

chriswarbo 1y ago

Your last paragraph is using terms like "correct", "fix" and "bug" as if they're absolute, when they're actually relative to some sort of spec (whether formal or informal, written or vibes-based, etc.). If the organisation controls the spec, then it can be perfectly reasonable for them to update that spec to e.g. allow certain behaviours that previously would have been considered bugs. In that case, we update the properties to reflect the new spec.

pfdietz 1y ago
The spec becomes "don't break important customers code". How could one possibly formalize that?
- chriswarbo 1y ago
  The same way one formalises the implementation (AKA writing code): iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc.
  - pfdietz 1y ago
    That makes no sense. How do you know what bugs the customer depends on, especially if you don't have access to their code? What's more PBT doesn't depend on having a spec, just on having some properties that hold. So you very possibly didn't have a spec to start with.
    - chriswarbo 1y ago
      > How do you know what bugs the customer depends on, especially if you don't have access to their code? You don't know, but you can ask, you can make educated guesses, etc. As I said, we do this stuff iteratively, on a best-effort basis, using one's own knowledge and experience, with input and feedback from colleagues and stakeholders, etc. That's what most programming is. > What's more PBT doesn't depend on having a spec, just on having some properties that hold I'd say that "having some properties that hold" certainly counts as "some sort of spec (whether formal or informal, written or vibes-based, etc.)". > So you very possibly didn't have a spec to start with. There's always "some sort of spec"; even if it starts as vague as "let's try to make some money using computers".