3 ms·

> When you have no tests your problems go away because you don’t see any test failures. The flip side of this is the quote that "tests can show the presence of

> When you have no tests your problems go away because you don’t see any test failures. The flip side of this is the quote that "tests can show the presence of bugs, but never their absence". It better fits my experience here; every few months I'd find a new bug and diligently write a test for it. But then there was a new bug in a few months, discovered by someone in the first 10 minutes of using it. I'm sure I have bugs to discover in the new version. But the data structures I chose make many of the old tests obsolete by construction. So I'm hopeful that I'm a few bugs away from something fairly stable at least for idle use. Tests are definitely invaluable for a large team constantly making changes to a codebase. But here I'm trying to build something with a frozen feature set.

monkpit 2y ago
If your tests break or go away when your implementation changes, aren’t those bad tests by definition?
- akkartik 2y ago
  I have two answers: 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. I'd love to have better programmers show me how I can make my tests better. The code is out there. 2. Even if I have good tests "by definition", a radical rewrite might make old tests look like "assert(2x1 == 2), assert (2x2 == 4)". Tests exist in a context, and radically changing the context can change the tests you need. --- This is not in OP, but I do also have a problem of brittle tests in my editor. In this case I need to test a word-wrapping algorithm. This depends intimately on pixel-precise details of the font. I'd love for better programmers than me to suggest how I can write tests that are robust and also self-evidently correct without magic constants that don't communicate anything to the reader. "Failure: 'x' started at x=67 rather than x=68." Reader's thought: "Why is this a problem?" etc. Comments appreciated on https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests.lua https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t.... The summary at https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t... might help orient readers.
  - AdieuToLogic 2y ago
    >> If your tests break or go away when your implementation changes, aren’t those bad tests by definition? > 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. Good and bad are forms of judgement, so let's eschew judgement for the purposes of this reply :-). > I'd love to have better programmers show me how I can make my tests better. Better is also a form of judgement and, so, I will not claim I am or am not. What I will claim to do is offer my perspective regarding: > This is not in OP, but I do also have a problem of brittle tests in my editor. Unfortunately, brittle tests are the result of being overly specific. This is usually due to tests enforcing implementation knowledge instead of verifying a usage contract. The example assertions above are good examples of this (consider "assert (canMultiply ...)" as a conceptual alternative). What helps mitigate this situation is use of key abstractions relevant to the problem domain along with insulating implementation logic (note that this is not the same as encapsulation, as insulation makes the implementation opaque to collaborators). In your post, you posit: > Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. I suggest they serve a purpose beyond when "in unfamiliar terrain." Specifically, these tools provide confidence in system correctness in the presence of change. They also allow people to reason about the nature of a system, including your future-self. Perhaps most relevant to "brittle tests" are the first two you enumerated - types and abstractions. Having them can allow test suites to be defined against the public contract they provide. And as you rightly point out in your post, having the wrong ones can lead to problems. The trick is, when incorrect types and/or abstractions are identified, this presents an opportunity to refine understanding of the problem domain and improve key abstractions/collaborations accordingly. Functional testing[0] is really handy to do this fairly rapidly when employed early and often. HTH 0 - https://en.wikipedia.org/wiki/Functional_testing https://en.wikipedia.org/wiki/Functional_testing
- Jtsummers 2y ago
  A lot of tests don't survive implementation changes, that doesn't make them "bad tests by definition". It means their value came and went. Think of it like scaffolding. You need it for a time, then the time is up, and it's removed. That doesn't make it bad, it was still necessary (or at least useful) for a time. When there's an implementation change you'll likely end up discarding a fair number of unit tests and creating new ones that reflect the new implementation details. That's just natural.
  - seanmcdirmid 2y ago
    A lot of tests, especially unit tests, are just change detectors and get updated/go away when change happens, that is just WAI. It is fairly hard to write non change detection tests, it requires for you to really reason abstractly about the contract of your module, or to write integration tests that are moving a bunch of things at once.
    - zem 2y ago
      small, fine-grained black box tests can be really good for this. in my last project, a type checker, the vast majority of the test suite was code snippets and assertions about expected errors the checker needed to catch, and it was an invaluable aid when making complex changes to the implementation.
      - seanmcdirmid 2y ago
        Anything that transforms or processes text, like a compiler or type checker, is pretty easy to test. You get into trouble with user interfaces, however.
    - watwut 2y ago
      If that is the case too often, I ditch them and write integration tests for that part.
  - codr7 2y ago
    Yeah, especially when you're exploring new ground. Unit tests are awesome for fleshing out APIs; but once the fundamentals are in place, the tests no longer add any value.
creesch 2y ago
Automated tests ideally don't entirely replace manually executed tests. What they do replace is repetitive regression tests that don't need to be executed manually. In an ideal world this opens up room for exploratory testing where someone goes "off-script" and focuses specifically on those areas that are not covered by your automated tests. The thing is that automated tests aren't really tests, even though we call them that. They are automated checks at specified points, so they only check the outcome at those point in time. So yeah, they are also completely blind from the sort of thing a human* might easily spot while using the application. *Just to be ahead of the AI bros, we are not there yet, hold your horses.

monkpit 2y ago

If your tests break or go away when your implementation changes, aren’t those bad tests by definition?

akkartik 2y ago
I have two answers: 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. I'd love to have better programmers show me how I can make my tests better. The code is out there. 2. Even if I have good tests "by definition", a radical rewrite might make old tests look like "assert(2x1 == 2), assert (2x2 == 4)". Tests exist in a context, and radically changing the context can change the tests you need. --- This is not in OP, but I do also have a problem of brittle tests in my editor. In this case I need to test a word-wrapping algorithm. This depends intimately on pixel-precise details of the font. I'd love for better programmers than me to suggest how I can write tests that are robust and also self-evidently correct without magic constants that don't communicate anything to the reader. "Failure: 'x' started at x=67 rather than x=68." Reader's thought: "Why is this a problem?" etc. Comments appreciated on https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests.lua https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t.... The summary at https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_tests https://git.sr.ht/~akkartik/lines.love/tree/main/item/text_t... might help orient readers.
- AdieuToLogic 2y ago
  >> If your tests break or go away when your implementation changes, aren’t those bad tests by definition? > 1. Yes. To the same extent that we are all bad people by definition, made of base material and unworthy urges. Good and bad are forms of judgement, so let's eschew judgement for the purposes of this reply :-). > I'd love to have better programmers show me how I can make my tests better. Better is also a form of judgement and, so, I will not claim I am or am not. What I will claim to do is offer my perspective regarding: > This is not in OP, but I do also have a problem of brittle tests in my editor. Unfortunately, brittle tests are the result of being overly specific. This is usually due to tests enforcing implementation knowledge instead of verifying a usage contract. The example assertions above are good examples of this (consider "assert (canMultiply ...)" as a conceptual alternative). What helps mitigate this situation is use of key abstractions relevant to the problem domain along with insulating implementation logic (note that this is not the same as encapsulation, as insulation makes the implementation opaque to collaborators). In your post, you posit: > Types, abstractions, tests, versions, state machines, immutability, formal analysis, all these are tools available to us in unfamiliar terrain. I suggest they serve a purpose beyond when "in unfamiliar terrain." Specifically, these tools provide confidence in system correctness in the presence of change. They also allow people to reason about the nature of a system, including your future-self. Perhaps most relevant to "brittle tests" are the first two you enumerated - types and abstractions. Having them can allow test suites to be defined against the public contract they provide. And as you rightly point out in your post, having the wrong ones can lead to problems. The trick is, when incorrect types and/or abstractions are identified, this presents an opportunity to refine understanding of the problem domain and improve key abstractions/collaborations accordingly. Functional testing[0] is really handy to do this fairly rapidly when employed early and often. HTH 0 - https://en.wikipedia.org/wiki/Functional_testing https://en.wikipedia.org/wiki/Functional_testing
Jtsummers 2y ago
A lot of tests don't survive implementation changes, that doesn't make them "bad tests by definition". It means their value came and went. Think of it like scaffolding. You need it for a time, then the time is up, and it's removed. That doesn't make it bad, it was still necessary (or at least useful) for a time. When there's an implementation change you'll likely end up discarding a fair number of unit tests and creating new ones that reflect the new implementation details. That's just natural.
- seanmcdirmid 2y ago
  A lot of tests, especially unit tests, are just change detectors and get updated/go away when change happens, that is just WAI. It is fairly hard to write non change detection tests, it requires for you to really reason abstractly about the contract of your module, or to write integration tests that are moving a bunch of things at once.
  - zem 2y ago
    small, fine-grained black box tests can be really good for this. in my last project, a type checker, the vast majority of the test suite was code snippets and assertions about expected errors the checker needed to catch, and it was an invaluable aid when making complex changes to the implementation.
    - seanmcdirmid 2y ago
      Anything that transforms or processes text, like a compiler or type checker, is pretty easy to test. You get into trouble with user interfaces, however.
  - watwut 2y ago
    If that is the case too often, I ditch them and write integration tests for that part.
- codr7 2y ago
  Yeah, especially when you're exploring new ground. Unit tests are awesome for fleshing out APIs; but once the fundamentals are in place, the tests no longer add any value.

creesch 2y ago

Automated tests ideally don't entirely replace manually executed tests. What they do replace is repetitive regression tests that don't need to be executed manually. In an ideal world this opens up room for exploratory testing where someone goes "off-script" and focuses specifically on those areas that are not covered by your automated tests. The thing is that automated tests aren't really tests, even though we call them that. They are automated checks at specified points, so they only check the outcome at those point in time. So yeah, they are also completely blind from the sort of thing a human* might easily spot while using the application. *Just to be ahead of the AI bros, we are not there yet, hold your horses.