3 ms·

could you give a concrete example of what you mean by this?

maxbond 2y ago
Not GP but when I feel like I'm going crazy I insert an "assert False" test into my test suite. It's a good way to reveal when you're testing a cached version of your code for some reason (for instance integration tests using Docker Compose that aren't picking up your changes because you've forgotten to specify --build or your .dockerignore is misconfigured). But I delete it when I'm done.
singron 2y ago
We once accidentally made a change to a python project test suite that caused it to successfully run none of the tests. Then we broke some stuff but the tests kept "passing". It's a little difficult to productionize an always_fail test since you do actually want the test suite to succeed. You could affirmatively test that you have non-zero passing tests, which is I think what we did. If you have an always_fail test, you could check that that's your only failure, but you have to be super careful that your test suite doesn't stop after a failure.
- maxbond 2y ago
  Maybe you could ignore that test by default, and then write a shell script to run your tests in two stages. First you run only the should-fail test(s) and assert that they fail. Then you can run your actual tests.
  - SoftTalker 2y ago
    Sounds like the old George Carlin one-liner. Or maybe it's a two-liner: The following statement is true. The preceeding statement is false.
    - robotresearcher 2y ago
      Even older than George Carlin. The Liar Paradox is documented from at least 400BC. https://en.m.wikipedia.org/wiki/Liar_paradox https://en.m.wikipedia.org/wiki/Liar_paradox
      - maxbond 2y ago
        I have to imagine it's about as old as propositional logic (so, as old as the hills). I most closely associate it with Gödel and his work on incompleteness.
- marcosdumay 2y ago
  > We once accidentally made a change to a python project test suite that caused it to successfully run none of the tests. That shouldn't be an easy mistake to make. Your test code should be clearly marked, and better if slightly separated from the rest of the code. Also, there should be some feedback about the amount of tests that run. And yeah, I know Python doesn't help you make those things.

Not GP but when I feel like I'm going crazy I insert an "assert False" test into my test suite. It's a good way to reveal when you're testing a cached version of your code for some reason (for instance integration tests using Docker Compose that aren't picking up your changes because you've forgotten to specify --build or your .dockerignore is misconfigured). But I delete it when I'm done.

singron 2y ago

We once accidentally made a change to a python project test suite that caused it to successfully run none of the tests. Then we broke some stuff but the tests kept "passing". It's a little difficult to productionize an always_fail test since you do actually want the test suite to succeed. You could affirmatively test that you have non-zero passing tests, which is I think what we did. If you have an always_fail test, you could check that that's your only failure, but you have to be super careful that your test suite doesn't stop after a failure.

maxbond 2y ago
Maybe you could ignore that test by default, and then write a shell script to run your tests in two stages. First you run only the should-fail test(s) and assert that they fail. Then you can run your actual tests.
- SoftTalker 2y ago
  Sounds like the old George Carlin one-liner. Or maybe it's a two-liner: The following statement is true. The preceeding statement is false.
  - robotresearcher 2y ago
    Even older than George Carlin. The Liar Paradox is documented from at least 400BC. https://en.m.wikipedia.org/wiki/Liar_paradox https://en.m.wikipedia.org/wiki/Liar_paradox
    - maxbond 2y ago
      I have to imagine it's about as old as propositional logic (so, as old as the hills). I most closely associate it with Gödel and his work on incompleteness.
marcosdumay 2y ago
> We once accidentally made a change to a python project test suite that caused it to successfully run none of the tests. That shouldn't be an easy mistake to make. Your test code should be clearly marked, and better if slightly separated from the rest of the code. Also, there should be some feedback about the amount of tests that run. And yeah, I know Python doesn't help you make those things.