4 ms·

Working in Python, my biggest wat is that a test can cause a failed assertion in the code and still succeed (if AssertionError is caught somewhere upwards in th

Working in Python, my biggest wat is that a test can cause a failed assertion in the code and still succeed (if AssertionError is caught somewhere upwards in the code, which will happen at any `except Exception`).

eru 11y ago
That's why you should inject defects into your code, to see whether your tests catch them. There are tools to do that automatically.
- dllthomas 11y ago
  I feel like I may be missing some of what you're trying to communicate. I agree that injecting defects can help test a test suite and that this can be a good practice. I don't quite see the relevance to the issue I described. Edited to add - thanks for responding, by the way. I'm confused by all the down-votes.
  - eru 11y ago
    Don't put too much store in early downvotes. It still usually balances itself out. I still don't quite get what you meant.
    - dllthomas 11y ago
      Assertions in code are for documenting and checking things that can't happen unless there's a bug. This helps programmers reason about the code as they read it, and helps us find errors closer to the defect that caused them. If a test case triggers an assertion violation down in some method, there is a bug. That should break the test, so that I'm told about the bug, and can investigate and fix it. If there happens to be a `try...except Exception` anywhere in the stack above that method, the test never learns that an assertion fired and might even pass. This makes every test less useful than it could be.
      - eru 11y ago
        Oh, that's why you inject defects, and if that injection doesn't cause your test to fail, eg because of a try..except Exception, you know that your test is wrong. (Then you investigate and hopefully remove the catch-all.)
        dllthomas 11y ago
        But catch-alls of that form can be correct. This isn't terribly relevant to narrow unit tests, where I should be able to know what I expect to have happened and presumably if an assertion pops it won't have happened, but it makes larger scale fuzzing substantially less useful.
ketralnis 11y ago
I've hit this too, it would be really nice if AssertError didn't inherit from Exception. There's a whole Exception object hierarchy but it seems like none of the parents of Exception are actually useable for anything
- dllthomas 11y ago
  That would partially solve this problem (it's far rarer to see bare excepts or `except BaseException`), but could cause others. The question there should be what is (or should be) intended when someone says `except Exception` - it's not clear to me whether that should or should not include AssertionError. A more direct solution might be to have a means of storing off every assertion failure into a (clearable) list. The test harness could then check that list at the end of each test. If a particular test should succeed after triggering an assertion failure, it can check the list to make sure it triggered exactly the expected failures and then clear it.
  - ketralnis 11y ago
    > The question there should be what is (or should be) intended when someone says `except Exception` - it's not clear to me whether that should or should not include AssertionError Right, I guess I'm not suggesting changing the language today to do this. Rather, I'm suggesting that it would be better if it were already done that way from the beginning. If it were designed that way from the start then there would be no question about the intention because it would just be part of the definition of Exception
    - dllthomas 11y ago
      "If it were designed that way from the start then there would be no question about the intention because it would just be part of the definition of Exception" Heh, doesn't that argument apply currently? I think I was trying to ask, "when someone fails to think deeply enough about it, what is likely to give the 'correct' results"?
  - masklinn 11y ago

eru 11y ago

That's why you should inject defects into your code, to see whether your tests catch them. There are tools to do that automatically.

dllthomas 11y ago
I feel like I may be missing some of what you're trying to communicate. I agree that injecting defects can help test a test suite and that this can be a good practice. I don't quite see the relevance to the issue I described. Edited to add - thanks for responding, by the way. I'm confused by all the down-votes.
- eru 11y ago
  Don't put too much store in early downvotes. It still usually balances itself out. I still don't quite get what you meant.
  - dllthomas 11y ago
    Assertions in code are for documenting and checking things that can't happen unless there's a bug. This helps programmers reason about the code as they read it, and helps us find errors closer to the defect that caused them. If a test case triggers an assertion violation down in some method, there is a bug. That should break the test, so that I'm told about the bug, and can investigate and fix it. If there happens to be a `try...except Exception` anywhere in the stack above that method, the test never learns that an assertion fired and might even pass. This makes every test less useful than it could be.
    - eru 11y ago
      Oh, that's why you inject defects, and if that injection doesn't cause your test to fail, eg because of a try..except Exception, you know that your test is wrong. (Then you investigate and hopefully remove the catch-all.)
      - dllthomas 11y ago
        But catch-alls of that form can be correct. This isn't terribly relevant to narrow unit tests, where I should be able to know what I expect to have happened and presumably if an assertion pops it won't have happened, but it makes larger scale fuzzing substantially less useful.

ketralnis 11y ago

I've hit this too, it would be really nice if AssertError didn't inherit from Exception. There's a whole Exception object hierarchy but it seems like none of the parents of Exception are actually useable for anything

dllthomas 11y ago
That would partially solve this problem (it's far rarer to see bare excepts or `except BaseException`), but could cause others. The question there should be what is (or should be) intended when someone says `except Exception` - it's not clear to me whether that should or should not include AssertionError. A more direct solution might be to have a means of storing off every assertion failure into a (clearable) list. The test harness could then check that list at the end of each test. If a particular test should succeed after triggering an assertion failure, it can check the list to make sure it triggered exactly the expected failures and then clear it.
- ketralnis 11y ago
  > The question there should be what is (or should be) intended when someone says `except Exception` - it's not clear to me whether that should or should not include AssertionError Right, I guess I'm not suggesting changing the language today to do this. Rather, I'm suggesting that it would be better if it were already done that way from the beginning. If it were designed that way from the start then there would be no question about the intention because it would just be part of the definition of Exception
  - dllthomas 11y ago
    "If it were designed that way from the start then there would be no question about the intention because it would just be part of the definition of Exception" Heh, doesn't that argument apply currently? I think I was trying to ask, "when someone fails to think deeply enough about it, what is likely to give the 'correct' results"?
- masklinn 11y ago