5 ms·

I use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive:

I use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive: self.assertTrue(matcher(text), f"Failed: {text}") Before I added the assertion error message, `f"Failed: {text}"`, it was quite difficult to tell WHICH example failed.

profunctor 4y ago
If you’re using pytest you just paramaterize the tests and it tells you the exact failing case. Seems to be a basic feature I would be surprised to know doesn’t exist across almost all commonly used frameworks.
- danuker 4y ago
  We are using unittest. But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests https://docs.python.org/3/library/unittest.html#distinguishi...
  - Dunedan 4y ago
    There is also `parameterized` [1] if you want to do it by using decorators. [1]: https://pypi.org/project/parameterized/ https://pypi.org/project/parameterized/
  - masklinn 4y ago
    > We are using unittest. Unless you need the library style in order to drive it, switch to pytest. Seriously. - assert rewriting is stellar, so much more comfortable than having to find the right assert* method, and tell people they're using the wrong one in reviews - runner is a lot more practical and flexible: nodeids, marks, -k, --lf, --sw, ... - extensions further add flexibility e.g. timeouts, maxfail, xdist (though it has a few drawbacks) - no need for classes (you can have them, mind, but if functions are sufficient, you can use functions) - fixtures > setup/teardown - parameterized tests
  - jjgreen 4y ago
    You mean "subTest", I switched to pytest just to get away from the camel (and it's in the standard library!)
  - ericvsmith 4y ago
    Came here to mention unittest.subtest. It's for exactly this case.
lancebeet 4y ago
That looks like a case where I would use a parameterized test rather than a for loop inside the test.
- dd82 4y ago
  this has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed. Since you already have the object right there, why not do the latter approach?
  - lancebeet 4y ago
    If the setup/teardown is expensive I would do it in reusable fixtures. The reason I wouldn't choose the latter approach is that it would usually be less convenient in the long run. You'd need to replace your asserts with expects to avoid it throwing on the first error (if this isn't what you want), you'll often need to manually add data to the assertion (as GP did) that you would otherwise get for free, and you'll need to look at the assertion error rather than the test case to know what actually failed. This can be quite inconvenient if you e.g. export your test results in a CI/CD pipeline.
    - dd82 4y ago
      normally in a CI/CD pipeline, you'll see what asserts failed in the log output. Github Actions with pytest shows the context of the failed asserts in the log output. TBH, thought this was standard behavior, do you have experience with a CI pipeline that differs? All the other points you make as negatives are all positives for me. Biggest thing is, if you're making this change that alters things so drastically, is that really a good approach. Also, fixtures aren't magic. If you can't scope the fixture to module or session, that means by default it runs in function scope, which would be the same thing as having expensive setup/teardown. And untangling fixtures can be a bgger PITA than untangling unexpected circular imports
olex 4y ago
This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate the others after that.
- danuker 4y ago
  > your example stops on the first one that fails, and does not evaluate the others after that. I don't think this is a big problem; trying to focus on multiple examples at once is difficult. It might be a problem if tests are slow and you are forced to work on all of them at once. But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the caller).
  - olex 4y ago
    Hm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them after seeing all the failures, having ran the test and modified the code a dozen times. Wouldn't it be nicer to see all fails right away and be able to find a solution to all of them, instead of fixing the inputs one-by-one?
    - danuker 4y ago
      In my experience, from doing some TDD Katas[0] and timing myself, I found coding slower and more difficult when focusing on multiple examples at once. I usually even comment out all the failing tests but the first one, after translating a bunch of specifications into tests, so I see the "green" when an example starts working. Maybe it would be easier to grok multiple regex examples than algorithmic ones, but at least for myself, I am skeptical, and I prefer taking them one at a time. [0] - https://kata-log.rocks/tdd https://kata-log.rocks/tdd
deleted 4y ago
[deleted]

profunctor 4y ago

If you’re using pytest you just paramaterize the tests and it tells you the exact failing case. Seems to be a basic feature I would be surprised to know doesn’t exist across almost all commonly used frameworks.

danuker 4y ago
We are using unittest. But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests https://docs.python.org/3/library/unittest.html#distinguishi...
- Dunedan 4y ago
  There is also `parameterized` [1] if you want to do it by using decorators. [1]: https://pypi.org/project/parameterized/ https://pypi.org/project/parameterized/
- masklinn 4y ago
  > We are using unittest. Unless you need the library style in order to drive it, switch to pytest. Seriously. - assert rewriting is stellar, so much more comfortable than having to find the right assert* method, and tell people they're using the wrong one in reviews - runner is a lot more practical and flexible: nodeids, marks, -k, --lf, --sw, ... - extensions further add flexibility e.g. timeouts, maxfail, xdist (though it has a few drawbacks) - no need for classes (you can have them, mind, but if functions are sufficient, you can use functions) - fixtures > setup/teardown - parameterized tests
- jjgreen 4y ago
  You mean "subTest", I switched to pytest just to get away from the camel (and it's in the standard library!)
- ericvsmith 4y ago
  Came here to mention unittest.subtest. It's for exactly this case.

lancebeet 4y ago

That looks like a case where I would use a parameterized test rather than a for loop inside the test.

dd82 4y ago
this has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed. Since you already have the object right there, why not do the latter approach?
- lancebeet 4y ago
  If the setup/teardown is expensive I would do it in reusable fixtures. The reason I wouldn't choose the latter approach is that it would usually be less convenient in the long run. You'd need to replace your asserts with expects to avoid it throwing on the first error (if this isn't what you want), you'll often need to manually add data to the assertion (as GP did) that you would otherwise get for free, and you'll need to look at the assertion error rather than the test case to know what actually failed. This can be quite inconvenient if you e.g. export your test results in a CI/CD pipeline.
  - dd82 4y ago
    normally in a CI/CD pipeline, you'll see what asserts failed in the log output. Github Actions with pytest shows the context of the failed asserts in the log output. TBH, thought this was standard behavior, do you have experience with a CI pipeline that differs? All the other points you make as negatives are all positives for me. Biggest thing is, if you're making this change that alters things so drastically, is that really a good approach. Also, fixtures aren't magic. If you can't scope the fixture to module or session, that means by default it runs in function scope, which would be the same thing as having expensive setup/teardown. And untangling fixtures can be a bgger PITA than untangling unexpected circular imports

olex 4y ago

This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate the others after that.

danuker 4y ago
> your example stops on the first one that fails, and does not evaluate the others after that. I don't think this is a big problem; trying to focus on multiple examples at once is difficult. It might be a problem if tests are slow and you are forced to work on all of them at once. But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the caller).
- olex 4y ago
  Hm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them after seeing all the failures, having ran the test and modified the code a dozen times. Wouldn't it be nicer to see all fails right away and be able to find a solution to all of them, instead of fixing the inputs one-by-one?
  - danuker 4y ago
    In my experience, from doing some TDD Katas[0] and timing myself, I found coding slower and more difficult when focusing on multiple examples at once. I usually even comment out all the failing tests but the first one, after translating a bunch of specifications into tests, so I see the "green" when an example starts working. Maybe it would be easier to grok multiple regex examples than algorithmic ones, but at least for myself, I am skeptical, and I prefer taking them one at a time. [0] - https://kata-log.rocks/tdd https://kata-log.rocks/tdd

deleted 4y ago

[deleted]