7 ms·

Personally, I've come to really dislike test names like "test_refresh_failure". They tell you what component is being tested, but not what kind of behavior is e

Personally, I've come to really dislike test names like "test_refresh_failure". They tell you what component is being tested, but not what kind of behavior is expected. Which can lead to a whole lot unnecessary confusion (or bugs) when you're trying to maintain a test whose implementation is difficult to understand, or if you're not sure it's asserting the right things. It also encourages tests that do too much. If the test is named "test_refresh", well, it says right there in the name that it's a test for any old generic refresh behavior. So why not just keep dumping assertions in there? I'm much more happy with names like, "test_displays_error_message_when_refresh_times_out". Right there, you know exactly what's being verified, because it's been written in plain English. Which means you can recognize a buggy test implementation when you see it, and you know what behavior you're supposed to be restoring if the test breaks, and you are prepared to recognize an erroneously passing test, and all sorts of useful things like that.

jxub 6y ago
I think that behave (https://behave.readthedocs.io/en/stable/ https://behave.readthedocs.io/en/stable/) is more useful for testing these more real-life usecases. I tend to have a `test_my_function` in pytest tests and the more integration and functionality-related testing in the behave tests.
BurningFrog 6y ago
We don't need to put this much responsibility on test names. If there is more to explain, write a few words in a comment.
- exdsq 6y ago
  You don’t see comments in a test report though — maybe have an optional description as part of the framework for more detail along with -v
  - munchbunny 6y ago
    I think this is one of those conventions where your team agrees on one convention, and you just try to follow it consistently. If that's looking at the test in code to find descriptive comments, great. If that's long test names, cool. Just do the same thing consistently.
    - exdsq 6y ago
      True - I think it depends on who uses your test outputs. If PMs and BAs care, having a more verbose output (descriptions, pretty graphs, etc) helps a lot. If it’s just for devs then they can more happily go through the code base.
  - nerdponx 6y ago
    I know that the docstring in Unittest is part of the reported output, and I was pretty sure that it's the same in Pytest.
    - masklinn 6y ago
      I would've thought so but no, pytest will show the docstring in `--collect-only -v` (badly), but it doesn't show any sort of description when running the tests, even in verbose mode (see issue #7005 which doesn't seem to be very popular)
  - masklinn 6y ago
    `unittest` prints the docstring alongside the name of the test in verbose mode (e.g. failure). pytest does not though.
- mumblemumble 6y ago
  I've just never seen the "this belongs in comments" approach work out in practice. Maybe it's something about human psychology. Perhaps things might seem obvious when you've just written the test, so you don't think to comment them. Perhaps code reviewers feel less comfortable saying, "I don't understand this, could you please comment it?" Perhaps it's the simple fact that optional means optional, which means, "You don't have to do it if you aren't in the mood." Regardless of the reason, though, it's a thing I've seen play out so many times that I've become convinced that asking people to do otherwise is spitting into the wind.
  - rbanffy 6y ago
    I really hate, but I can see its virtue, the pylint rule that complains about lacking docstrings. In tests, however, I prefer not to have them as my favorite test runners replace the full name of the test method with its docstring, which makes it a lot harder to find the test in the code.
    - ben509 6y ago
      My only issue is that the rule doesn't give you a good way to annotate that the object is documented by its signature. Sometimes devs are lazy, but the rule doesn't make them not lazy so you get pointless docs like: def matriculate_flange(via: Worble): "Matriculates flange via a Worble."
      - rbanffy 6y ago
        That's more or less why I hate this feature - it encourages lazy docstrings. Right after you named something is probably the worst possible time for you to explain it.
  - BurningFrog 6y ago
    All this is true, but doesn't it apply just as much to writing descriptive test function names? Test function names are less sensitive than regular functions, since they're not explicitly called, but I still don't want to read a_sentence_with_spaces_replaced_by_underscores.
- stinos 6y ago
  It's a choice of course, but I look at test functions like other functions. If you can name a function such that it doesn't need a comment (but also doesn't use 100 characters to do that), I'll gladly take that over a comment. Same like in all other code: if you need comments to explain what it does, it's likely not good enough and/or needs to be put in a function which tells what it does. Comments on why it does stuff the way it does are of course where it's at.
- xapata 6y ago
  s/comment/docstring/
  - BurningFrog 6y ago
    I have yet to understand the point of docstrings. How are they, in practical reality, better than comments?
    - powersnail 6y ago
      The difference is in tooling. Docstrings are collected and displayed in many contexts. The intended purpose is writing “documentation” in the same place as code. Comments are only seen if you open the source file.
      - BurningFrog 6y ago
        OK, I can see how that's useful for a certain workflow. The way we work, it's just a different comment syntax. I do like have a dedicated place for it.
        xapata 6y ago
        Python's builtins, the standard library, and many popular community libraries and tools use docstrings to generate tooling and documentation websites. Describing it as "useful for a certain workflow" is so understated it borders on false. The reverse is more true: docstrings are not useful for a certain workflow, but most Python programmers will find them quite helpful. All Python programmers already benefit from the docstrings written in the libraries they use, which generate tooltips and documentation websites.
    - f00_ 6y ago
      you can access it from object.__doc__, and there is tooling in ide's like pycharm to quick view them to see what a function does, auto generated documentation prior to mypy/type hints it allowed you to document the types of a function
wodenokoto 6y ago
One of the most common test modules in R is called “test that” and you invoke a test by calling a function (rather than defining one) called “test_that”, the first argument is a string containing a description of what you want to test and the second argument is the code you want to test. That way, all your unit tests reads: “test that error message is displayed when refresh times out” etc. I think it’s a really nice way to lay things out and it avoids all the “magic” of some functions being executed by virtue of their name.
darioush 6y ago
have you considered using "test_refresh__failure" instead? makes clear "refresh" is the component being tested and "failure" is a description of the behavior
deleted 6y ago
[deleted]
rowanseymour 6y ago
Sometimes there are practical reasons to avoid this approach and have fewer test methods that test multiple behaviors of a single thing. For example a lot of our projects setup and teardown a database between test methods so a few fatter test methods run a lot faster than a large number of small test methods. We rely on good commenting within the methods to understand what exactly is being checked.
- codethief 6y ago
  In that case, why not use some auxiliary function to load the resources (in your case, the database) and decorate it with functools.cache[0] to avoid the function from getting executed multiple times? Sure, this means re-using resources between multiple tests (which is discouraged for good reasons) but your current test effectively does the same thing, the only difference being that everything is being tested inside one single test function. PS: How come your setup and teardown operations are so expensive in the first place? Why do you even need to set up an entire database? Can't you mock out the database, set up only a few tables or use a lighter database layer? [0]: https://docs.python.org/3/library/functools.html#functools.cache https://docs.python.org/3/library/functools.html#functools.c...
  - rowanseymour 6y ago
    The difference is that it's very obvious in our current tests when the database is being reset. And to be clear it's not like we're testing the entire app in a single test function - our test functions just tend to cover more than one feature. We've also found over the years that if you want to be sure everything works on version X of database system Y, you run your CI tests on that.
codethief 6y ago
This. Like probably most programmers, I tend to read code more than I write code and when I read tests, I want to know immediately what a test does and checks. I don't want to read the test itself, unless I already know it's of interest to me. In my experience, when a unit test has no clear name, this is usually a sign that the test does too much (meaning that it's actually not a unit test anymore) or even comprises multiple tests, leading to all the known bad consequences this has (leaking state between tests, unclear test assumptions etc.)
herpderperator 6y ago
Call me crazy, but I go even more verbose than that. I go full Rspec/BDD mode: class TestAuthenticateMethod: class TestWhenUserExists: class TestWhenPasswordIsValid: def test_it_returns_200(self, app, user): assert ... class TestWhenPasswordIsInvalid: def test_it_returns_403(self, app, user): assert ... class TestWhenUserDoesNotExist: def test_it_returns_403(self, app, user): assert ... It is so worth it. I can look back at my tests 3 months later and know exactly what was intended wthout duplicating explanations with comments. Yes some methods can get a bit long and indented, but it's worth it for me. The other downside is you have to start all classes/methods with "test" but I got used to that very quickly. When you run pytest it's very clear what's being tested and what the expected outcome is since it gives you the full class hierarchy in the output.
- mumblemumble 6y ago
  This idea is both crazy and genius. Thanks for sharing.
- jkaptur 6y ago
  I’ve been working on a codebase like this, and my major complaint is that I’ll see a deeply-nested test and have to scroll waaaaay up to understand the context. I prefer ‘foo_bar_baz’, ‘foo_bar_qux’ to foo <snip 1000 lines> bar <snip 553 lines> baz <snip 243 lines> qux
- BerislavLopac 6y ago
  This is neat, but pytest allows for similar granulation using the right fixtures: E.g. in test_authenticate_method.py: def test_returns_200_on_valid_password(app, user_exists): ... def test_returns_403_on_valid_password(app, user_exists): ... def test_returns_403(app, user_not_exist): ...
BiteCode_dev 6y ago
I usually put that info in the asset: Assert foo(), "foo should bar" So that each failure has a clear meaning. Indeed, i may have several assert in a test.

jxub 6y ago

I think that behave (https://behave.readthedocs.io/en/stable/ https://behave.readthedocs.io/en/stable/) is more useful for testing these more real-life usecases. I tend to have a `test_my_function` in pytest tests and the more integration and functionality-related testing in the behave tests.

BurningFrog 6y ago

We don't need to put this much responsibility on test names. If there is more to explain, write a few words in a comment.

exdsq 6y ago
You don’t see comments in a test report though — maybe have an optional description as part of the framework for more detail along with -v
- munchbunny 6y ago
  I think this is one of those conventions where your team agrees on one convention, and you just try to follow it consistently. If that's looking at the test in code to find descriptive comments, great. If that's long test names, cool. Just do the same thing consistently.
  - exdsq 6y ago
    True - I think it depends on who uses your test outputs. If PMs and BAs care, having a more verbose output (descriptions, pretty graphs, etc) helps a lot. If it’s just for devs then they can more happily go through the code base.
- nerdponx 6y ago
  I know that the docstring in Unittest is part of the reported output, and I was pretty sure that it's the same in Pytest.
  - masklinn 6y ago
    I would've thought so but no, pytest will show the docstring in `--collect-only -v` (badly), but it doesn't show any sort of description when running the tests, even in verbose mode (see issue #7005 which doesn't seem to be very popular)
- masklinn 6y ago
  `unittest` prints the docstring alongside the name of the test in verbose mode (e.g. failure). pytest does not though.
mumblemumble 6y ago
I've just never seen the "this belongs in comments" approach work out in practice. Maybe it's something about human psychology. Perhaps things might seem obvious when you've just written the test, so you don't think to comment them. Perhaps code reviewers feel less comfortable saying, "I don't understand this, could you please comment it?" Perhaps it's the simple fact that optional means optional, which means, "You don't have to do it if you aren't in the mood." Regardless of the reason, though, it's a thing I've seen play out so many times that I've become convinced that asking people to do otherwise is spitting into the wind.
- rbanffy 6y ago
  I really hate, but I can see its virtue, the pylint rule that complains about lacking docstrings. In tests, however, I prefer not to have them as my favorite test runners replace the full name of the test method with its docstring, which makes it a lot harder to find the test in the code.
  - ben509 6y ago
    My only issue is that the rule doesn't give you a good way to annotate that the object is documented by its signature. Sometimes devs are lazy, but the rule doesn't make them not lazy so you get pointless docs like: def matriculate_flange(via: Worble): "Matriculates flange via a Worble."
    - rbanffy 6y ago
      That's more or less why I hate this feature - it encourages lazy docstrings. Right after you named something is probably the worst possible time for you to explain it.
- BurningFrog 6y ago
  All this is true, but doesn't it apply just as much to writing descriptive test function names? Test function names are less sensitive than regular functions, since they're not explicitly called, but I still don't want to read a_sentence_with_spaces_replaced_by_underscores.
stinos 6y ago
It's a choice of course, but I look at test functions like other functions. If you can name a function such that it doesn't need a comment (but also doesn't use 100 characters to do that), I'll gladly take that over a comment. Same like in all other code: if you need comments to explain what it does, it's likely not good enough and/or needs to be put in a function which tells what it does. Comments on why it does stuff the way it does are of course where it's at.
xapata 6y ago
s/comment/docstring/
- BurningFrog 6y ago
  I have yet to understand the point of docstrings. How are they, in practical reality, better than comments?
  - powersnail 6y ago
    The difference is in tooling. Docstrings are collected and displayed in many contexts. The intended purpose is writing “documentation” in the same place as code. Comments are only seen if you open the source file.
    - BurningFrog 6y ago
      OK, I can see how that's useful for a certain workflow. The way we work, it's just a different comment syntax. I do like have a dedicated place for it.
      - xapata 6y ago
        Python's builtins, the standard library, and many popular community libraries and tools use docstrings to generate tooling and documentation websites. Describing it as "useful for a certain workflow" is so understated it borders on false. The reverse is more true: docstrings are not useful for a certain workflow, but most Python programmers will find them quite helpful. All Python programmers already benefit from the docstrings written in the libraries they use, which generate tooltips and documentation websites.
  - f00_ 6y ago
    you can access it from object.__doc__, and there is tooling in ide's like pycharm to quick view them to see what a function does, auto generated documentation prior to mypy/type hints it allowed you to document the types of a function

wodenokoto 6y ago

One of the most common test modules in R is called “test that” and you invoke a test by calling a function (rather than defining one) called “test_that”, the first argument is a string containing a description of what you want to test and the second argument is the code you want to test. That way, all your unit tests reads: “test that error message is displayed when refresh times out” etc. I think it’s a really nice way to lay things out and it avoids all the “magic” of some functions being executed by virtue of their name.

darioush 6y ago

have you considered using "test_refresh__failure" instead? makes clear "refresh" is the component being tested and "failure" is a description of the behavior

deleted 6y ago

[deleted]

rowanseymour 6y ago

Sometimes there are practical reasons to avoid this approach and have fewer test methods that test multiple behaviors of a single thing. For example a lot of our projects setup and teardown a database between test methods so a few fatter test methods run a lot faster than a large number of small test methods. We rely on good commenting within the methods to understand what exactly is being checked.

codethief 6y ago
In that case, why not use some auxiliary function to load the resources (in your case, the database) and decorate it with functools.cache[0] to avoid the function from getting executed multiple times? Sure, this means re-using resources between multiple tests (which is discouraged for good reasons) but your current test effectively does the same thing, the only difference being that everything is being tested inside one single test function. PS: How come your setup and teardown operations are so expensive in the first place? Why do you even need to set up an entire database? Can't you mock out the database, set up only a few tables or use a lighter database layer? [0]: https://docs.python.org/3/library/functools.html#functools.cache https://docs.python.org/3/library/functools.html#functools.c...
- rowanseymour 6y ago
  The difference is that it's very obvious in our current tests when the database is being reset. And to be clear it's not like we're testing the entire app in a single test function - our test functions just tend to cover more than one feature. We've also found over the years that if you want to be sure everything works on version X of database system Y, you run your CI tests on that.

codethief 6y ago

This. Like probably most programmers, I tend to read code more than I write code and when I read tests, I want to know immediately what a test does and checks. I don't want to read the test itself, unless I already know it's of interest to me. In my experience, when a unit test has no clear name, this is usually a sign that the test does too much (meaning that it's actually not a unit test anymore) or even comprises multiple tests, leading to all the known bad consequences this has (leaking state between tests, unclear test assumptions etc.)

herpderperator 6y ago

Call me crazy, but I go even more verbose than that. I go full Rspec/BDD mode: class TestAuthenticateMethod: class TestWhenUserExists: class TestWhenPasswordIsValid: def test_it_returns_200(self, app, user): assert ... class TestWhenPasswordIsInvalid: def test_it_returns_403(self, app, user): assert ... class TestWhenUserDoesNotExist: def test_it_returns_403(self, app, user): assert ... It is so worth it. I can look back at my tests 3 months later and know exactly what was intended wthout duplicating explanations with comments. Yes some methods can get a bit long and indented, but it's worth it for me. The other downside is you have to start all classes/methods with "test" but I got used to that very quickly. When you run pytest it's very clear what's being tested and what the expected outcome is since it gives you the full class hierarchy in the output.

mumblemumble 6y ago
This idea is both crazy and genius. Thanks for sharing.
jkaptur 6y ago
I’ve been working on a codebase like this, and my major complaint is that I’ll see a deeply-nested test and have to scroll waaaaay up to understand the context. I prefer ‘foo_bar_baz’, ‘foo_bar_qux’ to foo <snip 1000 lines> bar <snip 553 lines> baz <snip 243 lines> qux
BerislavLopac 6y ago
This is neat, but pytest allows for similar granulation using the right fixtures: E.g. in test_authenticate_method.py: def test_returns_200_on_valid_password(app, user_exists): ... def test_returns_403_on_valid_password(app, user_exists): ... def test_returns_403(app, user_not_exist): ...

BiteCode_dev 6y ago

I usually put that info in the asset: Assert foo(), "foo should bar" So that each failure has a clear meaning. Indeed, i may have several assert in a test.