6 ms·

Pytest 3.2.0 released

124 points by happy-go-lucky 9y ago

fermigier 9y ago
The `--last-failed` option will change my life! <3
- masklinn 9y ago
  FWIW it was added back in 2.8.0 (though only as a short `--lf` options). 3.2.0 makes it smarter by improving the first-failed/last-failed clear behaviour: it now only clears tests which have succeeded. You can see the use case in the PR: make a change, it breaks a lot of tests, then fix module par module e.g. after an initial `pytest` (failed lots) fix `pytest core --lf`, then fix `pytest controller --lf`, … the problem pre-3.2 is that `pytest core --lf` would reset the cache with only its own failures, so the subsequent `pytest controller —lf` would start with an empty cache and run every single test rather than only those which had previously failed. This is problematic when you have a large expensive testing base (e.g. so much so that you xdist it, as the original PR did)
giancarlostoro 9y ago
I'm fairly new to Python and wondering of any reasons to use these as opposed to the standard unittest package that comes with Python?
- mivade 9y ago
  If for no other reason, pytest requires a lot less boilerplate. Pytest fixtures are also quite a bit more flexible than unittest's setUp and tearDown methods, although they take a little while to really understand (at least they did for me) since they use some magic.
  - bpicolo 9y ago
    Yield fixtures are just awesome for mocking, but the magical global dependency injection definitely makes you feel a bit weird on occasion
    - quodlibetor 9y ago
      especially in combination with pytest-mock.
  - deleted 9y ago
    [deleted]
- masklinn 9y ago
  It has an excellent runner and makes much better use of Python features: * you can define tests as free functions (you don't have to, it also supports classes and even regular unittest cases, you can use pytest to run your existing suite) * exceptions aside it uses the standard `assert` statement * fixtures are DI'd, independent from tests and much more flexible than the setUp/tearDown dance * tests are configured and fixtures are defined using decorators & generators * excellent support for data-driven tests (parametrized tests and fixtures) * there is a wealth of reusable third-party plugins[0] ranging from framework helpers[1] to test parallelisation and distribution[2] [0] http://plugincompat.herokuapp.com http://plugincompat.herokuapp.com [1] https://pypi.python.org/pypi/pytest-tornado https://pypi.python.org/pypi/pytest-tornado [2] https://pypi.python.org/pypi/pytest-xdist https://pypi.python.org/pypi/pytest-xdist
  - kstrauser 9y ago
    > fixtures are DI'd, independent from tests and much more flexible than the setUp/tearDown dance If this were the only difference, it'd still be enough. Inevitably I end up with a unittest.TestCase with 30 tests, all of which mock out some resource. Except for 1. That one wants to mock out a different resource while leaving the first intact. * With unittest: either split that test out into its own TestCase (duplicating most of the original, except for the part I don't want, or coming up with an inheritance tree with is always ugly with unittest), or * Remove all the unwanted mocks inside that test. * With pytest: don't pass the mocking fixture into that test.
    - masklinn 9y ago
      Yup. The ability to compose fixtures too, a test can depend on a fixture but so can an other fixture. Combined with flexible scopes and the ability to mark and select/deselect tests based on name and marks it makes for genuinely great stuff e.g. some of your tests depend on the database, you can have a session-scoped fixture handling a connection, then a function-scoped one depending on it and getting a cursor, and if you don't select any test depending on the latter no connection will be created at any point.
- Twirrim 9y ago
  I find pytest a lot easier to work with, in general, particularly with their approach to fixtures https://docs.pytest.org/en/latest/fixture.html https://docs.pytest.org/en/latest/fixture.html . I find I'm writing way less boilerplate and a lot cleaner and neater tests. Also I absolutely love parametrize: https://docs.pytest.org/en/latest/parametrize.html https://docs.pytest.org/en/latest/parametrize.html
- vbernat 9y ago
  Parametrized tests are quite powerful and you can be creative with them. For example, I am using them in integration testing (for this test, I need process A and process B as well as a string from set C): https://vincent.bernat.im/en/blog/2016-testing-pytest-linux-namespaces https://vincent.bernat.im/en/blog/2016-testing-pytest-linux-... You can define the lifetime of each parameter (just the test, the class or the whole testing session). This makes it very flexible. The use of "assert" is magic in the sense that in case of failure, you get helpful values. "assert a == b" will show you the values of "a" and "b" in case of failure.
- open_bear 9y ago
  'pytest --lf' reruns only tests that have failed during the previous run. Helps a lot when your test suite is not very fast.
- meitham 9y ago
  Pytest is more pythonic than the xUnitTest suite in the standard library. It is authored by the same genius who authored pypy, hence the wow about it. Nosetest was forked off pytest as a way to maintain the project several years ago, but pytest came back to life and eventually superseded nose. The fixtures in pytest is a very clever way to structure your tests around dynamic "given" set of inputs.
jwilk 9y ago
I've been a long time nosetests user, but it's become unmaintained, so I'd like to migrate something else. pytest seems to be an obvious choice, but I really dislike the way it formats test failures. Is there a way to make it use a more traditional format?
- JustSomeNobody 9y ago
  Is nose buggy or is it missing features you need?
  - Alex3917 9y ago
    Nose is buggy. Lots of memory leaks, things that don't work in parallel, etc.
- jeremya 9y ago
  Yes, you can alter the way tracebacks are printed (or not printed) for test failures: https://docs.pytest.org/en/latest/usage.html#modifying-python-traceback-printing https://docs.pytest.org/en/latest/usage.html#modifying-pytho...
  - jwilk 9y ago
    Thanks! --tb=native combined with --color=no makes it almost bearable. It still annoys me that ==================================================================================== FAILURES ===================================================================================== takes the whole screen width. I can pipe stdout through "cat" to fix it, but it's not very convenient.
    - variedthoughts 9y ago
      Try -q
- quodlibetor 9y ago
  Interesting. pytest's handling of `assert foo == bar` is one of my top two favorite things about it. My other favorite thing being fixtures.
noisy_boy 9y ago
I really like Pytest but I didn't see a way to easily customize the output format (not just traceback but overall output). If they can add documentation with examples of the hooks available (if present) to customize the output, I won't have anything further to complain about Pytest.
- joaodlf 9y ago
  This is a big one for me, too. Would make working with CI tools much easier.
  - noisy_boy 9y ago
    So I'm not the only one :) Do you have any suggestions about alternatives? The Python code base I work with isn't too big so I'm open to exploring other unit test frameworks.
  - luhn 9y ago
    One thing that might help is that pytest can output test results as JUnit XML files, which many CI services support.
    - kstrauser 9y ago
      That's how we wire it into Jenkins.
happy-go-lucky 9y ago
For those new to Pytest, here's a tutorial to get you started: Testing Python Applications with Pytest https://semaphoreci.com/community/tutorials/testing-python-applications-with-pytest https://semaphoreci.com/community/tutorials/testing-python-a...
O5vYtytb 9y ago
Oh boy! `PYTEST_CURRENT_TEST` let me take out a lot of hacks from my code. We're doing a lot with logging and parallel tests, so this was always a sore point.
theptip 9y ago
I've been pretty disappointed with PyCharm's support for Pytest in a Django project -- seems to be incapable of overriding the `manage.py` test runner consistently. pytest-django has some docs[1] for how to plug into `manage.py`, but they are broken as of Django 1.10. Anyone else had this problem? [1]: https://github.com/pytest-dev/pytest-django/blob/master/docs/faq.rst#how-can-i-use-managepy-test-with-pytest-django https://github.com/pytest-dev/pytest-django/blob/master/docs...
- gegenschall 9y ago
  It is actually discouraged by pytest-django[0] to run it using Django's test command. Just run `pytest` itself. [0]: https://pytest-django.readthedocs.io/en/latest/#why-would-i-use-this-instead-of-django-s-manage-py-test-command https://pytest-django.readthedocs.io/en/latest/#why-would-i-...
  - theptip 9y ago
    Yeah, see part one of my comment -- I've found that PyCharm is pretty bad at selecting `pytest` as the test runner. It works for test functions, but not for files/directories. Perhaps I'm missing something obvious, but PyCharm just uses `./manage.py test` even when I've selected Py.test as the project test runner.
pingpongchef 9y ago
Pytest's fixtures have ruined me for other testing frameworks.
theptip 9y ago
Any pointers on using fixtures with Django database models? Seems like it would be useful to be able to manage the lifecycle of fixtures more explicitly than Django's TestCase allows you to, but this could get gnarly if the test case transactions were rolling back changes to fixture objects.
- travisjungroth 9y ago
  Are you using https://pytest-django.readthedocs.io/en/latest/ https://pytest-django.readthedocs.io/en/latest/?
  - theptip 9y ago
    Yup

The `--last-failed` option will change my life! <3

masklinn 9y ago
FWIW it was added back in 2.8.0 (though only as a short `--lf` options). 3.2.0 makes it smarter by improving the first-failed/last-failed clear behaviour: it now only clears tests which have succeeded. You can see the use case in the PR: make a change, it breaks a lot of tests, then fix module par module e.g. after an initial `pytest` (failed lots) fix `pytest core --lf`, then fix `pytest controller --lf`, … the problem pre-3.2 is that `pytest core --lf` would reset the cache with only its own failures, so the subsequent `pytest controller —lf` would start with an empty cache and run every single test rather than only those which had previously failed. This is problematic when you have a large expensive testing base (e.g. so much so that you xdist it, as the original PR did)

I'm fairly new to Python and wondering of any reasons to use these as opposed to the standard unittest package that comes with Python?

mivade 9y ago
If for no other reason, pytest requires a lot less boilerplate. Pytest fixtures are also quite a bit more flexible than unittest's setUp and tearDown methods, although they take a little while to really understand (at least they did for me) since they use some magic.
- bpicolo 9y ago
  Yield fixtures are just awesome for mocking, but the magical global dependency injection definitely makes you feel a bit weird on occasion
  - quodlibetor 9y ago
    especially in combination with pytest-mock.
- deleted 9y ago
  [deleted]
masklinn 9y ago
It has an excellent runner and makes much better use of Python features: * you can define tests as free functions (you don't have to, it also supports classes and even regular unittest cases, you can use pytest to run your existing suite) * exceptions aside it uses the standard `assert` statement * fixtures are DI'd, independent from tests and much more flexible than the setUp/tearDown dance * tests are configured and fixtures are defined using decorators & generators * excellent support for data-driven tests (parametrized tests and fixtures) * there is a wealth of reusable third-party plugins[0] ranging from framework helpers[1] to test parallelisation and distribution[2] [0] http://plugincompat.herokuapp.com http://plugincompat.herokuapp.com [1] https://pypi.python.org/pypi/pytest-tornado https://pypi.python.org/pypi/pytest-tornado [2] https://pypi.python.org/pypi/pytest-xdist https://pypi.python.org/pypi/pytest-xdist
- kstrauser 9y ago
  > fixtures are DI'd, independent from tests and much more flexible than the setUp/tearDown dance If this were the only difference, it'd still be enough. Inevitably I end up with a unittest.TestCase with 30 tests, all of which mock out some resource. Except for 1. That one wants to mock out a different resource while leaving the first intact. * With unittest: either split that test out into its own TestCase (duplicating most of the original, except for the part I don't want, or coming up with an inheritance tree with is always ugly with unittest), or * Remove all the unwanted mocks inside that test. * With pytest: don't pass the mocking fixture into that test.
  - masklinn 9y ago
    Yup. The ability to compose fixtures too, a test can depend on a fixture but so can an other fixture. Combined with flexible scopes and the ability to mark and select/deselect tests based on name and marks it makes for genuinely great stuff e.g. some of your tests depend on the database, you can have a session-scoped fixture handling a connection, then a function-scoped one depending on it and getting a cursor, and if you don't select any test depending on the latter no connection will be created at any point.
Twirrim 9y ago
I find pytest a lot easier to work with, in general, particularly with their approach to fixtures https://docs.pytest.org/en/latest/fixture.html https://docs.pytest.org/en/latest/fixture.html . I find I'm writing way less boilerplate and a lot cleaner and neater tests. Also I absolutely love parametrize: https://docs.pytest.org/en/latest/parametrize.html https://docs.pytest.org/en/latest/parametrize.html
vbernat 9y ago
Parametrized tests are quite powerful and you can be creative with them. For example, I am using them in integration testing (for this test, I need process A and process B as well as a string from set C): https://vincent.bernat.im/en/blog/2016-testing-pytest-linux-namespaces https://vincent.bernat.im/en/blog/2016-testing-pytest-linux-... You can define the lifetime of each parameter (just the test, the class or the whole testing session). This makes it very flexible. The use of "assert" is magic in the sense that in case of failure, you get helpful values. "assert a == b" will show you the values of "a" and "b" in case of failure.
open_bear 9y ago
'pytest --lf' reruns only tests that have failed during the previous run. Helps a lot when your test suite is not very fast.
meitham 9y ago
Pytest is more pythonic than the xUnitTest suite in the standard library. It is authored by the same genius who authored pypy, hence the wow about it. Nosetest was forked off pytest as a way to maintain the project several years ago, but pytest came back to life and eventually superseded nose. The fixtures in pytest is a very clever way to structure your tests around dynamic "given" set of inputs.

jwilk 9y ago

I've been a long time nosetests user, but it's become unmaintained, so I'd like to migrate something else. pytest seems to be an obvious choice, but I really dislike the way it formats test failures. Is there a way to make it use a more traditional format?

JustSomeNobody 9y ago
Is nose buggy or is it missing features you need?
- Alex3917 9y ago
  Nose is buggy. Lots of memory leaks, things that don't work in parallel, etc.
jeremya 9y ago
Yes, you can alter the way tracebacks are printed (or not printed) for test failures: https://docs.pytest.org/en/latest/usage.html#modifying-python-traceback-printing https://docs.pytest.org/en/latest/usage.html#modifying-pytho...
- jwilk 9y ago
  Thanks! --tb=native combined with --color=no makes it almost bearable. It still annoys me that ==================================================================================== FAILURES ===================================================================================== takes the whole screen width. I can pipe stdout through "cat" to fix it, but it's not very convenient.
  - variedthoughts 9y ago
    Try -q
quodlibetor 9y ago
Interesting. pytest's handling of `assert foo == bar` is one of my top two favorite things about it. My other favorite thing being fixtures.

noisy_boy 9y ago

I really like Pytest but I didn't see a way to easily customize the output format (not just traceback but overall output). If they can add documentation with examples of the hooks available (if present) to customize the output, I won't have anything further to complain about Pytest.

joaodlf 9y ago
This is a big one for me, too. Would make working with CI tools much easier.
- noisy_boy 9y ago
  So I'm not the only one :) Do you have any suggestions about alternatives? The Python code base I work with isn't too big so I'm open to exploring other unit test frameworks.
- luhn 9y ago
  One thing that might help is that pytest can output test results as JUnit XML files, which many CI services support.
  - kstrauser 9y ago
    That's how we wire it into Jenkins.

happy-go-lucky 9y ago

For those new to Pytest, here's a tutorial to get you started: Testing Python Applications with Pytest https://semaphoreci.com/community/tutorials/testing-python-applications-with-pytest https://semaphoreci.com/community/tutorials/testing-python-a...

O5vYtytb 9y ago

Oh boy! `PYTEST_CURRENT_TEST` let me take out a lot of hacks from my code. We're doing a lot with logging and parallel tests, so this was always a sore point.

theptip 9y ago

I've been pretty disappointed with PyCharm's support for Pytest in a Django project -- seems to be incapable of overriding the `manage.py` test runner consistently. pytest-django has some docs[1] for how to plug into `manage.py`, but they are broken as of Django 1.10. Anyone else had this problem? [1]: https://github.com/pytest-dev/pytest-django/blob/master/docs/faq.rst#how-can-i-use-managepy-test-with-pytest-django https://github.com/pytest-dev/pytest-django/blob/master/docs...

gegenschall 9y ago
It is actually discouraged by pytest-django[0] to run it using Django's test command. Just run `pytest` itself. [0]: https://pytest-django.readthedocs.io/en/latest/#why-would-i-use-this-instead-of-django-s-manage-py-test-command https://pytest-django.readthedocs.io/en/latest/#why-would-i-...
- theptip 9y ago
  Yeah, see part one of my comment -- I've found that PyCharm is pretty bad at selecting `pytest` as the test runner. It works for test functions, but not for files/directories. Perhaps I'm missing something obvious, but PyCharm just uses `./manage.py test` even when I've selected Py.test as the project test runner.

pingpongchef 9y ago

Pytest's fixtures have ruined me for other testing frameworks.

theptip 9y ago

Any pointers on using fixtures with Django database models? Seems like it would be useful to be able to manage the lifecycle of fixtures more explicitly than Django's TestCase allows you to, but this could get gnarly if the test case transactions were rolling back changes to fixture objects.

travisjungroth 9y ago
Are you using https://pytest-django.readthedocs.io/en/latest/ https://pytest-django.readthedocs.io/en/latest/?
- theptip 9y ago
  Yup