6 ms·

Why No One Uses Your Test Framework

38 points by raymondh 15y ago

raymondh 15y ago
Python's unittest.py was a direct port from Java's JUnit. More recently, a Google contribution to unittest.py added a huge number of self.assertThisThatOrTheOther() methods. The new methods were designed to have better diagnostic messages for test failures (i.e. diffs between expected and actual). Much needed test discovery was added by Michael Foord. Third-party testing modules like py.test and nose.py are popular substitutes for unittest.py. They use a light-weight syntax and are joy to use.
igorgue 15y ago
Well there are very good efforts to make this situation better like Konira (RSpec like Python DSL) and Lettuce (BDD but you can also use it for unit tests). But nobody really pays attention to it (well maybe just Yipit and me). And its authors are really cool and active, you can even tweet with them about problems you have, I even had conversations about testing theory (and nomenclature) with them.
jrockway 15y ago
This is pretty much my impression every time I try to test something in Python. I've never liked xunit, and I especially dislike it when it's a direct port from Java. Yes, it's familiar to Java programmers. But when I'm writing Python, I want something that looks like Python. (XML DOM libraries also like to think they're Java. That makes Common Lisp my favorite language for munging XML, since it's the only language bold enough to come up with some other API. But I digress...) Also, I don't understand mocks in Python. Say I have a class Foo that has a method "bar" that I want to override. OK: class FakeFoo(Foo): def bar(self): print "OH HAI" Done. It's built in to the freakin' language :)
- kbd 15y ago
  Well, here's Michael Foord's (@voidspace, referenced in the slides) Mock library so you can see what extra stuff it gives you. http://www.voidspace.org.uk/python/mock/ http://www.voidspace.org.uk/python/mock/
  - raymondh 15y ago
    FWIW, Michael Foord's mock library has been approved for inclusion in the Python's standard library. See http://www.python.org/dev/peps/pep-0417/ http://www.python.org/dev/peps/pep-0417/ and http://www.gossamer-threads.com/lists/python/dev/973274 http://www.gossamer-threads.com/lists/python/dev/973274
- tdavis 15y ago
  Now make some other code use your "mock" without editing its source. Now assert on the calls made to "bar()," including arguments. Now make it raise an exception without writing a second class. And make it return something else without writing a third. Now patch a built-in. And a context manager. And a dictionary. When you're done with that, mock a class without writing a new class at all. If you honestly think that sub-classing is equivalent to mocking/stubbing, you don't understand the purpose of the activity.
  - jrockway 15y ago
    I know libraries can do all those things, but I've never preferred it over the real thing. As an example, I might do something like this (not Python): my $bar = 0; class FakeFoo { sub bar { return $bar } } my $thing = ThingToTest->new( foo => FakeFoo->new ); $foo = 42; is $thing->make_bar_two_bigger, 44; $foo = -10; is $thing->make_bar_two_bigger, -12; Mock enthusiasts would write: my $fake_foo = Mock->new( Foo->class )->instance; $fake_foo->return_sequence( bar => [42, -10] ); my $thing = ThingToTest->new( foo => $fake_foo ); is $fake_foo->make_bar_two_bigger, 44; is $fake_foo->make_bar_two_bigger, -12; ok $fake_foo->ensure_exhausted('bar'); Yes, it's less lines, but I think it's harder to read because the values are being set up long before they're used.
    - jrockway 15y ago
      Well, this was typo-y. $foo in the first codeblock should be $bar, and $fake_foo->make_bar_two_bigger should be $thing->make_bar_two_bigger. More coffee is required :)
- pjscott 15y ago
  If you want more pleasant test-writing in Python, nose is commonly used, high quality, and compatible with the standard library's unittest package: http://readthedocs.org/docs/nose/en/latest/ http://readthedocs.org/docs/nose/en/latest/
kbd 15y ago
Thanks for the flashbacks from the TiP BoF. I thought we'd never get to the end, but man that was a fun and nerdy time.
pragmatic 15y ago
There seems to be a lot of subtext here that I'd love to see more about. I'm only an occasional python hacker and I'd love to know more about this. One question, is there a web testing framework for python that they do like/recommend?
- keypusher 15y ago
  For a general purpose testing framework, Nose is great. For web testing, combine it with Selenium.
jamescarr 15y ago
I really liked the shot about web testing. It's a crying shame that a lot of web testing frameworks don't bother supporting javascript.
tmcneal 15y ago
It's hard to get the full context of this presentation just from the slides, but I'm assuming the author is talking about Python test frameworks. I've been using nose in combination with coverage (http://pypi.python.org/pypi/coverage http://pypi.python.org/pypi/coverage) for a year and have been very happy with it.
jarrett 15y ago
My pet reason: Your framework doesn't solve any of the genuinely hard problems that annoy us when we're trying to test stuff. Lots of people have written variations on assertions and mocks. But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML? Or, where's the tool for testing integration with REST services outside your control, that doesn't depend on the real service most of the time, but can be told to run against the real service as a sanity check when you want to? (Like Ruby's VCR, but better.) Or, how about a test framework that accurately emulates mobile browsers and automatically? I'm not getting on anyone's case for not having written this stuff. Because hey, I'm just as guilty of not solving these problems as is the rest of the community. But if you want to make a testing tool that people will absolutely love, solve some of these problems really elegantly. (I know, easier said than done. And yes, some tools have solved some of these problems to some extent. But I'd consider each of my examples to be an ongoing pain point for developers.)
- Fluxx 15y ago
  > But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML? If you're talking Python, Lettuce does this is Python, there also is Splinter. You can run selenium with a headless firefox quite easily with Xvfb. If you're talking rails, Capybara has drivers for Webkit (QI), selenium and even PhantomJS. Their speed is variable, but it's not too bad. On the "HTML changes break my tests," I'd argue that A) you should write your front end tests not tied to the HTML if at all possible. Refer to elements by semantic names, not XPath or CSS selectors. And be smart about your steps. Make your submit_form() method actually look for a submit button in more than one way. B) you shouldn't have more than a handful of full stack frond end tests. They're hard to write, slow, and don't give much help telling you WHAT went wrong. Most of the time the ROI is only there for verifying a few user paths, but not much more than that.
  - benjiweber 15y ago
    The Page Object Pattern can help reduce the brittleness of automated UI tests. http://code.google.com/p/selenium/wiki/PageObjects http://code.google.com/p/selenium/wiki/PageObjects
gaius 15y ago
No-one talk to me about unit testing unless you're already using strong typing.
- troyjfarrell 15y ago
  Python (the subject of the OP) is a strongly-typed, dynamically-typed language.
  - pbiggar 15y ago
    I think it's pretty clear from context that the OP meant static typing. Since he's being a dick about it, I very much understand your desire to pedantically point out his error of terminology. Unfortunately, "strong" typing is a very overloaded term, which has never had a widely agreed meaning amongst either academics or practitioners. There are generally two camps: "strong" === "static", and "strong" === "no dynamic coercions", and neither is technically incorrect.

38 points by raymondh 15y ago

Python's unittest.py was a direct port from Java's JUnit. More recently, a Google contribution to unittest.py added a huge number of self.assertThisThatOrTheOther() methods. The new methods were designed to have better diagnostic messages for test failures (i.e. diffs between expected and actual). Much needed test discovery was added by Michael Foord. Third-party testing modules like py.test and nose.py are popular substitutes for unittest.py. They use a light-weight syntax and are joy to use.

igorgue 15y ago

Well there are very good efforts to make this situation better like Konira (RSpec like Python DSL) and Lettuce (BDD but you can also use it for unit tests). But nobody really pays attention to it (well maybe just Yipit and me). And its authors are really cool and active, you can even tweet with them about problems you have, I even had conversations about testing theory (and nomenclature) with them.

jrockway 15y ago

This is pretty much my impression every time I try to test something in Python. I've never liked xunit, and I especially dislike it when it's a direct port from Java. Yes, it's familiar to Java programmers. But when I'm writing Python, I want something that looks like Python. (XML DOM libraries also like to think they're Java. That makes Common Lisp my favorite language for munging XML, since it's the only language bold enough to come up with some other API. But I digress...) Also, I don't understand mocks in Python. Say I have a class Foo that has a method "bar" that I want to override. OK: class FakeFoo(Foo): def bar(self): print "OH HAI" Done. It's built in to the freakin' language :)

kbd 15y ago
Well, here's Michael Foord's (@voidspace, referenced in the slides) Mock library so you can see what extra stuff it gives you. http://www.voidspace.org.uk/python/mock/ http://www.voidspace.org.uk/python/mock/
- raymondh 15y ago
  FWIW, Michael Foord's mock library has been approved for inclusion in the Python's standard library. See http://www.python.org/dev/peps/pep-0417/ http://www.python.org/dev/peps/pep-0417/ and http://www.gossamer-threads.com/lists/python/dev/973274 http://www.gossamer-threads.com/lists/python/dev/973274
tdavis 15y ago
Now make some other code use your "mock" without editing its source. Now assert on the calls made to "bar()," including arguments. Now make it raise an exception without writing a second class. And make it return something else without writing a third. Now patch a built-in. And a context manager. And a dictionary. When you're done with that, mock a class without writing a new class at all. If you honestly think that sub-classing is equivalent to mocking/stubbing, you don't understand the purpose of the activity.
- jrockway 15y ago
  I know libraries can do all those things, but I've never preferred it over the real thing. As an example, I might do something like this (not Python): my $bar = 0; class FakeFoo { sub bar { return $bar } } my $thing = ThingToTest->new( foo => FakeFoo->new ); $foo = 42; is $thing->make_bar_two_bigger, 44; $foo = -10; is $thing->make_bar_two_bigger, -12; Mock enthusiasts would write: my $fake_foo = Mock->new( Foo->class )->instance; $fake_foo->return_sequence( bar => [42, -10] ); my $thing = ThingToTest->new( foo => $fake_foo ); is $fake_foo->make_bar_two_bigger, 44; is $fake_foo->make_bar_two_bigger, -12; ok $fake_foo->ensure_exhausted('bar'); Yes, it's less lines, but I think it's harder to read because the values are being set up long before they're used.
  - jrockway 15y ago
    Well, this was typo-y. $foo in the first codeblock should be $bar, and $fake_foo->make_bar_two_bigger should be $thing->make_bar_two_bigger. More coffee is required :)
pjscott 15y ago
If you want more pleasant test-writing in Python, nose is commonly used, high quality, and compatible with the standard library's unittest package: http://readthedocs.org/docs/nose/en/latest/ http://readthedocs.org/docs/nose/en/latest/

kbd 15y ago

Thanks for the flashbacks from the TiP BoF. I thought we'd never get to the end, but man that was a fun and nerdy time.

pragmatic 15y ago

There seems to be a lot of subtext here that I'd love to see more about. I'm only an occasional python hacker and I'd love to know more about this. One question, is there a web testing framework for python that they do like/recommend?

keypusher 15y ago
For a general purpose testing framework, Nose is great. For web testing, combine it with Selenium.

jamescarr 15y ago

I really liked the shot about web testing. It's a crying shame that a lot of web testing frameworks don't bother supporting javascript.

tmcneal 15y ago

It's hard to get the full context of this presentation just from the slides, but I'm assuming the author is talking about Python test frameworks. I've been using nose in combination with coverage (http://pypi.python.org/pypi/coverage http://pypi.python.org/pypi/coverage) for a year and have been very happy with it.

jarrett 15y ago

My pet reason: Your framework doesn't solve any of the genuinely hard problems that annoy us when we're trying to test stuff. Lots of people have written variations on assertions and mocks. But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML? Or, where's the tool for testing integration with REST services outside your control, that doesn't depend on the real service most of the time, but can be told to run against the real service as a sanity check when you want to? (Like Ruby's VCR, but better.) Or, how about a test framework that accurately emulates mobile browsers and automatically? I'm not getting on anyone's case for not having written this stuff. Because hey, I'm just as guilty of not solving these problems as is the rest of the community. But if you want to make a testing tool that people will absolutely love, solve some of these problems really elegantly. (I know, easier said than done. And yes, some tools have solved some of these problems to some extent. But I'd consider each of my examples to be an ongoing pain point for developers.)

Fluxx 15y ago
> But where's the full-stack test harness for a Rails app with heavy JS usage, that runs at a reasonable speed, gives us useful traces on the client and server, and doesn't break when I slightly rework my HTML? If you're talking Python, Lettuce does this is Python, there also is Splinter. You can run selenium with a headless firefox quite easily with Xvfb. If you're talking rails, Capybara has drivers for Webkit (QI), selenium and even PhantomJS. Their speed is variable, but it's not too bad. On the "HTML changes break my tests," I'd argue that A) you should write your front end tests not tied to the HTML if at all possible. Refer to elements by semantic names, not XPath or CSS selectors. And be smart about your steps. Make your submit_form() method actually look for a submit button in more than one way. B) you shouldn't have more than a handful of full stack frond end tests. They're hard to write, slow, and don't give much help telling you WHAT went wrong. Most of the time the ROI is only there for verifying a few user paths, but not much more than that.
- benjiweber 15y ago
  The Page Object Pattern can help reduce the brittleness of automated UI tests. http://code.google.com/p/selenium/wiki/PageObjects http://code.google.com/p/selenium/wiki/PageObjects

gaius 15y ago

No-one talk to me about unit testing unless you're already using strong typing.

troyjfarrell 15y ago
Python (the subject of the OP) is a strongly-typed, dynamically-typed language.
- pbiggar 15y ago
  I think it's pretty clear from context that the OP meant static typing. Since he's being a dick about it, I very much understand your desire to pedantically point out his error of terminology. Unfortunately, "strong" typing is a very overloaded term, which has never had a widely agreed meaning amongst either academics or practitioners. There are generally two camps: "strong" === "static", and "strong" === "no dynamic coercions", and neither is technically incorrect.