3 ms·

So your example still requires that state exist, it's just far more global. I.E. there must be some prerequisite code that sets up the thing to which "I" refer

by jfrisby 13y ago

So your example still requires that state exist, it's just far more global. I.E. there must be some prerequisite code that sets up the thing to which "I" refers such that it's available to any step at any time. There may be some good patterns for that -- and that approach may be an excellent way to keep step definitions clean and simple, but I think it's somewhat orthogonal to my point. I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. We address the need for it to be able to see "back in time" by storing a history of the state of the object in a way that is accessible to the steps. I.E. For steps like this: Given a credential for a supported vendor When I change the nickname Then the edit version should not be changed The step definitions might look like: Given /a credential for a supported vendor/ do @thing = FactoryGirl.create(:'credential/amazon', :valid) @thing_history = [@thing.attributes.dup] end When /I change the nickname/ do @thing.nickname += " meh" @thing.save! @thing_history << @thing.attributes.dup end Then /the edit version should not be changed/ do @thing_history[-1][:edit_version].should == @thing_history[0][:edit_version] end (The use of "@thing" is a way to encourage myself to only be talking about one object at a time...) Perhaps that's a somewhat obtuse way of handling things -- and there's likely ways of DRYing up that pattern, if it's worth preserving, but it's proven to be fairly effective for me in the past. And yeah, Cucumber might make a good vehicle for that sort of testing. Quickcheck sounds like a brilliant idea -- although a bit terrifying in the context of a slow language like Ruby. Would love to get my hands on / build a tool like that...

IanCal 13y ago
> So your example still requires that state exist, it's just far more global. The state exists only in your application, not in the test. Your given sets up the environment in the way you want, the "when" manipulates the environment and the "then" checks that the environment exists in a particular setting. > I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. This doesn't apply to your tests. What I've seen before is a step that says Then X changes Which is implemented as previous = X.state wait 5 seconds X.state != previous This relies on the previous step taking long enough to actually change the state that the change happens during the wait. All of the tests I was dealing with were asynchronous. Your pattern does look quite nice but there are still dependencies that aren't specified in code. For example, assuming you have a complementary step about the edit version changing written in the same way. Given a credential for a supported vendor When I do something that changes the edit version Then the edit version should be changed When I do something irrelevant Then the edit version should not be changed This would fail, saying the edit version has changed. Contrived, I know, but the problem with dependencies that aren't specified is that you start having valid tests that fail because they simply do not do what they say. I'm strongly in the camp of "If it shouldn't work, it shouldn't build". My core suggestion for all of this is to write your behavioural tests as a test script. What does the user do to get to that point, how do they interact and what's the result. The state exists either in your application (which is fine) or is explicit in the writing of the tests (Given I log in as person X rather than Given I log in). Cucumber is a great tool, but it leaves a lot of things implicit, which to be fair could probably be solved as a library. I'd have much less of a problem with all of this if when I wrote an invalid test, something warned me. I've spent far too much time battling with cucumber tests which lied about what they were actually testing.
- jfrisby 13y ago
  > The state exists only in your application, not in the test. > Your given sets up the environment in the way you want, the > "when" manipulates the environment and the "then" checks > that the environment exists in a particular setting. Err, are you asserting that that is what I am doing, or what I ought to be doing? And yes, time-dependent code is evil. I should probably add commentary to my style-guide to explicitly call that out, but thankfully we never ran into that despite testing of distributed-job-queue functionality, by virtue of having a queue so simple stubbing its main loop to work in-process in a deterministic way was trivial. Your example should be in violation of the style guide for precisely the reason you state, among several others (blurring of concerns, etc). If my style guide isn't clear on that point -- that a When should NEVER follow a Then -- I need to clarify that. :) I recall a Gherkin-based testing framework that handled the actual step definitions MUCH differently and much more cleanly but I never got around to fully switching us over and don't recall the name... :-S
  - IanCal 13y ago
    > Err, are you asserting that that is what I am doing, or what I ought to be doing? Ought to be. Your Given and When manipulate the environment and store state in the test. Your then doesn't check the environment at all, it checks it's own internal state. I think in your example this is less of an issue than the cases I've worked with before (usually testing remote running apps). > And yes, time-dependent code is evil. I should probably add commentary to my style-guide to explicitly call that out, but thankfully we never ran into that despite testing of distributed-job-queue functionality, by virtue of having a queue so simple stubbing its main loop to work in-process in a deterministic way was trivial. Yes, as I say I wasn't claiming that was what you were doing, but it's the only other way I've seen the same kind of tests written. Before the first reply, I didn't know which it would be. I'm very glad it wasn't :) > If my style guide isn't clear on that point -- that a When should NEVER follow a Then -- I need to clarify that. :) My example was contrived, but I think you can probably see the point I'm making. What the test does is not clear without understanding the ruby underneath, and by storing state you can have tests which don't do what you expect because they aren't just querying the state of your application. I'm aware I've been quite ranty about this, it's mostly I've had to deal with bad tests written in this style. Good tests written in this style seem fine generally. I think this is more of an issue with cucumber than the tests themselves. > I recall a Gherkin-based testing framework that handled the actual step definitions MUCH differently and much more cleanly but I never got around to fully switching us over and don't recall the name... :-S I might have a look around. It's something I've probably spent more time writing about it than it would have spent trying to write something better (/at least with dependency docs, maybe quickcheck style).

by jfrisby 13y ago

IanCal 13y ago

> So your example still requires that state exist, it's just far more global. The state exists only in your application, not in the test. Your given sets up the environment in the way you want, the "when" manipulates the environment and the "then" checks that the environment exists in a particular setting. > I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. This doesn't apply to your tests. What I've seen before is a step that says Then X changes Which is implemented as previous = X.state wait 5 seconds X.state != previous This relies on the previous step taking long enough to actually change the state that the change happens during the wait. All of the tests I was dealing with were asynchronous. Your pattern does look quite nice but there are still dependencies that aren't specified in code. For example, assuming you have a complementary step about the edit version changing written in the same way. Given a credential for a supported vendor When I do something that changes the edit version Then the edit version should be changed When I do something irrelevant Then the edit version should not be changed This would fail, saying the edit version has changed. Contrived, I know, but the problem with dependencies that aren't specified is that you start having valid tests that fail because they simply do not do what they say. I'm strongly in the camp of "If it shouldn't work, it shouldn't build". My core suggestion for all of this is to write your behavioural tests as a test script. What does the user do to get to that point, how do they interact and what's the result. The state exists either in your application (which is fine) or is explicit in the writing of the tests (Given I log in as person X rather than Given I log in). Cucumber is a great tool, but it leaves a lot of things implicit, which to be fair could probably be solved as a library. I'd have much less of a problem with all of this if when I wrote an invalid test, something warned me. I've spent far too much time battling with cucumber tests which lied about what they were actually testing.

jfrisby 13y ago
> The state exists only in your application, not in the test. > Your given sets up the environment in the way you want, the > "when" manipulates the environment and the "then" checks > that the environment exists in a particular setting. Err, are you asserting that that is what I am doing, or what I ought to be doing? And yes, time-dependent code is evil. I should probably add commentary to my style-guide to explicitly call that out, but thankfully we never ran into that despite testing of distributed-job-queue functionality, by virtue of having a queue so simple stubbing its main loop to work in-process in a deterministic way was trivial. Your example should be in violation of the style guide for precisely the reason you state, among several others (blurring of concerns, etc). If my style guide isn't clear on that point -- that a When should NEVER follow a Then -- I need to clarify that. :) I recall a Gherkin-based testing framework that handled the actual step definitions MUCH differently and much more cleanly but I never got around to fully switching us over and don't recall the name... :-S
- IanCal 13y ago
  > Err, are you asserting that that is what I am doing, or what I ought to be doing? Ought to be. Your Given and When manipulate the environment and store state in the test. Your then doesn't check the environment at all, it checks it's own internal state. I think in your example this is less of an issue than the cases I've worked with before (usually testing remote running apps). > And yes, time-dependent code is evil. I should probably add commentary to my style-guide to explicitly call that out, but thankfully we never ran into that despite testing of distributed-job-queue functionality, by virtue of having a queue so simple stubbing its main loop to work in-process in a deterministic way was trivial. Yes, as I say I wasn't claiming that was what you were doing, but it's the only other way I've seen the same kind of tests written. Before the first reply, I didn't know which it would be. I'm very glad it wasn't :) > If my style guide isn't clear on that point -- that a When should NEVER follow a Then -- I need to clarify that. :) My example was contrived, but I think you can probably see the point I'm making. What the test does is not clear without understanding the ruby underneath, and by storing state you can have tests which don't do what you expect because they aren't just querying the state of your application. I'm aware I've been quite ranty about this, it's mostly I've had to deal with bad tests written in this style. Good tests written in this style seem fine generally. I think this is more of an issue with cucumber than the tests themselves. > I recall a Gherkin-based testing framework that handled the actual step definitions MUCH differently and much more cleanly but I never got around to fully switching us over and don't recall the name... :-S I might have a look around. It's something I've probably spent more time writing about it than it would have spent trying to write something better (/at least with dependency docs, maybe quickcheck style).