5 ms·

I don't quite get how any when/then could NOT be dependent upon previous givens. Also, despite the job-centric nature of the .feature, it wasn't timing-depende

by jfrisby 13y ago

I don't quite get how any when/then could NOT be dependent upon previous givens. Also, despite the job-centric nature of the .feature, it wasn't timing-dependent because we mocked Resque to keep an in-process queue that we could dispatch in whatever order we want. So out-of-order arrival was deterministically achievable, etc. And yes, that test is testing a simple property -- it's effectively a unit test, with greater clarity of intent. Not familiar with quickcheck, but I will look into it, thanks!

IanCal 13y ago
> I don't quite get how any when/then could NOT be dependent upon previous givens. Given I am called Bob And I change my name to Steve Then my name is not Bob None of these steps are dependent on the ones previous to it, and can be reused elsewhere. The check for "my name is bob" can be used whether or not the previous step has been called. Your test won't pass, which is fine, but it will run and do exactly what it says. The step does what it says even if none of the others are used. It also means I can write a new test using bits from others without worrying about what's in the source code. A test that looks like this: Given I am a person And I change my name Then my name is different Cannot be re-used. The "my name is different" can only be used if you know the ruby code underneath. "my name is different" may not do what you expect unless it comes after "I change my name". What I meant with timing is the only way you can get around storing state is to have the "my name is different" watch for a change in the name, which brings in timing issues. I've seen both implementations used, and both caused problems (the timing one for obvious reasons). > Not familiar with quickcheck, but I will look into it, thanks! I thoroughly recommend it. The haskell version is probably the most advanced, but there are similar versions for most languages (and you can write your own, I had to for AS3). The idea is you express general properties about your system, and then it auto-generates thousands of examples (and if you have a nice library, automatically shrink failing cases for you). For example, the test above would be nicer as something like: X is a string Y is a string X =/= Y Given I am called X And I change my name to Y Then my name is not X And my name is Y Or something like that. This would then generate examples with no-length strings, crazy unicode characters, long strings, different mixings of RTL sections, etc. Much more likely to drive out bugs than a test for Bob and Steve. Some more useful ones would look like this (the first is a test I've written before, but not in this format): X is a number >= 1 Y is an interface element Given I am on element Y When I press Tab X times And I press Shift-Tab X times Then I am on element Y A similar version for navigating in a website and pressing "back" to get back to where you were (to ensure you're not breaking the back button). These are really simple, but powerful tests. I was most sold on the idea when I wrote this (not in this format, but this logic): X is a positive integer Y is a positive integer INSTRUCTION is one of [addElement, removeElement(X), setFocus(X)] MENU is an interface When I perform Y INSTRUCTIONS Then MENU has one focused item or no items at all This then generated thousands of menus of each valid type (vertical, horizontal, grids, etc) and then called library functions to add or remove elements, or move focus. Millions of tests overall. I was using this as a test for my quickcheck implementation, and found it failed. If I set the focus, deleted all the elements and then added a single new one it wasn't focused. When I fixed it, a unit test failed. We has previously specified that was to be the behaviour, but also specified that no matter what there would always be a focused element (if there were any elements at all). The general test drew out an inconsistency in our spec because we were forced to write general rules. Mixing quickcheck and cucumber has been one of my "Some weekend I'll do it" projects for a couple of years now.
- jfrisby 13y ago
  So your example still requires that state exist, it's just far more global. I.E. there must be some prerequisite code that sets up the thing to which "I" refers such that it's available to any step at any time. There may be some good patterns for that -- and that approach may be an excellent way to keep step definitions clean and simple, but I think it's somewhat orthogonal to my point. I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. We address the need for it to be able to see "back in time" by storing a history of the state of the object in a way that is accessible to the steps. I.E. For steps like this: Given a credential for a supported vendor When I change the nickname Then the edit version should not be changed The step definitions might look like: Given /a credential for a supported vendor/ do @thing = FactoryGirl.create(:'credential/amazon', :valid) @thing_history = [@thing.attributes.dup] end When /I change the nickname/ do @thing.nickname += " meh" @thing.save! @thing_history << @thing.attributes.dup end Then /the edit version should not be changed/ do @thing_history[-1][:edit_version].should == @thing_history[0][:edit_version] end (The use of "@thing" is a way to encourage myself to only be talking about one object at a time...) Perhaps that's a somewhat obtuse way of handling things -- and there's likely ways of DRYing up that pattern, if it's worth preserving, but it's proven to be fairly effective for me in the past. And yeah, Cucumber might make a good vehicle for that sort of testing. Quickcheck sounds like a brilliant idea -- although a bit terrifying in the context of a slow language like Ruby. Would love to get my hands on / build a tool like that...
  - IanCal 13y ago
    > So your example still requires that state exist, it's just far more global. The state exists only in your application, not in the test. Your given sets up the environment in the way you want, the "when" manipulates the environment and the "then" checks that the environment exists in a particular setting. > I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. This doesn't apply to your tests. What I've seen before is a step that says Then X changes Which is implemented as previous = X.state wait 5 seconds X.state != previous This relies on the previous step taking long enough to actually change the state that the change happens during the wait. All of the tests I was dealing with were asynchronous. Your pattern does look quite nice but there are still dependencies that aren't specified in code. For example, assuming you have a complementary step about the edit version changing written in the same way. Given a credential for a supported vendor When I do something that changes the edit version Then the edit version should be changed When I do something irrelevant Then the edit version should not be changed This would fail, saying the edit version has changed. Contrived, I know, but the problem with dependencies that aren't specified is that you start having valid tests that fail because they simply do not do what they say. I'm strongly in the camp of "If it shouldn't work, it shouldn't build". My core suggestion for all of this is to write your behavioural tests as a test script. What does the user do to get to that point, how do they interact and what's the result. The state exists either in your application (which is fine) or is explicit in the writing of the tests (Given I log in as person X rather than Given I log in). Cucumber is a great tool, but it leaves a lot of things implicit, which to be fair could probably be solved as a library. I'd have much less of a problem with all of this if when I wrote an invalid test, something warned me. I've spent far too much time battling with cucumber tests which lied about what they were actually testing.

by jfrisby 13y ago

IanCal 13y ago

> I don't quite get how any when/then could NOT be dependent upon previous givens. Given I am called Bob And I change my name to Steve Then my name is not Bob None of these steps are dependent on the ones previous to it, and can be reused elsewhere. The check for "my name is bob" can be used whether or not the previous step has been called. Your test won't pass, which is fine, but it will run and do exactly what it says. The step does what it says even if none of the others are used. It also means I can write a new test using bits from others without worrying about what's in the source code. A test that looks like this: Given I am a person And I change my name Then my name is different Cannot be re-used. The "my name is different" can only be used if you know the ruby code underneath. "my name is different" may not do what you expect unless it comes after "I change my name". What I meant with timing is the only way you can get around storing state is to have the "my name is different" watch for a change in the name, which brings in timing issues. I've seen both implementations used, and both caused problems (the timing one for obvious reasons). > Not familiar with quickcheck, but I will look into it, thanks! I thoroughly recommend it. The haskell version is probably the most advanced, but there are similar versions for most languages (and you can write your own, I had to for AS3). The idea is you express general properties about your system, and then it auto-generates thousands of examples (and if you have a nice library, automatically shrink failing cases for you). For example, the test above would be nicer as something like: X is a string Y is a string X =/= Y Given I am called X And I change my name to Y Then my name is not X And my name is Y Or something like that. This would then generate examples with no-length strings, crazy unicode characters, long strings, different mixings of RTL sections, etc. Much more likely to drive out bugs than a test for Bob and Steve. Some more useful ones would look like this (the first is a test I've written before, but not in this format): X is a number >= 1 Y is an interface element Given I am on element Y When I press Tab X times And I press Shift-Tab X times Then I am on element Y A similar version for navigating in a website and pressing "back" to get back to where you were (to ensure you're not breaking the back button). These are really simple, but powerful tests. I was most sold on the idea when I wrote this (not in this format, but this logic): X is a positive integer Y is a positive integer INSTRUCTION is one of [addElement, removeElement(X), setFocus(X)] MENU is an interface When I perform Y INSTRUCTIONS Then MENU has one focused item or no items at all This then generated thousands of menus of each valid type (vertical, horizontal, grids, etc) and then called library functions to add or remove elements, or move focus. Millions of tests overall. I was using this as a test for my quickcheck implementation, and found it failed. If I set the focus, deleted all the elements and then added a single new one it wasn't focused. When I fixed it, a unit test failed. We has previously specified that was to be the behaviour, but also specified that no matter what there would always be a focused element (if there were any elements at all). The general test drew out an inconsistency in our spec because we were forced to write general rules. Mixing quickcheck and cucumber has been one of my "Some weekend I'll do it" projects for a couple of years now.

jfrisby 13y ago
So your example still requires that state exist, it's just far more global. I.E. there must be some prerequisite code that sets up the thing to which "I" refers such that it's available to any step at any time. There may be some good patterns for that -- and that approach may be an excellent way to keep step definitions clean and simple, but I think it's somewhat orthogonal to my point. I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. We address the need for it to be able to see "back in time" by storing a history of the state of the object in a way that is accessible to the steps. I.E. For steps like this: Given a credential for a supported vendor When I change the nickname Then the edit version should not be changed The step definitions might look like: Given /a credential for a supported vendor/ do @thing = FactoryGirl.create(:'credential/amazon', :valid) @thing_history = [@thing.attributes.dup] end When /I change the nickname/ do @thing.nickname += " meh" @thing.save! @thing_history << @thing.attributes.dup end Then /the edit version should not be changed/ do @thing_history[-1][:edit_version].should == @thing_history[0][:edit_version] end (The use of "@thing" is a way to encourage myself to only be talking about one object at a time...) Perhaps that's a somewhat obtuse way of handling things -- and there's likely ways of DRYing up that pattern, if it's worth preserving, but it's proven to be fairly effective for me in the past. And yeah, Cucumber might make a good vehicle for that sort of testing. Quickcheck sounds like a brilliant idea -- although a bit terrifying in the context of a slow language like Ruby. Would love to get my hands on / build a tool like that...
- IanCal 13y ago
  > So your example still requires that state exist, it's just far more global. The state exists only in your application, not in the test. Your given sets up the environment in the way you want, the "when" manipulates the environment and the "then" checks that the environment exists in a particular setting. > I'm still not with you on the timing thing. By the time that step is reached, either the state has changed, or it hasn't. This doesn't apply to your tests. What I've seen before is a step that says Then X changes Which is implemented as previous = X.state wait 5 seconds X.state != previous This relies on the previous step taking long enough to actually change the state that the change happens during the wait. All of the tests I was dealing with were asynchronous. Your pattern does look quite nice but there are still dependencies that aren't specified in code. For example, assuming you have a complementary step about the edit version changing written in the same way. Given a credential for a supported vendor When I do something that changes the edit version Then the edit version should be changed When I do something irrelevant Then the edit version should not be changed This would fail, saying the edit version has changed. Contrived, I know, but the problem with dependencies that aren't specified is that you start having valid tests that fail because they simply do not do what they say. I'm strongly in the camp of "If it shouldn't work, it shouldn't build". My core suggestion for all of this is to write your behavioural tests as a test script. What does the user do to get to that point, how do they interact and what's the result. The state exists either in your application (which is fine) or is explicit in the writing of the tests (Given I log in as person X rather than Given I log in). Cucumber is a great tool, but it leaves a lot of things implicit, which to be fair could probably be solved as a library. I'd have much less of a problem with all of this if when I wrote an invalid test, something warned me. I've spent far too much time battling with cucumber tests which lied about what they were actually testing.