5 ms·

An Unusual Case for Cucumber

15 points by jfrisby 13y ago

IanCal 13y ago
I kind of agree with this, but then I see the same tests I've seen elsewhere: Scenario: Editing the nickname of a credential does not change the version When the nickname is modified Then the edit version of the credential should not change This looks like an awesome test. Simple, clear, anyone can read it. What's wrong? One of the following: * It's based on timing and unreliable * It stores state and the steps can't be reused. You simply cannot put the line "Then the edit version of the credentials should not change" elsewhere because the step involves the previous state of the system. This step can only come after this step: Given a credential for a supported vendor These tests all look like they're testing really nice general properties, but they aren't. They're almost always just checking little properties here and there, and rarely doing just what the step says. Which could be fine, but the language doesn't support any idea of specified dependencies. I cannot change a test without understanding the code behind it. What you end up with is a big codebase based on regexes, with very heavy (non-specified) dependencies. You have to be extremely careful not to end up like this, although the tests look more complex afterwards (but you have re-usable steps). I think there's a great place for a quickcheck & cucumber combination.
- jms_ 13y ago
  Given that the system these scenarios test also manage state, it's not surprising that the step definitions used in the scenario also need to manage some form of state. While it's advisable to avoid unnecessary state between steps in a scenario like this, removing all state limits the ability for a scenario to express the intent the author speaks about.
  - IanCal 13y ago
    One option is to move the state to be explicit in the call to the step, which means the test becomes an example rather than a general rule (although that's what it always was, now it's just a bit clearer). It also means the steps can be reused, rather than having steps with dependencies (which aren't specified anywhere or statically enforced). > removing all state limits the ability for a scenario to express the intent the author speaks about. However, writing them like that means each step moves further away from what it says it's doing. The more you do this, the more you might as well not execute them and go and write more examples.
    - jfrisby 13y ago
      To use your example -- how desirable is it to generalize "I change my name" to something "reusable"? I'm finding these days that a pairing of .feature to corresponding step definition file allows for fluid language, simple step definition code, and well-compartmentalized messes (as opposed to the big ball of mud that Cucumber step definitions often become). Code reuse can happen through test helper classes that are called from step definitions as needed. (FactoryGirl would be an example of a very generalized test helper class, in my mind -- but more system-specific ones can be handy, albeit one wants to avoid making stateful / complex test helper code!)
- jfrisby 13y ago
  I don't quite get how any when/then could NOT be dependent upon previous givens. Also, despite the job-centric nature of the .feature, it wasn't timing-dependent because we mocked Resque to keep an in-process queue that we could dispatch in whatever order we want. So out-of-order arrival was deterministically achievable, etc. And yes, that test is testing a simple property -- it's effectively a unit test, with greater clarity of intent. Not familiar with quickcheck, but I will look into it, thanks!
  - IanCal 13y ago
    > I don't quite get how any when/then could NOT be dependent upon previous givens. Given I am called Bob And I change my name to Steve Then my name is not Bob None of these steps are dependent on the ones previous to it, and can be reused elsewhere. The check for "my name is bob" can be used whether or not the previous step has been called. Your test won't pass, which is fine, but it will run and do exactly what it says. The step does what it says even if none of the others are used. It also means I can write a new test using bits from others without worrying about what's in the source code. A test that looks like this: Given I am a person And I change my name Then my name is different Cannot be re-used. The "my name is different" can only be used if you know the ruby code underneath. "my name is different" may not do what you expect unless it comes after "I change my name". What I meant with timing is the only way you can get around storing state is to have the "my name is different" watch for a change in the name, which brings in timing issues. I've seen both implementations used, and both caused problems (the timing one for obvious reasons). > Not familiar with quickcheck, but I will look into it, thanks! I thoroughly recommend it. The haskell version is probably the most advanced, but there are similar versions for most languages (and you can write your own, I had to for AS3). The idea is you express general properties about your system, and then it auto-generates thousands of examples (and if you have a nice library, automatically shrink failing cases for you). For example, the test above would be nicer as something like: X is a string Y is a string X =/= Y Given I am called X And I change my name to Y Then my name is not X And my name is Y Or something like that. This would then generate examples with no-length strings, crazy unicode characters, long strings, different mixings of RTL sections, etc. Much more likely to drive out bugs than a test for Bob and Steve. Some more useful ones would look like this (the first is a test I've written before, but not in this format): X is a number >= 1 Y is an interface element Given I am on element Y When I press Tab X times And I press Shift-Tab X times Then I am on element Y A similar version for navigating in a website and pressing "back" to get back to where you were (to ensure you're not breaking the back button). These are really simple, but powerful tests. I was most sold on the idea when I wrote this (not in this format, but this logic): X is a positive integer Y is a positive integer INSTRUCTION is one of [addElement, removeElement(X), setFocus(X)] MENU is an interface When I perform Y INSTRUCTIONS Then MENU has one focused item or no items at all This then generated thousands of menus of each valid type (vertical, horizontal, grids, etc) and then called library functions to add or remove elements, or move focus. Millions of tests overall. I was using this as a test for my quickcheck implementation, and found it failed. If I set the focus, deleted all the elements and then added a single new one it wasn't focused. When I fixed it, a unit test failed. We has previously specified that was to be the behaviour, but also specified that no matter what there would always be a focused element (if there were any elements at all). The general test drew out an inconsistency in our spec because we were forced to write general rules. Mixing quickcheck and cucumber has been one of my "Some weekend I'll do it" projects for a couple of years now.

15 points by jfrisby 13y ago

I kind of agree with this, but then I see the same tests I've seen elsewhere: Scenario: Editing the nickname of a credential does not change the version When the nickname is modified Then the edit version of the credential should not change This looks like an awesome test. Simple, clear, anyone can read it. What's wrong? One of the following: * It's based on timing and unreliable * It stores state and the steps can't be reused. You simply cannot put the line "Then the edit version of the credentials should not change" elsewhere because the step involves the previous state of the system. This step can only come after this step: Given a credential for a supported vendor These tests all look like they're testing really nice general properties, but they aren't. They're almost always just checking little properties here and there, and rarely doing just what the step says. Which could be fine, but the language doesn't support any idea of specified dependencies. I cannot change a test without understanding the code behind it. What you end up with is a big codebase based on regexes, with very heavy (non-specified) dependencies. You have to be extremely careful not to end up like this, although the tests look more complex afterwards (but you have re-usable steps). I think there's a great place for a quickcheck & cucumber combination.

jms_ 13y ago
Given that the system these scenarios test also manage state, it's not surprising that the step definitions used in the scenario also need to manage some form of state. While it's advisable to avoid unnecessary state between steps in a scenario like this, removing all state limits the ability for a scenario to express the intent the author speaks about.
- IanCal 13y ago
  One option is to move the state to be explicit in the call to the step, which means the test becomes an example rather than a general rule (although that's what it always was, now it's just a bit clearer). It also means the steps can be reused, rather than having steps with dependencies (which aren't specified anywhere or statically enforced). > removing all state limits the ability for a scenario to express the intent the author speaks about. However, writing them like that means each step moves further away from what it says it's doing. The more you do this, the more you might as well not execute them and go and write more examples.
  - jfrisby 13y ago
    To use your example -- how desirable is it to generalize "I change my name" to something "reusable"? I'm finding these days that a pairing of .feature to corresponding step definition file allows for fluid language, simple step definition code, and well-compartmentalized messes (as opposed to the big ball of mud that Cucumber step definitions often become). Code reuse can happen through test helper classes that are called from step definitions as needed. (FactoryGirl would be an example of a very generalized test helper class, in my mind -- but more system-specific ones can be handy, albeit one wants to avoid making stateful / complex test helper code!)
jfrisby 13y ago
I don't quite get how any when/then could NOT be dependent upon previous givens. Also, despite the job-centric nature of the .feature, it wasn't timing-dependent because we mocked Resque to keep an in-process queue that we could dispatch in whatever order we want. So out-of-order arrival was deterministically achievable, etc. And yes, that test is testing a simple property -- it's effectively a unit test, with greater clarity of intent. Not familiar with quickcheck, but I will look into it, thanks!
- IanCal 13y ago
  > I don't quite get how any when/then could NOT be dependent upon previous givens. Given I am called Bob And I change my name to Steve Then my name is not Bob None of these steps are dependent on the ones previous to it, and can be reused elsewhere. The check for "my name is bob" can be used whether or not the previous step has been called. Your test won't pass, which is fine, but it will run and do exactly what it says. The step does what it says even if none of the others are used. It also means I can write a new test using bits from others without worrying about what's in the source code. A test that looks like this: Given I am a person And I change my name Then my name is different Cannot be re-used. The "my name is different" can only be used if you know the ruby code underneath. "my name is different" may not do what you expect unless it comes after "I change my name". What I meant with timing is the only way you can get around storing state is to have the "my name is different" watch for a change in the name, which brings in timing issues. I've seen both implementations used, and both caused problems (the timing one for obvious reasons). > Not familiar with quickcheck, but I will look into it, thanks! I thoroughly recommend it. The haskell version is probably the most advanced, but there are similar versions for most languages (and you can write your own, I had to for AS3). The idea is you express general properties about your system, and then it auto-generates thousands of examples (and if you have a nice library, automatically shrink failing cases for you). For example, the test above would be nicer as something like: X is a string Y is a string X =/= Y Given I am called X And I change my name to Y Then my name is not X And my name is Y Or something like that. This would then generate examples with no-length strings, crazy unicode characters, long strings, different mixings of RTL sections, etc. Much more likely to drive out bugs than a test for Bob and Steve. Some more useful ones would look like this (the first is a test I've written before, but not in this format): X is a number >= 1 Y is an interface element Given I am on element Y When I press Tab X times And I press Shift-Tab X times Then I am on element Y A similar version for navigating in a website and pressing "back" to get back to where you were (to ensure you're not breaking the back button). These are really simple, but powerful tests. I was most sold on the idea when I wrote this (not in this format, but this logic): X is a positive integer Y is a positive integer INSTRUCTION is one of [addElement, removeElement(X), setFocus(X)] MENU is an interface When I perform Y INSTRUCTIONS Then MENU has one focused item or no items at all This then generated thousands of menus of each valid type (vertical, horizontal, grids, etc) and then called library functions to add or remove elements, or move focus. Millions of tests overall. I was using this as a test for my quickcheck implementation, and found it failed. If I set the focus, deleted all the elements and then added a single new one it wasn't focused. When I fixed it, a unit test failed. We has previously specified that was to be the behaviour, but also specified that no matter what there would always be a focused element (if there were any elements at all). The general test drew out an inconsistency in our spec because we were forced to write general rules. Mixing quickcheck and cucumber has been one of my "Some weekend I'll do it" projects for a couple of years now.