3 ms·

This may sound a bit snarky, but does anyone here recommend Cucumber tests? I've found they add a lot of overhead to writing without much benefit. Non-technical

This may sound a bit snarky, but does anyone here recommend Cucumber tests? I've found they add a lot of overhead to writing without much benefit. Non-technical employees can't really write them either so they be revised by a developer. If that's the case then we're back to where we started.

reitanqild 9y ago
Using it actively (not voluntarily though) and I see where you come from. That said I haven't made up my mind yet if it is good and I just don't get it yet or if it is a mostly a waste of time.
triskweline 9y ago
We're heavy users of Cucumber at our development agency. I've never seen non-technical users write Cucumber scenarios either, but we find them very useful for another reason. The problem with test suites is that, as they grow and age, it becomes harder and harder to understand why tests are testing what they test. When a years old test fails because 5 is not 7, you cannot really fix it unless you understand why it was supposed to equal 7 in the first place. Cucumber is a way to describe requirements at a higher level of abstraction. They provide rationale on top of mere assertions of behavior. This is worth the small overhead of writing step definitions. I guess you can get the same by heavily commenting your tests. But you cannot enforce this easily in a team.
- choward 9y ago
  Nothing is stopping you from using functions with descriptive names without cucumber. Cucumber forces you to write functions (steps) however you can still create bad descriptions just like function names. Cucumber just allows you to define functions using a different syntax than the language your using. I don't see any wins here.
  - maxxxxx 9y ago
    That has been my recommendation when people started talking about Cucumber at my compnay. Write a library that has super descriptive names like "EnterTextIntoPriceField(400)" and you get the same benefit Cucumber gives you but you have real programming language if needed. Our biz people can't write useful requirements so I doubt they can write meaningful Cucumber scripts either.
    - jondubois 9y ago
      I worked for a very large company that started rolling out cucumber. It didn't go anywhere - Product owners simply did not have the technical ability to write cucumber tests. I think that they add way too much complexity. It doesn't make sense to try to write technical software tests at such a high level of abstraction. When you try to test something at a really high level along the lines of "As a user, when I click on X, I want Y to happen" the low level technical ramifications of this can be enormous. There is a huge amount of underlying complexity which the one-line sentence simply cannot capture and which is prone to change and needs to be tested separately... Probably not in a way that a non-technical manager would understand unfortunately. Ultimately the tests are not granular enough to be useful as tests for developers and they have too much overhead to be useful as documentation for non managers.
bpicolo 9y ago
I've had mostly negative experiences with them. With a DSL-driven BDD framework (like rspec) you get autocomplete and debugging and every other language feature without really losing any of the descriptiveness. With cucumber you get lots of pain in both those areas, without anything that feels like an improvement along another metric.
optimuspaul 9y ago
We don't use Cucumber, but another BDD framework. We have sort of non-technical employees that write them out. Generally the developers do a little tweaking, but mostly for spelling inconsistencies where reuse of steps is expected. I really think it's great, especially when steps can be reused across many tests. I don't think that you can't achieve the "same" effect with unit test frameworks, but I like the approach.
Zhyl 9y ago
Using it for a current project: Phase 1 Non-technical and semi-technical business analysts write scenarios as acceptance criteria for the requirements. They are still so high level that the devs need to re-write or refactor the majority of them to be able to get any sort of re-use out of them. Phase 2 Writing scenarios is largely seen as a time sink and the test coverage is falling behind. This is then changes so that the BAs write a headline scenario and some high-level bullet points (higher level than they were doing already) which are then fleshed out by devs. Devs re-write statements in ways that aren't very good english ('When I log in as a "Tim_test5"') and scenarios are no longer good to show to the customer. Phase 3 Test writing is still slow, so BAs and testers start writing and performing manual tests to close the user stories. Step 'library' and feature files are building up but still only a relative handful of tests are automated. So yeah, in an ideal world they would be perfect, but I would recommend the following: * Don't include BDD in your 'definition of done'. You will never close anything. * Only write BDD for things that are stable - some changes can mean that tests need to be updated, split or take more arguments. Syncing up changes to statements in feature files is extra overhead. * Deciding on wording to reflect business meaning of a statement and technical meaning of a statement is harder than you think. Factor in time for this. * If your team haven't used BDD before, create some central resource with statements in that can be used by non-technical users to build up scenarios lego-style. If what they need isn't there, then it can be added. This is especially true when a set of actions is easily abstracted to a single business action ("I send the request" or something which takes many clicks and form filling).
odammit 9y ago
I’ve used it frequently in the past. Biz would write up tickets in our own CMS. The form would help them format their stories and would also autocomplete based on steps in previous stories. Upon submitting/editing it would create a ticket with the story. We also ended up having it create a branch once it was scheduled by product so engineers could git checkout storyXXX and see where the cucumber steps went yellow. There is a pretty cool SaaS product that does this now more or less that I’ve only gotten to play with http://www.simian.io/ http://www.simian.io/
seanwilson 9y ago
> Non-technical employees can't really write them either so they be revised by a developer This is the part I could never buy into either. You cannot escape that writing tests is the same thing as programming (you need to break the problem down into steps, formalise what you want to check, use abstractions) so non-programmers aren't good at writing tests without rigorous training. Non-programmers tend to need a lot of help even describing what they want in plain English as well because they just aren't used to thinking like programmers. I've worked with non-technical people for years who still don't understand that "it's not working" isn't enough information to go on. As for overhead, it might make sense if you're working on one project in a single language for a long time but the setup and configuration overhead has been significant when I've tried BDD tests. I'd rather just use descriptive function names in tests as then you can take advantage of autocompletion, automated refactoring, jump to definition shortcuts, static checking, language features etc.

reitanqild 9y ago

Using it actively (not voluntarily though) and I see where you come from. That said I haven't made up my mind yet if it is good and I just don't get it yet or if it is a mostly a waste of time.

triskweline 9y ago

We're heavy users of Cucumber at our development agency. I've never seen non-technical users write Cucumber scenarios either, but we find them very useful for another reason. The problem with test suites is that, as they grow and age, it becomes harder and harder to understand why tests are testing what they test. When a years old test fails because 5 is not 7, you cannot really fix it unless you understand why it was supposed to equal 7 in the first place. Cucumber is a way to describe requirements at a higher level of abstraction. They provide rationale on top of mere assertions of behavior. This is worth the small overhead of writing step definitions. I guess you can get the same by heavily commenting your tests. But you cannot enforce this easily in a team.

choward 9y ago
Nothing is stopping you from using functions with descriptive names without cucumber. Cucumber forces you to write functions (steps) however you can still create bad descriptions just like function names. Cucumber just allows you to define functions using a different syntax than the language your using. I don't see any wins here.
- maxxxxx 9y ago
  That has been my recommendation when people started talking about Cucumber at my compnay. Write a library that has super descriptive names like "EnterTextIntoPriceField(400)" and you get the same benefit Cucumber gives you but you have real programming language if needed. Our biz people can't write useful requirements so I doubt they can write meaningful Cucumber scripts either.
  - jondubois 9y ago
    I worked for a very large company that started rolling out cucumber. It didn't go anywhere - Product owners simply did not have the technical ability to write cucumber tests. I think that they add way too much complexity. It doesn't make sense to try to write technical software tests at such a high level of abstraction. When you try to test something at a really high level along the lines of "As a user, when I click on X, I want Y to happen" the low level technical ramifications of this can be enormous. There is a huge amount of underlying complexity which the one-line sentence simply cannot capture and which is prone to change and needs to be tested separately... Probably not in a way that a non-technical manager would understand unfortunately. Ultimately the tests are not granular enough to be useful as tests for developers and they have too much overhead to be useful as documentation for non managers.

bpicolo 9y ago

I've had mostly negative experiences with them. With a DSL-driven BDD framework (like rspec) you get autocomplete and debugging and every other language feature without really losing any of the descriptiveness. With cucumber you get lots of pain in both those areas, without anything that feels like an improvement along another metric.

optimuspaul 9y ago

We don't use Cucumber, but another BDD framework. We have sort of non-technical employees that write them out. Generally the developers do a little tweaking, but mostly for spelling inconsistencies where reuse of steps is expected. I really think it's great, especially when steps can be reused across many tests. I don't think that you can't achieve the "same" effect with unit test frameworks, but I like the approach.

Zhyl 9y ago

Using it for a current project: Phase 1 Non-technical and semi-technical business analysts write scenarios as acceptance criteria for the requirements. They are still so high level that the devs need to re-write or refactor the majority of them to be able to get any sort of re-use out of them. Phase 2 Writing scenarios is largely seen as a time sink and the test coverage is falling behind. This is then changes so that the BAs write a headline scenario and some high-level bullet points (higher level than they were doing already) which are then fleshed out by devs. Devs re-write statements in ways that aren't very good english ('When I log in as a "Tim_test5"') and scenarios are no longer good to show to the customer. Phase 3 Test writing is still slow, so BAs and testers start writing and performing manual tests to close the user stories. Step 'library' and feature files are building up but still only a relative handful of tests are automated. So yeah, in an ideal world they would be perfect, but I would recommend the following: * Don't include BDD in your 'definition of done'. You will never close anything. * Only write BDD for things that are stable - some changes can mean that tests need to be updated, split or take more arguments. Syncing up changes to statements in feature files is extra overhead. * Deciding on wording to reflect business meaning of a statement and technical meaning of a statement is harder than you think. Factor in time for this. * If your team haven't used BDD before, create some central resource with statements in that can be used by non-technical users to build up scenarios lego-style. If what they need isn't there, then it can be added. This is especially true when a set of actions is easily abstracted to a single business action ("I send the request" or something which takes many clicks and form filling).

odammit 9y ago

I’ve used it frequently in the past. Biz would write up tickets in our own CMS. The form would help them format their stories and would also autocomplete based on steps in previous stories. Upon submitting/editing it would create a ticket with the story. We also ended up having it create a branch once it was scheduled by product so engineers could git checkout storyXXX and see where the cucumber steps went yellow. There is a pretty cool SaaS product that does this now more or less that I’ve only gotten to play with http://www.simian.io/ http://www.simian.io/

seanwilson 9y ago

> Non-technical employees can't really write them either so they be revised by a developer This is the part I could never buy into either. You cannot escape that writing tests is the same thing as programming (you need to break the problem down into steps, formalise what you want to check, use abstractions) so non-programmers aren't good at writing tests without rigorous training. Non-programmers tend to need a lot of help even describing what they want in plain English as well because they just aren't used to thinking like programmers. I've worked with non-technical people for years who still don't understand that "it's not working" isn't enough information to go on. As for overhead, it might make sense if you're working on one project in a single language for a long time but the setup and configuration overhead has been significant when I've tried BDD tests. I'd rather just use descriptive function names in tests as then you can take advantage of autocompletion, automated refactoring, jump to definition shortcuts, static checking, language features etc.