5 ms·

Cucumber: BDD Testing and Collaboration Tools

14 points by kaycebasques 3y ago

packetlost 3y ago
I cannot overstate how much I detest BDD and Cucumber.
- dlahoda 3y ago
  why? i know that 99% is badly written sure. but bdd cucumber style testing if done right can be good. problem i saw people just incapable.
  - ljm 3y ago
    “If done right” is carrying a lot of weight there. I don’t mind it if you use cucumber and it still serves the purpose of describing your code’s behaviour, so the output is readable. RSpec’s documentation is a nice example of that. The problem is if people just use it as a syntax over a webdriver, which means that your test behaviour and logic is arbitrarily split into two different languages, the cucumber output is useless, the tests are needlessly slow, and it’s really hard to debug. Cucumber itself is not able to add value there. But that’s not unique to cucumber either. I don’t think many people realise that test writing and architecture is its own skill in programming and so many suites turn into write-only cludges that take 12 hours to run on a cluster of 50 CI workers.
    - drewcoo 3y ago
      > The problem is if people just use it as a syntax over a webdriver Cucumber and WebDriver? Two problematic frameworks does not equal a sane one.
  - throwbadubadu 3y ago
    Aren't incapable people the norm and usual problem? ;D Found it quite neutral, not too bad, but also not really helpful.. some nice ideas especially that cross functional collaboration, but in the end the engineers had overhead and hated to write the wrappers, and the other stakeholders still couldn't or didn't want to read/write docs/test, or however it was intended. Admitted, maybe I belong to that incapable crowd as well ;)
- DarkNova6 3y ago
  What's wrong with BDD? Admittedly, I never dealt with Cucumber.
  - packetlost 3y ago
    BDD is, IME, only really good for testing UI behavior (I've seen some pretty decent JS BDD testing frameworks, but have limitex xp with them). It can be good at that because those behaviors can usually be expressed well enough in natural language. But I've seen it shoehorned into data-centric UI-less applications that are just a massive pain to deal with. Half the point of most testing frameworks is getting more done with less typing/overhead, but Cucumber and BDD is sorta the exact opposite: do less with more typing by trying to describe complex concepts in natural language. Cucumber mostly exists to upsell naive MBAs and project managers on their "pro" tools.
  - hitchstory 3y ago
    BDD is great as a concept but Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL for implementing it: * It's verbose. * It's not type safe. * It desperately needs some kind of inheritance - most user stories branch from other user stories (e.g. put product in shopping cart->purchase product). The only way people who use it can deal with it is by concealing critical details in code or by writing massively repetitive stories. * What other language has a parser that routinely asks you to write regexes to get things to parse? None of these things are fixable and the creators have largely reacted to the complaints with gaslighting: "you're just using it wrong". It can be used effectively if you align the stars correctly but usually not. The one thing it does get right is not being turing complete - there's clear value in not using code to describe scenarios.
    - drewcoo 3y ago
      > Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL I wouldn't lump RSpec in with the rest of those. Spec-style frameworks (like granddaddy RSpec) are terser. They don't have layers of indirection. They're intended to help devs think about how code will be used, not to make code look like English.
hmmdog 3y ago
Poorly done BDD has made me detest bdd and all of the tools, especially when cucumber is used as a testing tool and not as a collaborative tool. I would say most companies use cucumber as a testing tool and say they doing bdd. I would actually like to see a company doing true bdd I have not seen one. Didn’t the founder post here and said something along the lines of hating bdd now? Edit: oh I see cucumber was bought by smartbear and then they fired the last cofounder https://mattwynne.net/new-beginning https://mattwynne.net/new-beginning
jaggederest 3y ago
Contrary to many people I've had a bunch of positive experience using Cucumber in very specific situation(s): - I was an outside consultant working with nontechnical but very process oriented management teams - There was low trust in existing technical workflows: they had been burned by bugs, failure to deliver, and overpromising so a big part of my task was building rigor and consistency into the engineering workflows - They were extremely engaged in managing the workflows (the core of the business was doing the thing that we built an app to do) In that context, they loved having a printed out (!!) copy of all the cucumber tests running on the most recent build so that they could see what was passing and what was failing: Color printer, green line passes, red line fails. Once I got them into the habit of writing "given when then" statements to define existing features, bugs, and functionality, they were happy as a clam being able to specify things. Eliminated a lot of the usual back and forth over "yes but what exactly do you mean" that non-technical people often create due to not being clear and exact enough for technical purposes. I would implement the parsing and selenium-based testing that validated whether the cucumber was passing and we would have a shared understanding of where the code was at and whether there were any regressions. In the occasional case where my understanding didn't match what they expected, we had context to work together to figure it out. All in all it earned me a fair amount of money and lowered the stress of being a consultant for those companies substantially so I have a lingering fondness for it. But like I said it's a very specific situation to deliver that kind of value.
- hitchstory 3y ago
  I've had, I think, 1 working experience and 6 or so highly negative and it was mostly the nature of the system under of test that made it work or fail. For simple and very clear Given/When/Then scenarios with clear inputs and results that were easily represented with a small amount of text it worked. Think a trading system with a few codes and a small amount of data on the left and an expected result like a calculation on the right. Where there was a lot of stories that branched off from one another (e.g. a website with lots of potential user flows) it inevitably led to stories that were too vague or too repetitive or more usually, both. I'm convinced that it does work for a certain kind of application for which the DSL is capable of representing its scenarios, but it's not the kind most of us develop.
kaycebasques 3y ago
Hi HN, I submitted this because I'm not that familiar with BDD [1] and I'm curious about people's experiences with it. Glad to see that the discussion is progressing as I had hoped! [1] https://en.wikipedia.org/wiki/Behavior-driven_development https://en.wikipedia.org/wiki/Behavior-driven_development
liampulles 3y ago
Getting cucumber setup is consistently a huge pain in the ass. But it's worth it - I find acceptance tests are great for testing systems from a data perspective, i.e. what comes in to our app, what we send to other apps, what we respond with, etc. The data coming in and out is ultimately what I care about ensuring. I haven't yet seen acceptance tests that are readable by "business", but they do help remind me what the usecases in our system do.
- bradgessler 3y ago
  The same can be achieved with Rspec these days. That wasn't true when Cucumber arrived on scene, but that's changed.
  - liampulles 3y ago
    Does it work with Go?
    - bradgessler 3y ago
      Nope, just Ruby. Ginko (https://onsi.github.io/ginkgo/ https://onsi.github.io/ginkgo/) looks the closest, but I'm not sure how close. I can speak for RSpec and say the output of the test runs can be formatted such that it generates specifications that are as "non-technical" as Cucumber. If you scroll down a bit from https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-traverses-the-spec-hierarchy https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-trave... you'll see a similar output.
deleted 3y ago
[deleted]
jlkuester7 3y ago
My experience with using Cucumber was mostly positive, but had to be done very carefully or it would become a huge mess of rube goldberg code behind the scenes. The number of scenarios and steps can become crazy repetitive unless you leverage Rules, Backgrounds, and Scenario Outlines. Probably the most important thing was that with a lot of test frameworks, having poorly architected test utility code is annoying and more time-consuming, but has a limited blast radius. Hacky tests can be easily isolated from the other test code. In Cucumber, proper architecture and reusability are paramount. If you screw this up, your tests become unmaintainable really quickly!
- jlkuester7 3y ago
  The main value I say in it was that our company has some regulatory obligations to: - Maintain documented requirements for ALL CODE FUNCTIONALITY - Maintain tests for ALL REQUIREMENTS - Be able to produce reports for auditors that prove linkage between the requirements and the tests (to prove that all requirements were tested) In this context, Cucumber was really nice since it automatically solved all three of these at once! For general testing purposes, though, I think Cucumber has too much overhead for it to be worth using...
  - jaggederest 3y ago
    Yes, those are exactly the scenarios I bring it out for - when someone wants a VERY clear, VERY well defined, completely comprehensive list of every single thing the system can do, and whether they are working today in an automated fashion.
_joac_ 3y ago
I manage a behemot with 13k cucumber tests. It incentivizes DRY, that for testing is not good. You end up having step definitions with highly complex implementations, with loops and conditionals. Just use a programming language.

14 points by kaycebasques 3y ago

packetlost 3y ago

I cannot overstate how much I detest BDD and Cucumber.

dlahoda 3y ago
why? i know that 99% is badly written sure. but bdd cucumber style testing if done right can be good. problem i saw people just incapable.
- ljm 3y ago
  “If done right” is carrying a lot of weight there. I don’t mind it if you use cucumber and it still serves the purpose of describing your code’s behaviour, so the output is readable. RSpec’s documentation is a nice example of that. The problem is if people just use it as a syntax over a webdriver, which means that your test behaviour and logic is arbitrarily split into two different languages, the cucumber output is useless, the tests are needlessly slow, and it’s really hard to debug. Cucumber itself is not able to add value there. But that’s not unique to cucumber either. I don’t think many people realise that test writing and architecture is its own skill in programming and so many suites turn into write-only cludges that take 12 hours to run on a cluster of 50 CI workers.
  - drewcoo 3y ago
    > The problem is if people just use it as a syntax over a webdriver Cucumber and WebDriver? Two problematic frameworks does not equal a sane one.
- throwbadubadu 3y ago
  Aren't incapable people the norm and usual problem? ;D Found it quite neutral, not too bad, but also not really helpful.. some nice ideas especially that cross functional collaboration, but in the end the engineers had overhead and hated to write the wrappers, and the other stakeholders still couldn't or didn't want to read/write docs/test, or however it was intended. Admitted, maybe I belong to that incapable crowd as well ;)
DarkNova6 3y ago
What's wrong with BDD? Admittedly, I never dealt with Cucumber.
- packetlost 3y ago
  BDD is, IME, only really good for testing UI behavior (I've seen some pretty decent JS BDD testing frameworks, but have limitex xp with them). It can be good at that because those behaviors can usually be expressed well enough in natural language. But I've seen it shoehorned into data-centric UI-less applications that are just a massive pain to deal with. Half the point of most testing frameworks is getting more done with less typing/overhead, but Cucumber and BDD is sorta the exact opposite: do less with more typing by trying to describe complex concepts in natural language. Cucumber mostly exists to upsell naive MBAs and project managers on their "pro" tools.
- hitchstory 3y ago
  BDD is great as a concept but Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL for implementing it: * It's verbose. * It's not type safe. * It desperately needs some kind of inheritance - most user stories branch from other user stories (e.g. put product in shopping cart->purchase product). The only way people who use it can deal with it is by concealing critical details in code or by writing massively repetitive stories. * What other language has a parser that routinely asks you to write regexes to get things to parse? None of these things are fixable and the creators have largely reacted to the complaints with gaslighting: "you're just using it wrong". It can be used effectively if you align the stars correctly but usually not. The one thing it does get right is not being turing complete - there's clear value in not using code to describe scenarios.
  - drewcoo 3y ago
    > Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL I wouldn't lump RSpec in with the rest of those. Spec-style frameworks (like granddaddy RSpec) are terser. They don't have layers of indirection. They're intended to help devs think about how code will be used, not to make code look like English.

hmmdog 3y ago

Poorly done BDD has made me detest bdd and all of the tools, especially when cucumber is used as a testing tool and not as a collaborative tool. I would say most companies use cucumber as a testing tool and say they doing bdd. I would actually like to see a company doing true bdd I have not seen one. Didn’t the founder post here and said something along the lines of hating bdd now? Edit: oh I see cucumber was bought by smartbear and then they fired the last cofounder https://mattwynne.net/new-beginning https://mattwynne.net/new-beginning

jaggederest 3y ago

Contrary to many people I've had a bunch of positive experience using Cucumber in very specific situation(s): - I was an outside consultant working with nontechnical but very process oriented management teams - There was low trust in existing technical workflows: they had been burned by bugs, failure to deliver, and overpromising so a big part of my task was building rigor and consistency into the engineering workflows - They were extremely engaged in managing the workflows (the core of the business was doing the thing that we built an app to do) In that context, they loved having a printed out (!!) copy of all the cucumber tests running on the most recent build so that they could see what was passing and what was failing: Color printer, green line passes, red line fails. Once I got them into the habit of writing "given when then" statements to define existing features, bugs, and functionality, they were happy as a clam being able to specify things. Eliminated a lot of the usual back and forth over "yes but what exactly do you mean" that non-technical people often create due to not being clear and exact enough for technical purposes. I would implement the parsing and selenium-based testing that validated whether the cucumber was passing and we would have a shared understanding of where the code was at and whether there were any regressions. In the occasional case where my understanding didn't match what they expected, we had context to work together to figure it out. All in all it earned me a fair amount of money and lowered the stress of being a consultant for those companies substantially so I have a lingering fondness for it. But like I said it's a very specific situation to deliver that kind of value.

hitchstory 3y ago
I've had, I think, 1 working experience and 6 or so highly negative and it was mostly the nature of the system under of test that made it work or fail. For simple and very clear Given/When/Then scenarios with clear inputs and results that were easily represented with a small amount of text it worked. Think a trading system with a few codes and a small amount of data on the left and an expected result like a calculation on the right. Where there was a lot of stories that branched off from one another (e.g. a website with lots of potential user flows) it inevitably led to stories that were too vague or too repetitive or more usually, both. I'm convinced that it does work for a certain kind of application for which the DSL is capable of representing its scenarios, but it's not the kind most of us develop.

kaycebasques 3y ago

Hi HN, I submitted this because I'm not that familiar with BDD [1] and I'm curious about people's experiences with it. Glad to see that the discussion is progressing as I had hoped! [1] https://en.wikipedia.org/wiki/Behavior-driven_development https://en.wikipedia.org/wiki/Behavior-driven_development

liampulles 3y ago

Getting cucumber setup is consistently a huge pain in the ass. But it's worth it - I find acceptance tests are great for testing systems from a data perspective, i.e. what comes in to our app, what we send to other apps, what we respond with, etc. The data coming in and out is ultimately what I care about ensuring. I haven't yet seen acceptance tests that are readable by "business", but they do help remind me what the usecases in our system do.

bradgessler 3y ago
The same can be achieved with Rspec these days. That wasn't true when Cucumber arrived on scene, but that's changed.
- liampulles 3y ago
  Does it work with Go?
  - bradgessler 3y ago
    Nope, just Ruby. Ginko (https://onsi.github.io/ginkgo/ https://onsi.github.io/ginkgo/) looks the closest, but I'm not sure how close. I can speak for RSpec and say the output of the test runs can be formatted such that it generates specifications that are as "non-technical" as Cucumber. If you scroll down a bit from https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-traverses-the-spec-hierarchy https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-trave... you'll see a similar output.

deleted 3y ago

[deleted]

jlkuester7 3y ago

My experience with using Cucumber was mostly positive, but had to be done very carefully or it would become a huge mess of rube goldberg code behind the scenes. The number of scenarios and steps can become crazy repetitive unless you leverage Rules, Backgrounds, and Scenario Outlines. Probably the most important thing was that with a lot of test frameworks, having poorly architected test utility code is annoying and more time-consuming, but has a limited blast radius. Hacky tests can be easily isolated from the other test code. In Cucumber, proper architecture and reusability are paramount. If you screw this up, your tests become unmaintainable really quickly!

jlkuester7 3y ago
The main value I say in it was that our company has some regulatory obligations to: - Maintain documented requirements for ALL CODE FUNCTIONALITY - Maintain tests for ALL REQUIREMENTS - Be able to produce reports for auditors that prove linkage between the requirements and the tests (to prove that all requirements were tested) In this context, Cucumber was really nice since it automatically solved all three of these at once! For general testing purposes, though, I think Cucumber has too much overhead for it to be worth using...
- jaggederest 3y ago
  Yes, those are exactly the scenarios I bring it out for - when someone wants a VERY clear, VERY well defined, completely comprehensive list of every single thing the system can do, and whether they are working today in an automated fashion.

_joac_ 3y ago

I manage a behemot with 13k cucumber tests. It incentivizes DRY, that for testing is not good. You end up having step definitions with highly complex implementations, with loops and conditionals. Just use a programming language.