3 ms·

I find utility in them being understood by PMs, even if not written. Also, the overly-parameterized kinds of reusable statements like "Click on 'whatever' then

I find utility in them being understood by PMs, even if not written. Also, the overly-parameterized kinds of reusable statements like "Click on 'whatever' then fill in 'name' with 'stuff'" are an anti-pattern (they describe what not why, and they fail to provide actual reuse like a page object would provide so you get costly maintenance). They should read "When I log in" or "When I register as a user" they should interact with high-level objects provided in your test suite,they should be purely whole-system-user-interface integration tests, and you should have very few of them, just enough to answer "At the human level, what does this thing do for users, and why" Any tool can be misapplied, poison is in the dosage, etc.

crdoconnor 9y ago
>They should read "When I log in" or "When I register as a user" I hate those kinds of stories. Half of the relevant scenario information is concealed in a mess of turing complete code. Which user logged in? How did you register as a user? I've read tons of stories like this and they're often next to useless. You might as well just write the test in regular code and put a comment at the top of the file and get the PM to read that instead. If it's something the user does I don't want the executable spec to bury those details in turing complete code because writing it in the higher level language is "hard". I want them surfaced in the readable scenario. The fact that Cucumber makes it difficult to write readable, terse, deduplicated, parameterized user stories is, IMHO, a problem with cucumber, not a problem with the people who write it.
- lotyrin 9y ago
  If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs. If they only matter to the implementation, they should be in the implementation's integration spec suite (which, discussing technical detail, is not likely to be consumed or produced by business, and so has nothing to gain from being written in natural-language, can simply have natural language comments as you mention. You shouldn't e.g. try to get to any coverage level with only a NL story suite, etc., that is a gross misapplication. NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on, before implementation. Being able to automate testing of that document as validated by the customer is just a bonus. 90% of the value happens before any code is written. If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation, but they are entirely avoidable. A test suite is like any other piece of software and has to be designed (and factored) for the reality that its behavior will change.
  - crdoconnor 9y ago
    >If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs IME it's usually left up to the test programmer what to put in them and they often put vague stuff in there that looks exactly like what you just wrote and that becomes useless both for PMs and non-PMs reading the natural language suite because it leaves out business critical details. It's also a hack often done in order to keep the story suite short because (as I mention below), no inheritance in cucumber. There's a reason why the whole world has not yet flocked to BDD and I don't think it's an issue with BDD itself (I'm a big fan). >NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on I think the idea that it has to be natural language so that a customer can read them is bullshit. This is the exact same mistake the creators of COBOL made, thinking that natural language naturally elucidates things. It doesn't. It's parses very ambiguously. That's a feature if you're flirting with a girl perhaps, but a bug if you're trying to write a precise executable specification. I feel very strongly that the story suite should be written in a language that is easy to parse, can handle parameterization and inheritance but isn't turing complete. It ought to be readable for PMs and still maintainable as part of an integration test suite. >If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. If they glaze over that might just be because they're a bad PM. I've had PMs that glaze over when trying to figure out business-critical edge cases with them because they liked to think of themselves as "big picture guys". That's fine, they just shouldn't be PMs. I think that the divide shouldn't be between "important to business" and "not important to business" but simply "test implementation" vs "specification". >edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. This is actually the main reason why I think PMs typically shouldn't write executable specs. Inexperienced programmers often don't yet have the DRY instinct and the ability to keep a strict separation between implementation details and specification. It's a rare PM that has those skills. Then again, maybe they just need to be trained. I've also had the problem of massive headache inducing repetition and a blurred distinction between implementation detail and specification in huge word document specs. >Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation Yeah, well, repetitive code and poor abstractions are basically a problem writing code in any language. Poor tools make that worse, however. It's one of the reasons why cucumber is a pile of shit: no inheritance. Most stories in a business app are actually forks off existing stories. It's utterly inexcusable that it doesn't have this feature. Integration test suites with high coverage and readable stories do not have to be repetitive. Mine aren't.

crdoconnor 9y ago

>They should read "When I log in" or "When I register as a user" I hate those kinds of stories. Half of the relevant scenario information is concealed in a mess of turing complete code. Which user logged in? How did you register as a user? I've read tons of stories like this and they're often next to useless. You might as well just write the test in regular code and put a comment at the top of the file and get the PM to read that instead. If it's something the user does I don't want the executable spec to bury those details in turing complete code because writing it in the higher level language is "hard". I want them surfaced in the readable scenario. The fact that Cucumber makes it difficult to write readable, terse, deduplicated, parameterized user stories is, IMHO, a problem with cucumber, not a problem with the people who write it.

lotyrin 9y ago
If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs. If they only matter to the implementation, they should be in the implementation's integration spec suite (which, discussing technical detail, is not likely to be consumed or produced by business, and so has nothing to gain from being written in natural-language, can simply have natural language comments as you mention. You shouldn't e.g. try to get to any coverage level with only a NL story suite, etc., that is a gross misapplication. NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on, before implementation. Being able to automate testing of that document as validated by the customer is just a bonus. 90% of the value happens before any code is written. If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation, but they are entirely avoidable. A test suite is like any other piece of software and has to be designed (and factored) for the reality that its behavior will change.
- crdoconnor 9y ago
  >If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs IME it's usually left up to the test programmer what to put in them and they often put vague stuff in there that looks exactly like what you just wrote and that becomes useless both for PMs and non-PMs reading the natural language suite because it leaves out business critical details. It's also a hack often done in order to keep the story suite short because (as I mention below), no inheritance in cucumber. There's a reason why the whole world has not yet flocked to BDD and I don't think it's an issue with BDD itself (I'm a big fan). >NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on I think the idea that it has to be natural language so that a customer can read them is bullshit. This is the exact same mistake the creators of COBOL made, thinking that natural language naturally elucidates things. It doesn't. It's parses very ambiguously. That's a feature if you're flirting with a girl perhaps, but a bug if you're trying to write a precise executable specification. I feel very strongly that the story suite should be written in a language that is easy to parse, can handle parameterization and inheritance but isn't turing complete. It ought to be readable for PMs and still maintainable as part of an integration test suite. >If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. If they glaze over that might just be because they're a bad PM. I've had PMs that glaze over when trying to figure out business-critical edge cases with them because they liked to think of themselves as "big picture guys". That's fine, they just shouldn't be PMs. I think that the divide shouldn't be between "important to business" and "not important to business" but simply "test implementation" vs "specification". >edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. This is actually the main reason why I think PMs typically shouldn't write executable specs. Inexperienced programmers often don't yet have the DRY instinct and the ability to keep a strict separation between implementation details and specification. It's a rare PM that has those skills. Then again, maybe they just need to be trained. I've also had the problem of massive headache inducing repetition and a blurred distinction between implementation detail and specification in huge word document specs. >Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation Yeah, well, repetitive code and poor abstractions are basically a problem writing code in any language. Poor tools make that worse, however. It's one of the reasons why cucumber is a pile of shit: no inheritance. Most stories in a business app are actually forks off existing stories. It's utterly inexcusable that it doesn't have this feature. Integration test suites with high coverage and readable stories do not have to be repetitive. Mine aren't.