3 ms·

If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural lang

If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs. If they only matter to the implementation, they should be in the implementation's integration spec suite (which, discussing technical detail, is not likely to be consumed or produced by business, and so has nothing to gain from being written in natural-language, can simply have natural language comments as you mention. You shouldn't e.g. try to get to any coverage level with only a NL story suite, etc., that is a gross misapplication. NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on, before implementation. Being able to automate testing of that document as validated by the customer is just a bonus. 90% of the value happens before any code is written. If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation, but they are entirely avoidable. A test suite is like any other piece of software and has to be designed (and factored) for the reality that its behavior will change.

crdoconnor 9y ago
>If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs IME it's usually left up to the test programmer what to put in them and they often put vague stuff in there that looks exactly like what you just wrote and that becomes useless both for PMs and non-PMs reading the natural language suite because it leaves out business critical details. It's also a hack often done in order to keep the story suite short because (as I mention below), no inheritance in cucumber. There's a reason why the whole world has not yet flocked to BDD and I don't think it's an issue with BDD itself (I'm a big fan). >NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on I think the idea that it has to be natural language so that a customer can read them is bullshit. This is the exact same mistake the creators of COBOL made, thinking that natural language naturally elucidates things. It doesn't. It's parses very ambiguously. That's a feature if you're flirting with a girl perhaps, but a bug if you're trying to write a precise executable specification. I feel very strongly that the story suite should be written in a language that is easy to parse, can handle parameterization and inheritance but isn't turing complete. It ought to be readable for PMs and still maintainable as part of an integration test suite. >If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. If they glaze over that might just be because they're a bad PM. I've had PMs that glaze over when trying to figure out business-critical edge cases with them because they liked to think of themselves as "big picture guys". That's fine, they just shouldn't be PMs. I think that the divide shouldn't be between "important to business" and "not important to business" but simply "test implementation" vs "specification". >edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. This is actually the main reason why I think PMs typically shouldn't write executable specs. Inexperienced programmers often don't yet have the DRY instinct and the ability to keep a strict separation between implementation details and specification. It's a rare PM that has those skills. Then again, maybe they just need to be trained. I've also had the problem of massive headache inducing repetition and a blurred distinction between implementation detail and specification in huge word document specs. >Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation Yeah, well, repetitive code and poor abstractions are basically a problem writing code in any language. Poor tools make that worse, however. It's one of the reasons why cucumber is a pile of shit: no inheritance. Most stories in a business app are actually forks off existing stories. It's utterly inexcusable that it doesn't have this feature. Integration test suites with high coverage and readable stories do not have to be repetitive. Mine aren't.
- lotyrin 9y ago
  I think we're in agreement then. It's indeed nice to have a PM that could produce details that can be synthesized into a technical spec. It's nice when the PM can work with the customer to actually understand those details. It's nice when PMs can actually care about their products, and work with devs to weigh options. I don't usually have those PMs. Lots of organizations have JIRA Babysitters who show up at your desk whenever the political climate changes to let you know that you should drop the urgent thing he asked you to do yesterday, because there's an urgent thing he's got for you today. The customer though, always wants what they want, and you can carrot instead of stick them, and I have found some cases where I can sit with the customer and do BA/PM with them despite the people with those roles, explain the idea of test automation, show them how it can drive the browser and ask them to describe the application (at a very high level) in natural language in terms of examples, which I translate into Given/When/Then as we are working. I can then explain in status updates "so, the 'simple' scenario for 'user submits an order' is implemented <point to CI output> but we don't yet pass the story for when a coupon code is used. Should we prioritize coupon code scenario or a different feature?" or "So, we have implemented this story according to the examples we were given in May, the behavior you described on this morning's call would read "Given..." instead of "Given...". I can update the story and prioritize that ahead of <whatever>, or should we ship with the original behavior and proceed with <whatever>?" Is it my job or the customer's job to have to do that? Probably not. But it's a way to build a bridge from concrete stupid machines land into fuzzy people land where everything is negotiable, in the absence of the roles or skills to do so without such a tool. Cucumber is indeed poor, the simple 1:1 of step definition strings to their functions and of 1:1 scenario blocks to test executions are unfortunate. I'd like to see a natural language tool where I am specifying invariants to a property-based testing tool. "Given a user" implementation = what does a user mean, well, it has a unicode string 'name' etc. and those map to database model as follows, "when I am on the login page" here are the five pages that have login forms, "When I enter my username" find a field 'username' and fill it with the generated user name ... "Then I should be logged in" logged in users see log out, their username in the corner, have access to their models, logged out users do not, etc. Now this blows up into all the combinatorial options, I find out "hey, if a user has emoji in their password, several invariants are violated." Generally I do not use NL tools or recommend they be used, I'd definitely prefer the environment where they are superfluous as there is a QA automation engineer working with a PM that is willing and able to elicit the necessary details from the customer. Worse yet, all the current implementations seem to be sorely lacking. Just, I don't think all hypothetical NL tools are categorically useless.

crdoconnor 9y ago

>If it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs IME it's usually left up to the test programmer what to put in them and they often put vague stuff in there that looks exactly like what you just wrote and that becomes useless both for PMs and non-PMs reading the natural language suite because it leaves out business critical details. It's also a hack often done in order to keep the story suite short because (as I mention below), no inheritance in cucumber. There's a reason why the whole world has not yet flocked to BDD and I don't think it's an issue with BDD itself (I'm a big fan). >NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on I think the idea that it has to be natural language so that a customer can read them is bullshit. This is the exact same mistake the creators of COBOL made, thinking that natural language naturally elucidates things. It doesn't. It's parses very ambiguously. That's a feature if you're flirting with a girl perhaps, but a bug if you're trying to write a precise executable specification. I feel very strongly that the story suite should be written in a language that is easy to parse, can handle parameterization and inheritance but isn't turing complete. It ought to be readable for PMs and still maintainable as part of an integration test suite. >If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. If they glaze over that might just be because they're a bad PM. I've had PMs that glaze over when trying to figure out business-critical edge cases with them because they liked to think of themselves as "big picture guys". That's fine, they just shouldn't be PMs. I think that the divide shouldn't be between "important to business" and "not important to business" but simply "test implementation" vs "specification". >edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. This is actually the main reason why I think PMs typically shouldn't write executable specs. Inexperienced programmers often don't yet have the DRY instinct and the ability to keep a strict separation between implementation details and specification. It's a rare PM that has those skills. Then again, maybe they just need to be trained. I've also had the problem of massive headache inducing repetition and a blurred distinction between implementation detail and specification in huge word document specs. >Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation Yeah, well, repetitive code and poor abstractions are basically a problem writing code in any language. Poor tools make that worse, however. It's one of the reasons why cucumber is a pile of shit: no inheritance. Most stories in a business app are actually forks off existing stories. It's utterly inexcusable that it doesn't have this feature. Integration test suites with high coverage and readable stories do not have to be repetitive. Mine aren't.

lotyrin 9y ago
I think we're in agreement then. It's indeed nice to have a PM that could produce details that can be synthesized into a technical spec. It's nice when the PM can work with the customer to actually understand those details. It's nice when PMs can actually care about their products, and work with devs to weigh options. I don't usually have those PMs. Lots of organizations have JIRA Babysitters who show up at your desk whenever the political climate changes to let you know that you should drop the urgent thing he asked you to do yesterday, because there's an urgent thing he's got for you today. The customer though, always wants what they want, and you can carrot instead of stick them, and I have found some cases where I can sit with the customer and do BA/PM with them despite the people with those roles, explain the idea of test automation, show them how it can drive the browser and ask them to describe the application (at a very high level) in natural language in terms of examples, which I translate into Given/When/Then as we are working. I can then explain in status updates "so, the 'simple' scenario for 'user submits an order' is implemented <point to CI output> but we don't yet pass the story for when a coupon code is used. Should we prioritize coupon code scenario or a different feature?" or "So, we have implemented this story according to the examples we were given in May, the behavior you described on this morning's call would read "Given..." instead of "Given...". I can update the story and prioritize that ahead of <whatever>, or should we ship with the original behavior and proceed with <whatever>?" Is it my job or the customer's job to have to do that? Probably not. But it's a way to build a bridge from concrete stupid machines land into fuzzy people land where everything is negotiable, in the absence of the roles or skills to do so without such a tool. Cucumber is indeed poor, the simple 1:1 of step definition strings to their functions and of 1:1 scenario blocks to test executions are unfortunate. I'd like to see a natural language tool where I am specifying invariants to a property-based testing tool. "Given a user" implementation = what does a user mean, well, it has a unicode string 'name' etc. and those map to database model as follows, "when I am on the login page" here are the five pages that have login forms, "When I enter my username" find a field 'username' and fill it with the generated user name ... "Then I should be logged in" logged in users see log out, their username in the corner, have access to their models, logged out users do not, etc. Now this blows up into all the combinatorial options, I find out "hey, if a user has emoji in their password, several invariants are violated." Generally I do not use NL tools or recommend they be used, I'd definitely prefer the environment where they are superfluous as there is a QA automation engineer working with a PM that is willing and able to elicit the necessary details from the customer. Worse yet, all the current implementations seem to be sorely lacking. Just, I don't think all hypothetical NL tools are categorically useless.