4 ms·
Are natural language specifications useful?
- tluyben2 9y agoWe started with everything in natural on our products, but rapidly I found myself wanting something more formal. I have been working on formal specs for our products in Coq in the past but it took me too much time and outside solely software dev it got me stuck. So lately I have been using TLA+ (I used it many years ago but did not find it formal enough in my then naive age and experience) and I must say it is great. The learning curve is quite steep but not as steep as Idris or Coq (also less formal) and far more practical. Think the author could have used TLA+ although I did not get a full appreciation about his executable specs from that article.
- balfirevic 9y agoCan you share more about what kind of programs you're specifying with TLA+? Business applications, particular algorithms, or something else entirely?
- tluyben2 9y agoOur firmware, encryption algorithms, app/web and deployments. I currently finished the firmware and deployments and working on the rest now. The firmware helped a lot as we cannot change that in the field so we need to agree on a non ambiguous way of communicating the specs for it.
- gerbilly 9y agoI've never found natural language interfaces to be useful in programming. The hard part of programming is specifying the problem in an unambiguous way. If anything natural languages can make this harder to accomplish.
- mrec 9y agoBDD test definitions are the one that really annoy me. As far as I can tell, they're written in "natural language" (i.e. a blessed set of specific parameterizable phrases) purely so that they can be written by PMs rather than devs. If that has ever actually happened in the entire history of anything, I haven't witnessed it.
- lotyrin 9y agoI find utility in them being understood by PMs, even if not written. Also, the overly-parameterized kinds of reusable statements like "Click on 'whatever' then fill in 'name' with 'stuff'" are an anti-pattern (they describe what not why, and they fail to provide actual reuse like a page object would provide so you get costly maintenance). They should read "When I log in" or "When I register as a user" they should interact with high-level objects provided in your test suite,they should be purely whole-system-user-interface integration tests, and you should have very few of them, just enough to answer "At the human level, what does this thing do for users, and why" Any tool can be misapplied, poison is in the dosage, etc.
- crdoconnor 9y ago>They should read "When I log in" or "When I register as a user" I hate those kinds of stories. Half of the relevant scenario information is concealed in a mess of turing complete code. Which user logged in? How did you register as a user? I've read tons of stories like this and they're often next to useless. You might as well just write the test in regular code and put a comment at the top of the file and get the PM to read that instead. If it's something the user does I don't want the executable spec to bury those details in turing complete code because writing it in the higher level language is "hard". I want them surfaced in the readable scenario. The fact that Cucumber makes it difficult to write readable, terse, deduplicated, parameterized user stories is, IMHO, a problem with cucumber, not a problem with the people who write it.
- lotyrin 9y agoIf it matters what the user's username was or what button they clicked or what form they're on --- to the business --- then sure, describe those in natural language specs. If they only matter to the implementation, they should be in the implementation's integration spec suite (which, discussing technical detail, is not likely to be consumed or produced by business, and so has nothing to gain from being written in natural-language, can simply have natural language comments as you mention. You shouldn't e.g. try to get to any coverage level with only a NL story suite, etc., that is a gross misapplication. NL Story suite has a very simple utility: the customer can read them and say "yeah, this is an example of what should happen" or go "no, that's not quite what I meant" early on, before implementation. Being able to automate testing of that document as validated by the customer is just a bonus. 90% of the value happens before any code is written. If you are writing an executable natural language story and have no customers to read it, or if you are covering details they will glaze over while reading because they can't imagine or validate what is described, then you're using the wrong tool. edit: Even a suite in code, for technical users, should strive to have more reuse than having each test describe exactly what actions are taken, since that's too hard to maintain. If your tests aren't DRY, you spend too much time updating the 50 places your login form selectors are reflected in the suite and the test suite acts as scar tissue that prevents changes instead of a protective but flexible skin and skeleton that helps the program adapt. Over-using story tests and dealing with the maintenance burden / poor abstractions, overly-repetitive integration suites that make changes hard and inappropriate use of unit testing that makes refactoring difficult are all really common traps when people don't get the big picture of test automation, but they are entirely avoidable. A test suite is like any other piece of software and has to be designed (and factored) for the reality that its behavior will change.
- Ace17 9y ago“The act of describing a program in unambiguous detail and the act of programming are one and the same.” - Kevlin Henney While I mostly agree with this quote, please note that it doesn't mean our formal programming languages can't be improved to feel more "natural" (actually, we've been doing this for decades).
- Ace17 9y agoAnswering to the main point of the author: Architectural intent can be expressed in a formal way, but it requires a formal language that allows you to define new abstractions. And designing such a language is way harder than defining a small DSL with just enough features to formally express your specification.
- adreid 9y agoYes, that is a large part of what I was saying. Also, some things are so hard to specify formally that we still don't know have any kind of formal spec. Memory concurrency semantics is an example. It is only in the last couple of years that we got a good spec of fixed size memory accesses. Then Peter Sewell's group drops the bombshell that if you have mixed size memory accesses then you can't make programs sequentially consistent even if you add a memory barrier after every single memory access. But we still don't know how to formally specify the memory orderings associated with atomic accesses, instruction fetches, page table walks or device accesses. So, until then, we use the best natural language definition we can and hope we will be able to formalise it soon. Also, there are parts of the natural language spec that I had not seen any value in... until I started worrying about whether the spec itself was correct or could possibly be shown to be correct. And now that I do worry about that, I am seeing new value in those parts.