3 ms·
That's not so much a business case as a contrived example. I absolutely would expect system testing and QA to find such a bug or the testing isn't up to scratch
by Nursie 7y ago
That's not so much a business case as a contrived example. I absolutely would expect system testing and QA to find such a bug or the testing isn't up to scratch.
What you're proposing is that all engineers now effectively learn a new language and translate requirements into it. This expects requirements to be fully fleshed out themselves (a luxury few have these days) and relatively static (or everything needs to be reworked).
Good luck with that...
- zozbot234 7y ago> I absolutely would expect system testing and QA to find such a bug You can't test what you don't have. With design-based formal methods you can "fuzz" the high-level design and spot very real problems in it, before writing a single line of ordinary code.
- Nursie 7y agoThat's not what the article is claiming though, and in fact when the code is written is irrelevant to the business case. We're talking about spotting production issues before they arise. The article makes the very strong claim that QA aren't going to catch the bug that arises from this design omission and it's going to cost your company money as everything goes wrong in the field. I don't believe it would get that far IRL as any dev with a brain is going to spot it, and even so QA should certainly have test cases for this sort of situation if they're doing their job right. You can argue this may be a better approach, and catch potential problems earlier, have at it. You can argue that there are classes of error that this will catch that normally you wouldn't (some of the preamble and larger scale stuff does seem to show this) But the illustrated example doesn't hold up to scrutiny and is overstated. As such I think it undermines the point that's the author is trying to make, because it looks to me like something we'd catch anyway.
- zozbot234 7y ago> That's not what the article is claiming though I'm not sure how I can respond to that, other than pointing you to the Intro section (it's only a single, short paragraph). It seems clear to me that this is what the article is supposed to be about.
- Nursie 7y agoI'm not sure how I can respond to that, given explicitly in the text we have - > The only symptom is credits mysteriously disappearing from people’s accounts. Your QA and monitoring won’t find this. and > It will cause your customers to lose money and stop trusting you. Thread over I guess... :shrug:
- deleted 7y ago[deleted]
- dwb 7y agoThe article is implicitly asking you to make a small imaginative leap – that the bug expressed is in fact subtle enough it would have never been caught otherwise – in the service of providing a concise, digestible example. It might have been better for the author to spell this out but I don't think it's really necessary.
- ones_and_zeros 7y agoI think the argument is if you've been around distributed systems long enough you will encounter race conditions. Sure, it's ok to say "Well, the testing infrastructure isn't up to snuff, so we just need to fix it" but at scale this is impractical. Check out the fallacies of distributed computing[0]. If your testing system can simulate all of those edge cases, it probably looks a lot like TLA+. [0] https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
- Nursie 7y agoI agree, once you've been around parallel and distributed computing for a while you do notice this stuff, which is why I think the example given in the article isn't a particularlyt good one - that's a really noticeable design omission and I would expect it to be caught pretty early in any dev/QA process as the developer implements the functionality and thinks "Hey, what if .. don't we need to invalidate the other offers?" I'm sure there are good cases for using TLA+, I'm sure there are situations where it's not only useful for catching errors before they even happen, but in which this more than offsets the upfront costs of the exercise. I guess I just came away from the article not feeling that such had been demonstrated, in fact I came away with the feeling that the example was contrived to fit the agenda and didn't actually show much.
- ones_and_zeros 7y agoI think it's a great example. It boils down to simple code that may not be obvious how it behaves within a degraded network. Your dev/QA could even have the thought and actually test it but it work under ideal circumstances giving them false confidence. I'll go out on a limb and say it is nearly impossible to design and build a test environment that can simulate all network conditions, so that even in trivial cases where a dev might know for a fact that there is an issue, it'll be incredibly hard to reproduce it. Maybe put another way, formal methods give cover to dev/QA to avoid shipping known but hard to prove buggy code. Bugs they will ultimately be held responsible for.
- zzzcpan 7y ago
- zozbot234 7y ago> But the illustrated example doesn't hold up to scrutiny and is overstated. The illustrated example is intended to be comparatively trivial. It's just clarifying what this kind of verification actually involves in practice. It makes no sense to describe it as "overstated", since the article actually mentions several case studies as demonstrating the usefulness of this approach; it's just that these cannot be examined in detail in an introductory article.
- Nursie 7y agoIt's attempting, AFAICT, to demonstrate TLA+ catching a class of very dangerous problem that traditional approaches won't/can't catch. It fails and in doing so it undermines the business case the wider article attempts to make. I think perhaps the example has just been simplified too far.