4 ms·
Two sides of software testing: Checking and exploring (2011)
- zubairq 3y agoExploratory testing has been a huge way for me to build user friendly products. But I guess when you work in a large project that you are not paid to do exploratory testing, therefore product usability can end up terrible with end users hating ever using the thing!
- mrkeen 3y ago> "If they can't tell me what the software is supposed to do, I can't test it," Francine, the test manager, scowled. "So, I tell them that I won't start testing until they produce a detailed requirements document." I think this attitude could only work if you expected end-users to do the same: no user interactions except things explicitly written in the manual.
- CoastalCoder 3y agoThis is not the topic I expected from the title :)
- frogulis 3y agoReally? It was exactly what I expected. What did you expect?
- CoastalCoder 3y ago> Really? It was exactly what I expected. Shoot, somehow I actually commented on the wrong article. That's a first. TL;DR: There was an article about "architecture", but it was about building architecture, not computer architecture. That's what I was trying to comment about originally.
- rqtwteye 3y agoExploratory testing by experienced testers is extremely valuable. I am a dev but have done some testing. I could break the systems in various ways because I had a pretty good feel for what can go wrong. Beginner users are also very good because they do stuff you never expected.
- eru 3y ago> Beginner users are also very good because they do stuff you never expected. People using software for a while undergo operant conditioning: consciously or subconsciously they quickly learn to internalise what makes the software 'angry', and learn to avoid that. That makes them more productive users, but also less useful as explorers. See https://blog.regehr.org/archives/861 https://blog.regehr.org/archives/861 > Have you ever used a new program or system and found it to be obnoxiously buggy, but then after a while you didn’t notice the bugs anymore? If so, then congratulations: you have been trained by the computer to avoid some of its problems. For example, I used to have a laptop that would lock up to the point where the battery needed to be removed when I scrolled down a web page for too long (I’m guessing the video driver’s logic for handling a full command queue was defective). Messing with the driver version did not solve the problem and I soon learned to take little breaks when scrolling down a long web page. To this day I occasionally feel a twinge of guilt or fear when rapidly scrolling a web page. > The extent to which we technical people have become conditioned by computers became apparent to me when one of my kids, probably three years old at the time, sat down at a Windows machine and within minutes rendered the GUI unresponsive. Even after watching which keys he pressed, I was unable to reproduce this behavior, at least partially because decades of training in how to use a computer have made it very hard for me to use one in such an inappropriate fashion. By now, this child (at 8 years old) has been brought into the fold: like millions of other people he can use a Windows machine for hours at a time without killing it.
- julian_t 3y agoI think this is true whenever we learn to interact with a system. When I was learning the saxophone I had real trouble getting one note to sound properly, and it seemed to me that there was a leak on that pad. My teacher couldn't get my instrument to misbehave at all. He said there probably was something slightly wrong with that pad, but over the years he'd learned to play 'with' an instrument, adjusting for its particular behavior. The same goes for things like driving a temperamental car.
- pfdietz 3y agoThe article presents two forms of testing, but it implies checking is only possible with a complete specification of the program. But that's not true: one can check incomplete specifications, both positive and negative. To the extent one has any specification, one can automate testing, and in doing so execute far more test cases than any manual exploration ever could. To a significant extent, testing is about sheer volume of tests, not cleverness in constructing the tests.
- tveyben 3y ago> To a significant extent, testing is about sheer volume of tests Well, quantity != quality Just because you have 1E10 test cases does not mean you have a large coverage… One cleverly constructed testcase might return more insights regarding the product than 100 testcases made just to increase the number of test cases. Goodhart's law: "When a measure becomes a target, it ceases to be a good measure" comes to mind…
- pfdietz 3y agoQuantity has a quality all its own. Time and again, testing complex programs, I've seen this approach find bugs one could not have conceived when creating manual tests. You mention coverage. Interestingly, if you read the Csmith paper on random testing of C compilers, they found that while it found plenty of bugs, it didn't increase coverage (over previous tests) more than a tiny amount. Coverage is a great example of Goodhart's law.
- drewcoo 3y ago> it implies checking is only possible with a complete specification of the program No, it doesn't. If exploratory testing has any value at all, that implies that there cannot be complete test coverage with checks, thus there cannot be checks matching a complete specification. > incomplete specifications, both positive and negative There's more to testing than positive and negative test cases. > execute far more test cases than any manual exploration ever could Exploratory testing does not "execute test cases," so you have a tautology here. > testing is about sheer volume of tests The purpose of testing is open to wildly contradictory opinions. I usually claim that it's to find user-impacting bugs before users do. Maybe that can happen with lots of checks and maybe those can be automated. Maybe you're dealing with a system with lots of hidden interdependencies and some "I'm not gonna touch that" dev code and even if you want to check everything you just can't. Maybe the user experience matters more and that goes far beyond positive/negative test cases. There's no simple one-size-fits-all solution.
- satisfice 3y agoAccording to the guys who introduced the terminology of "checking" as a term of art in 2010 (Michael Bolton and me), checking is not a "form" of testing. It's not testing in and of itself. But it is a part of testing. We introduced the terminology because we think it's important for the craft of testing that we reserve the word "testing" for an essentially human process. Testing is what people do. Testing is evaluating a product by learning about it via exploring, experiencing, and experimenting. Checking is a mechanical process of operating a product and evaluating its output algorithmically. A tester frequently performs a check. And all automation that is called a "test" is actually a check by this definition. All testing is exploratory to some degree. Much of testing involves performing a check at some point. This way of thinking leads to a richer and deeper craft of testing than the alternative popular notion. See: https://www.satisfice.com/blog/archives/1509 https://www.satisfice.com/blog/archives/1509 https://www.satisfice.com/blog/archives/856 https://www.satisfice.com/blog/archives/856
- tveyben 3y agoI’m glad to see you here James, I was just about to make a comment linking to you’re site ;-) /A happy exploratory tester
- CipherPilot 3y agoWhile I understand the motivation for disambiguating "checking" from "testing" conceptually, I have some concerns about how this framework may play out in practice. What constitutes a "mechanical" check vs a human "exploration", for example, could likely be a source of endless debate. And is it truly useful to separate the two when most testing undoubtedly involves elements of both? A more pragmatic definition may be preferable to one that sets up a dubious distinction prone to subjective interpretation. Nonetheless, I appreciate the attempt to enrich discussion around strategic testing methodologies at a time when clear terminology is needed to match the complexity of modern software systems.
- satisfice 3y agoI can explain the differences. I've been using these distinctions in practice for many years. "Mechanical" means algorithmic. It's what Turing meant when he wrote "The idea behind digital computers may be explained by saying that these machines are intended to carry out any operations which could be done by a human computer. The human computer is supposed to be following fixed rules; he has no authority to deviate from them in any detail. We may suppose that these rules are supplied in a book, which is altered whenever he is put on to a new job." See: https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf Checking is an evaluation activity that can be COMPLETELY automated. Testing is a social activity done by humans. It cannot be automated because it incorporates social judgement. Social judgement requires collective tacit social knowledge. See: The Shape of Actions, by Harry Collins for a very deep dive into that. A human exploration is distinguished by the presence of choosing. Humans make choices. You could claim that human choices are an epiphenomenon of bio-mechanics, if you want to. I actually believe that, myself. However, we are speaking of the social/practical realm. In this realm we perceive each other and treat each other as if we are making free choices (although perhaps influenced by known sources of bias). Thus the difference between exploring and checking is that, in every moment, a check is controlled by an explicit script that is knowable in principle; whereas in exploration, a person is forming new thoughts and theories, through a process of real-time sensemaking (otherwise known as learning). Checking is a process of operating a product and verifying a particular fact. A check does not change the checker. Exploration is a process of interacting, experiencing, and being transformed by this process. Exploration grows the explorer.