3 ms·

Guerilla testing is a great starting point. Do you usualy give one or multiple scenarios to complete? Be aware that you are not controlling the environment and

by daleco 8y ago

Guerilla testing is a great starting point. Do you usualy give one or multiple scenarios to complete? Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)

pests 8y ago
> not controlling the environment and the users may be distracted by outside factors So probably how they will be when actually using your app? I do agree though in a reproduce-ability/data-integrity standpoint.
hluska 8y ago
I'm not OP, but I've used similar styles of pub/coffee shop testing. When I do it, I don't give scenarios or bother measuring things like success rate. It's very freeform, unscripted and raw. It's kind of like hallway testing for people who work alone. And, instead of coworkers, you're working with complete strangers who might not have any idea what's happening. If I had to pick one thing to measure, I'd be more inclined to measure positive versus negative fucks (I'm not joking), but honestly, it's more about the experience and the totally raw, sometimes terrifyingly honest feedback I hear. Numbers aren't useful because the most useful sessions go way off the rails.
zer00eyz 8y ago
> Do you usualy give one or multiple scenarios to complete? I am OP, and it depends. Im old enough that I have done this with paper prototypes, and it is shockingly effective. Now most of the code I write is compartmentalized enough that with a bit of effort I can come up with a one off digital artifact. Just rip out the chunk of your app that you want to test and present ONLY that. Spend a day building ugly shims early enough in the process and you might make some dramatic changes! > Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...) Even with those things going on your going to be shocked at not only what issues come up but how frequently. That thing you did that you thought was simple and sane, no one gets it. Do more than half of your test subjects comment on the same thing (good, bad, both) then you have a problem. The key isn't controlling the key is listening... I also want to state that a slighty drunk, distracted bar patron, getting through your process without quitting, telling you your product is shit, or giving up is a positive sign that in the real world people who are more motivated than "beer" are going to have a fairly positive and expedient experience.

by daleco 8y ago

pests 8y ago

> not controlling the environment and the users may be distracted by outside factors So probably how they will be when actually using your app? I do agree though in a reproduce-ability/data-integrity standpoint.

hluska 8y ago

I'm not OP, but I've used similar styles of pub/coffee shop testing. When I do it, I don't give scenarios or bother measuring things like success rate. It's very freeform, unscripted and raw. It's kind of like hallway testing for people who work alone. And, instead of coworkers, you're working with complete strangers who might not have any idea what's happening. If I had to pick one thing to measure, I'd be more inclined to measure positive versus negative fucks (I'm not joking), but honestly, it's more about the experience and the totally raw, sometimes terrifyingly honest feedback I hear. Numbers aren't useful because the most useful sessions go way off the rails.

zer00eyz 8y ago

> Do you usualy give one or multiple scenarios to complete? I am OP, and it depends. Im old enough that I have done this with paper prototypes, and it is shockingly effective. Now most of the code I write is compartmentalized enough that with a bit of effort I can come up with a one off digital artifact. Just rip out the chunk of your app that you want to test and present ONLY that. Spend a day building ugly shims early enough in the process and you might make some dramatic changes! > Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...) Even with those things going on your going to be shocked at not only what issues come up but how frequently. That thing you did that you thought was simple and sane, no one gets it. Do more than half of your test subjects comment on the same thing (good, bad, both) then you have a problem. The key isn't controlling the key is listening... I also want to state that a slighty drunk, distracted bar patron, getting through your process without quitting, telling you your product is shit, or giving up is a positive sign that in the real world people who are more motivated than "beer" are going to have a fairly positive and expedient experience.