24 ms·

One hack we've employed with our customers in interviews and through surveys: have them rank/prioritize our current features, and consider/rank a set of potenti

One hack we've employed with our customers in interviews and through surveys: have them rank/prioritize our current features, and consider/rank a set of potential new features. It's much easier for someone to order a list (reaction), than to provide an explanation (creation). Not a substitute for a full interview, but a quick way to determine what your customer values.

longerthoughts 8y ago
This is really interesting. Have results tended to be relatively uniform across customers or do you end up having to break a lot of hearts when you ultimately prioritize features on your roadmap? Do you have them rank UX-driven changes (i.e. reframing existing features) in addition to entirely new features?
revazquez 8y ago
Very interesting. What tool/s do you use to gather this feedback and get the aggregated view across all clients? Do you weigh all clients the same?
- rafd 8y ago
  We do these sort of surveys a lot with our team. We found that users had a hard time coming up with the ranked list, but found making A vs B comparisons easy. We slapped together a quick tool that lets you create pairwise surveys and then aggregates their results using the Schulze method (as if it were a preferential election). We've released it for free: https://decidedly.so https://decidedly.so
jfindley 8y ago
How do you control for people's abstract desires not mapping well to what they actually want? Example to illustrate my point: Focus groups over the past 15 years or so have consistently told car mfrs that consumers want larger cars - larger cars are assumed to be more luxurious and associated with financial success, and thus are desirable. The end result is that previous small, manoeuvrable, city cars are now bigger and harder to park. Consumers or small cars are generally unhappy with this trend, as they wanted a small car for the ease of parking etc, but focus groups are frequently insufficiently well tuned to pick up up on this nuance. My experience in this industry over many years is that users often don't have a clear idea of what they want, and in the cases when they do a non-trivial amount of time it's a so tightly tethered to one special-snowflake usecase that virtually noone else finds value in it.
- ASpring 8y ago
  This is why user experience researchers have a job. It's not about asking the user what they want, it's about observing how they work, their problems, their current workarounds, and then building something they need.
- crottypeter 8y ago
  Re the special-snowflake usecase: there was a Simpson's episode about this where Homer helped create a terrible car: http://simpsons.wikia.com/wiki/The_Homer http://simpsons.wikia.com/wiki/The_Homer
- kakwa_ 8y ago
  Just like any 90ies Disney movie: Want vs Need We want fancy stuff, or at least stuff that look fancy/luxurious/desirable. If we were rational, small cars is actually what we need (or no car at all in fact, but that's another matter). However I'm wondering if when it came to buying the actual car, people will tend to follow the "need" or the "want" or something in between.
deytempo 8y ago
Yeah I have also found that asking cusomers to rank or sort features using a question similar to this one: https://www.surveyking.com/help/ranking-question.php https://www.surveyking.com/help/ranking-question.php It’s a very simple but effective way to gauge levels of reaction without losing the interest of the audience.
slavik81 8y ago
An important effect of asking people to rank choices is that it forces people to consider costs. An increase in the rank of one item inherently decreases the rank of another.
wastedhours 8y ago
One thing I've found that works is asking them to write down free-form what they value, and then give them the list you've prepared and then ask them to rank. Unless you're detached from your customer, they should be in similar ballparks, but getting the words they actually use out of them, and then ranking what you had in mind can be powerful for clarifying their thought process (and gives your marketing team the messaging from the horse's mouth).

longerthoughts 8y ago

This is really interesting. Have results tended to be relatively uniform across customers or do you end up having to break a lot of hearts when you ultimately prioritize features on your roadmap? Do you have them rank UX-driven changes (i.e. reframing existing features) in addition to entirely new features?

revazquez 8y ago

Very interesting. What tool/s do you use to gather this feedback and get the aggregated view across all clients? Do you weigh all clients the same?

rafd 8y ago
We do these sort of surveys a lot with our team. We found that users had a hard time coming up with the ranked list, but found making A vs B comparisons easy. We slapped together a quick tool that lets you create pairwise surveys and then aggregates their results using the Schulze method (as if it were a preferential election). We've released it for free: https://decidedly.so https://decidedly.so

jfindley 8y ago

How do you control for people's abstract desires not mapping well to what they actually want? Example to illustrate my point: Focus groups over the past 15 years or so have consistently told car mfrs that consumers want larger cars - larger cars are assumed to be more luxurious and associated with financial success, and thus are desirable. The end result is that previous small, manoeuvrable, city cars are now bigger and harder to park. Consumers or small cars are generally unhappy with this trend, as they wanted a small car for the ease of parking etc, but focus groups are frequently insufficiently well tuned to pick up up on this nuance. My experience in this industry over many years is that users often don't have a clear idea of what they want, and in the cases when they do a non-trivial amount of time it's a so tightly tethered to one special-snowflake usecase that virtually noone else finds value in it.

ASpring 8y ago
This is why user experience researchers have a job. It's not about asking the user what they want, it's about observing how they work, their problems, their current workarounds, and then building something they need.
crottypeter 8y ago
Re the special-snowflake usecase: there was a Simpson's episode about this where Homer helped create a terrible car: http://simpsons.wikia.com/wiki/The_Homer http://simpsons.wikia.com/wiki/The_Homer
kakwa_ 8y ago
Just like any 90ies Disney movie: Want vs Need We want fancy stuff, or at least stuff that look fancy/luxurious/desirable. If we were rational, small cars is actually what we need (or no car at all in fact, but that's another matter). However I'm wondering if when it came to buying the actual car, people will tend to follow the "need" or the "want" or something in between.

deytempo 8y ago

Yeah I have also found that asking cusomers to rank or sort features using a question similar to this one: https://www.surveyking.com/help/ranking-question.php https://www.surveyking.com/help/ranking-question.php It’s a very simple but effective way to gauge levels of reaction without losing the interest of the audience.

slavik81 8y ago

An important effect of asking people to rank choices is that it forces people to consider costs. An increase in the rank of one item inherently decreases the rank of another.

wastedhours 8y ago

One thing I've found that works is asking them to write down free-form what they value, and then give them the list you've prepared and then ask them to rank. Unless you're detached from your customer, they should be in similar ballparks, but getting the words they actually use out of them, and then ranking what you had in mind can be powerful for clarifying their thought process (and gives your marketing team the messaging from the horse's mouth).