6 ms·

Using mTurk to interview 100 customers (in 4 hours)

67 points by craigkerstiens 14y ago

MarkHoroszowski 14y ago
Really great post with 2 super relevant examples to interview and test the theories that make up your business. I'm curious how other people are using mTurk (and other scalable solutions) to help test their theories. Any winning ideas out there?
hansef 14y ago
In my opinion probably a more viable approach for testing consumer-oriented ideas (e.g. "Daycare Review Service") than B2B products (e.g "Lesson Scheduling for Music Teachers"), but some really useful and actionable stuff here.
- MarkHoroszowski 14y ago
  @hansef - this was (is?) definitely my impression. But turns out mTurk users, on average, are more educated than the average population. Many in business roles. We were surprised by some of the responses, and indeed the b2b profiles on some of our profiles. I guess like everything startup, worth testing and challenging assumptions.
- justinwi 14y ago
  By and large, I'd agree with you, but consider that people are still people when they're off work. By that I mean, if the decision maker you're looking for could easily be on Mechanical Turk in their off time. I can promise there are Music Teachers on mTurk, just like there are marketers, hr reps, sales people, real estate agents, etc.
ovi256 14y ago
I wonder how well this would work for B2B apps, in which case you need to interview professionals of some sort. For example, I would love to interview PR professionals. Compared to parents, I would venture a guess that they're much harder to find on MT. I wonder if Facebook ads would work better.
- MarkHoroszowski 14y ago
  At the price point for interviews and with the ability to validate, it's worth a try. Using the guide, it can't take more than 1 hour to set this up... faster and cheaper then getting assets up and ready on FB.
- SoftwareMaven 14y ago
  I'm in a similar boat, but with entrepreneurs instead of PR people. I've been trawling various user groups and will be going to some local meetups, but it isn't highly efficient. Ideas appreciated! (Speaking of which, if you are an entrepreneur or an early employee at a startup (like <= employee 3), I'd love your feedback via a quick survey! http://www.surveymonkey.com/s/F7W9P5P http://www.surveymonkey.com/s/F7W9P5P )
- grueful 14y ago
  LinkedIn and Facebook can both be pretty good when you need narrow targeting. There are three big issues with using MTurk for business research. You can't do much to refine who you get as respondents, the respondents are incentivized to complete the task without regard for the effect on your data, and you have restrictions on what you can do due to the anonymity rules. If you're following the way of the niche, MTurk may be too problematic compared to just slogging through some cold calls. The benefit for more broadly targeted projects is that it lets you grab a large sample size with fairly low effort.
SoftwareMaven 14y ago
This is a great option if your target is likely to be an mTurk user. Further, this is a great write-up on how to do it! If you do this, make sure to validate your psychographics and demographics. You will also want to validate your responses with people you've identified in your psychographics and demographics. In other words, make sure the data you are getting is actually valid data and not coming from some 12 year old playing with his mom's phone. :)
- SoftwareMaven 14y ago
  I decided to do some digging and found some demographics[1] of mTurk users. It's far more diverse than I thought. 1. http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html http://www.behind-the-enemy-lines.com/2010/03/new-demographi...
  - MarkHoroszowski 14y ago
    Really interesting... thanks for sharing! For our surveys, we also captured specific demographic information to get the most representative sample as possible.
deleted 14y ago
[deleted]
nrmehta 14y ago
Really appreciate the post - especially the thought in making the content relevant for the readers (e.g., the video).
wamatt 14y ago
A great entrepreneurial hack and interesting mix of hustling,marketing and tech. (ps - watch the vid)
mmahemoff 14y ago
Anyone suggest a similar tactic for people outside US, given that Google Voice and mTurk are (mostly?) limited to US?
- justinwi 14y ago
  In mTurk, you can select which countries your workers are in and you could use Skype instead of Google Voice.
  - mmahemoff 14y ago
    For workers yes, but I don't believe you can be a mTurk provider if you're outside the US though. Skype would work fine here, true.
jyu 14y ago
This is a pretty well known strategy. Good to see people using this still and getting great results from it. You can get some surprisingly useful feedback for just $X00, especially if you note down your target user group in the HIT.
medinismo 14y ago
from the guy who got into TechStars with Nick - he is one of best hustlers I know
evmar 14y ago
Google Consumer Surveys (http://www.google.com/insights/consumersurveys/home http://www.google.com/insights/consumersurveys/home) achieves a similar end for a surprisingly low price, and they've figured out all the boring details like how to poll a representative sample of the US internet population and how to get stats right. Disclaimer: I work on the product. I think it'd be pretty useful for startups to gauge the market.
- justinwi 14y ago
  Sounds like a really interesting product, and something we'll definitely check out, but to be very clear...interviews != surveys. In terms of customer discovery and problem identification, absolutely nothing beats interviews. Surveys may have their place when it comes to getting demographic information or validating a market size, but interviews should not be replaced with surveys.
  - evmar 14y ago
    Wow, I somehow missed that the original article was about interviews despite it being all over the article. I think I read it earlier in the day but only came back later to comment on it and forgot that important part. :( Thanks for being polite about my error.
bravura 14y ago
What is a useful way to find initial customers for a B2B product? (As other commenters have noted, mTurk is more useful in a consumer setting.) Is there a way to find initial customers in a scalable manner? i.e. scaling it across many different business ideas and potential customers? This would be a great service that a lot of people would pay for. I know, I know, I'm asking for marketing to be commoditized and turned into a service, which is hard. I don't really know how to do it.
- trapexit 14y ago
  Check out Elastic: https://elasticsales.com/what-we-do/ https://elasticsales.com/what-we-do/ (I don't work for them and haven't used them, but it sounds like they're doing exactly the thing you're looking for.)
  - bravura 14y ago
    Is elastic appropriate for customer development and initial marketing? My understanding was that their use-case was if you wanted to scale out sales on a product for which you have already identified need, price, customer profile, etc.
deleted 14y ago
[deleted]
mpunaskar 14y ago
Any idea how to open non-US requester account? Right now amazon will only let you signup - if you have US billing address :(

Really great post with 2 super relevant examples to interview and test the theories that make up your business. I'm curious how other people are using mTurk (and other scalable solutions) to help test their theories. Any winning ideas out there?

hansef 14y ago

In my opinion probably a more viable approach for testing consumer-oriented ideas (e.g. "Daycare Review Service") than B2B products (e.g "Lesson Scheduling for Music Teachers"), but some really useful and actionable stuff here.

MarkHoroszowski 14y ago
@hansef - this was (is?) definitely my impression. But turns out mTurk users, on average, are more educated than the average population. Many in business roles. We were surprised by some of the responses, and indeed the b2b profiles on some of our profiles. I guess like everything startup, worth testing and challenging assumptions.
justinwi 14y ago
By and large, I'd agree with you, but consider that people are still people when they're off work. By that I mean, if the decision maker you're looking for could easily be on Mechanical Turk in their off time. I can promise there are Music Teachers on mTurk, just like there are marketers, hr reps, sales people, real estate agents, etc.

ovi256 14y ago

I wonder how well this would work for B2B apps, in which case you need to interview professionals of some sort. For example, I would love to interview PR professionals. Compared to parents, I would venture a guess that they're much harder to find on MT. I wonder if Facebook ads would work better.

MarkHoroszowski 14y ago
At the price point for interviews and with the ability to validate, it's worth a try. Using the guide, it can't take more than 1 hour to set this up... faster and cheaper then getting assets up and ready on FB.
SoftwareMaven 14y ago
I'm in a similar boat, but with entrepreneurs instead of PR people. I've been trawling various user groups and will be going to some local meetups, but it isn't highly efficient. Ideas appreciated! (Speaking of which, if you are an entrepreneur or an early employee at a startup (like <= employee 3), I'd love your feedback via a quick survey! http://www.surveymonkey.com/s/F7W9P5P http://www.surveymonkey.com/s/F7W9P5P )
grueful 14y ago
LinkedIn and Facebook can both be pretty good when you need narrow targeting. There are three big issues with using MTurk for business research. You can't do much to refine who you get as respondents, the respondents are incentivized to complete the task without regard for the effect on your data, and you have restrictions on what you can do due to the anonymity rules. If you're following the way of the niche, MTurk may be too problematic compared to just slogging through some cold calls. The benefit for more broadly targeted projects is that it lets you grab a large sample size with fairly low effort.

SoftwareMaven 14y ago

This is a great option if your target is likely to be an mTurk user. Further, this is a great write-up on how to do it! If you do this, make sure to validate your psychographics and demographics. You will also want to validate your responses with people you've identified in your psychographics and demographics. In other words, make sure the data you are getting is actually valid data and not coming from some 12 year old playing with his mom's phone. :)

SoftwareMaven 14y ago
I decided to do some digging and found some demographics[1] of mTurk users. It's far more diverse than I thought. 1. http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html http://www.behind-the-enemy-lines.com/2010/03/new-demographi...
- MarkHoroszowski 14y ago
  Really interesting... thanks for sharing! For our surveys, we also captured specific demographic information to get the most representative sample as possible.

deleted 14y ago

[deleted]

nrmehta 14y ago

Really appreciate the post - especially the thought in making the content relevant for the readers (e.g., the video).

wamatt 14y ago

A great entrepreneurial hack and interesting mix of hustling,marketing and tech. (ps - watch the vid)

mmahemoff 14y ago

Anyone suggest a similar tactic for people outside US, given that Google Voice and mTurk are (mostly?) limited to US?

justinwi 14y ago
In mTurk, you can select which countries your workers are in and you could use Skype instead of Google Voice.
- mmahemoff 14y ago
  For workers yes, but I don't believe you can be a mTurk provider if you're outside the US though. Skype would work fine here, true.

jyu 14y ago

This is a pretty well known strategy. Good to see people using this still and getting great results from it. You can get some surprisingly useful feedback for just $X00, especially if you note down your target user group in the HIT.

medinismo 14y ago

from the guy who got into TechStars with Nick - he is one of best hustlers I know

evmar 14y ago

Google Consumer Surveys (http://www.google.com/insights/consumersurveys/home http://www.google.com/insights/consumersurveys/home) achieves a similar end for a surprisingly low price, and they've figured out all the boring details like how to poll a representative sample of the US internet population and how to get stats right. Disclaimer: I work on the product. I think it'd be pretty useful for startups to gauge the market.

justinwi 14y ago
Sounds like a really interesting product, and something we'll definitely check out, but to be very clear...interviews != surveys. In terms of customer discovery and problem identification, absolutely nothing beats interviews. Surveys may have their place when it comes to getting demographic information or validating a market size, but interviews should not be replaced with surveys.
- evmar 14y ago
  Wow, I somehow missed that the original article was about interviews despite it being all over the article. I think I read it earlier in the day but only came back later to comment on it and forgot that important part. :( Thanks for being polite about my error.

bravura 14y ago

What is a useful way to find initial customers for a B2B product? (As other commenters have noted, mTurk is more useful in a consumer setting.) Is there a way to find initial customers in a scalable manner? i.e. scaling it across many different business ideas and potential customers? This would be a great service that a lot of people would pay for. I know, I know, I'm asking for marketing to be commoditized and turned into a service, which is hard. I don't really know how to do it.

trapexit 14y ago
Check out Elastic: https://elasticsales.com/what-we-do/ https://elasticsales.com/what-we-do/ (I don't work for them and haven't used them, but it sounds like they're doing exactly the thing you're looking for.)
- bravura 14y ago
  Is elastic appropriate for customer development and initial marketing? My understanding was that their use-case was if you wanted to scale out sales on a product for which you have already identified need, price, customer profile, etc.

deleted 14y ago

[deleted]

mpunaskar 14y ago

Any idea how to open non-US requester account? Right now amazon will only let you signup - if you have US billing address :(