6 ms·
Launch HN: Prolific (YC S19) – Quickly find high-quality survey participants
Hey HN,
We’re Katia and Phelim, cofounders of Prolific (https://www.prolific.co https://www.prolific.co). We help psychological and behavioral researchers quickly find participants they can trust.
We built Prolific because Katia had a hard time finding participants for her psychology studies during her PhD. She briefly used Amazon's Mechanical Turk (MTurk), but didn’t like the user experience and couldn’t get the data she wanted (UK participants). The fundamental problem we’re hoping to help with is better access to psychological and behavioral data. This is challenging in many ways: You have to balance the growth of a multi-sided platform, achieve high data quality, align incentives for all stakeholders (researchers, participants, ourselves, society), diversify the participant pool, to name some. We’re first-time founders and we’ve been bootstrapping our startup for the past 5 years during our PhDs.
Researchers build their surveys using Google Forms, Qualtrics, SurveyMonkey, Typeform, or another tool; all you need is a survey URL to get started. We verify and monitor participants so you can get data fast (most surveys are completed in <2 hours). Studies range from one-to-one interviews to surveys of thousands of people and you can retarget participants anonymously for follow up studies. You only pay for data you approve. Our business model is to charge a % service charge (typically around 25-35%) on top of rewards researchers pay the participants.
We have 70,000+ survey takers in Europe and North America (for distributions of demographic variables see https://www.prolific.co/demographics https://www.prolific.co/demographics) and 100s of demographic filters (try our audience checker via https://app.prolific.co/audience-checker https://app.prolific.co/audience-checker). This means we can find many target demographics for you. For example, you can filter for Democrats vs. Republicans, old vs. young people, students vs. professionals, different ethnicities, people with health problems, Brexit voters, and even collect nationally representative samples!
Anyone can sign up as a participant and start earning a little extra cash.
It's possible to do research using existing platforms like MTurk. Actually, over 50% of behavioral research is now run online, mostly on MTurk. But there are problems with the quality of data you get from existing platforms, and worse, problems with how the people who participate get treated [1]. Our approach addresses these issues. We think the key differences are:
It’s data you can trust: We mandate a minimum hourly reward of $6.50, and often rewards are even higher than that. As a result, participants feel respected and treated like valuable contributors, providing high quality data. We comply with data protection regulation and have a range of technical and behavioral checks in place to ensure high quality data [2].
Demographic prescreening is flexible and free: You can easily invite participants for follow-up studies at no extra cost. You can get niche or even nationally representative samples on-demand.
Prolific is built by researchers for researchers. We try to distribute studies as evenly as possible across our participant pool through rate limiting, so have less of a problem with “professional survey takers” than MTurk.
Our bigger vision is to build tech infrastructure that empowers behavioral research on the internet. The market opportunity is significant because any individuals, businesses, and governments would benefit from better access to rigorous behavioral data when making decisions. For example, what could we do to best curb climate change? What’s the best way to change unhealthy habits? How can we reduce hate crime and political polarization? The stakes are high, and behavioral research can help us find better answers to these kinds of questions.
Moreover, although we built Prolific primarily to help academics, we've noticed that businesses have been using the platform for things like market research and idea validation. This is a new market for us that we're excited to explore. We’d love to hear about any ideas, experiences, and feedback you might have. Thank you!
[1] https://news.ycombinator.com/item?id=19719197 https://news.ycombinator.com/item?id=19719197
[2] https://blog.prolific.co/bots-and-data-quality-on-crowdsourcing-platforms/ https://blog.prolific.co/bots-and-data-quality-on-crowdsourc...
- rococode 7y agoThis is definitely a space that needs innovation! How do you plan to handle the case of survey takers being real people but just skimming through the survey? In my experience running research studies, this was the main problem with MTurk. Things like bots and blatantly junk answers were relatively rare and easy enough to detect and filter. What was much harder to deal with was the fairly high volume of users who just want to finish the survey as fast as possible. It's not so bad for short surveys, but anything over 5 minutes starts to have issues with low quality responses. We had to introduce several control questions to check for consistency in answers and measure time to find outliers. But, it was not an ideal setup and there were still many survey takers we suspected were not paying much attention. The breakdown for our studies was something like 60% good quality answers, 35% low quality answers that are hard to distinguish from high quality, and 5% total junk answers. We ended up doing an in-person study where we got much cleaner results, presumably because people pay better attention when they feel someone is watching them. Wages don't seem to do much for this. We were paying a relatively generous $4 per HIT, estimating 30 minutes per HIT when in reality in reality the average time was under 20. So I'm wondering, are you able to share with us how exactly the "unusual data patterns" and "technical and behavioral checks" can help ensure quality in an ecosystem where users are generally motivated to 1) finish surveys as fast as possible, and 2) appear as if they are giving high quality responses when they are not?
- psb31 7y agoIt’s a great question and a tough problem. We have written a little bit about the problem of ‘slackers’ (participants with low attentiveness and engagement) [1,2]. The things we’re doing right now to address this problem is to 1) test for attentiveness and engagement before participants take part in real studies, 2) distribute surveys more broadly to reduce the % of “professional survey takers” in each study and 3) educate researchers about ways to use good attention and engagement tests [3] in their studies. We can then feed this data back into our system so we can iteratively improve on data quality. I can’t go into too much detail about the attention and engagement checks we have built in, but if you sign up as a participant and pay attention you might spot some. In the long run, we think good feedback systems and high trust will be key so that our pool iteratively gets better and participants don’t feel as incentivized to cheat. It’s key to make sure that participants feel that their high effort responses get fairly rewarded (both financially and with social appreciation). [1] https://blog.prolific.co/how-to-improve-your-data-quality/ https://blog.prolific.co/how-to-improve-your-data-quality/ [2] https://blog.prolific.co/bots-and-data-quality-on-crowdsourcing-platforms/ https://blog.prolific.co/bots-and-data-quality-on-crowdsourc... [3] https://researcher-help.prolific.co/hc/en-gb/categories/360000850653-Prolific-s-Best-Practice-Guide https://researcher-help.prolific.co/hc/en-gb/categories/3600...