3 ms·

Out of curiosity, how would you propose taking a random sample of all possible questions?

• pkrs 12y ago
  I don't think it's possible, people on different platforms issue different queries partially due to marketing and insiticts on what works and what doesn't. Also commands that already sort of work lead to more queries issues lead to better quality.
• sytelus 12y ago
  If you want to do this kind of test with more discipline, you would give out random phones to randomly selected N people who have never used any of these services before. Then log each query they do for a week or two. Afterwards you can run same queries through all 3 services, generate results and do human judgement on mechanical turk about which one is the best. The science of measuring these stuff is complex and I've omitted many complexities here, for example, you want to do multiple judgement for same query, you need someway to measure judgement quality itself, you need great judgement guidelines that covers edge cases etc. Usually in organizations that work with big data, you will find competent measurement team building tools for these kind of measurements with years of investment.