3 ms·

I see the logic here, but I’m highly skeptical about how valid such a tool would be. If a researcher comes out and says, “Surveys show that people want X, and

I see the logic here, but I’m highly skeptical about how valid such a tool would be. If a researcher comes out and says, “Surveys show that people want X, and they do not like Y,” and then others ask the researcher if they surveyed people, the answer would be “no.” Fundamentally, people wanting feedback from humans will not get that by using your product. The best you can say is this: “Our product is guessing people will say X.”

timshell 3y ago
We're trying to figure out the optimal use case for this, i.e. whether it's internal or client-facing (your example). Internal purposes include stuff like optimally rewording questions and getting priors. A hybrid approach would be something like - hey let's not ask someone 100 questions because we can accurately predict 80%. Let's just ask them the hard-to-estimate 20 questions
- tcgv 3y ago
  I think it's less about "prediction" and more about mapped cohort behaviors and opinions, especially those that change slowly over time. The LLM model will likely be a picture of how the population and each demographic group behaved and what they believed at a specific time window (i.e. when the data set was collected), and will produce answers that reflect that. It will most likely be lagging behind new trends and how they shape population behaviors and beliefs over time. In any case I think even the most experienced market research professionals would agree that discovering new trends before they become mainstream is really challenging.
- quadrature 3y ago
  > optimally rewording questions This kind of concerns me because you could use this to bias surveys in different directions. This obviously already happens, so maybe it just part of the status quo.
  - ddq 3y ago
    There’s lies, damned lies, statistics, and now fake statistics!
digitcatphd 3y ago
I have been using AI generated surveys using the playground and have found them quite effective in simulating responses. In fact they are incredibly similar to my experience asking the same questions IRL. The challenge is people don’t trust them and AI still have this negative association. So yes I mean to say it’s yet another human error.
famouswaffles 3y ago
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543 https://arxiv.org/abs/2301.07543) Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899 https://arxiv.org/abs/2209.06899) There's been some research in this vain. To answer your question, seemingly very valid.
- puppy_nap 3y ago
  These papers suggest that LLMs do something a lot more specific (when asked to simulate a certain political background, they're able to give responses to questions in a way that's consistent with those political backgrounds). That's not particularly surprising to me as I would expect a human to be also able to simulate this kind of thing pretty accurately. I don't think it implies that LLMs would be good at answering typical business survey questions.
tchock23 3y ago
I’ve worked in this industry for a while and in the ‘faster, cheaper, better - pick two’ trade off, many will select faster and cheaper. That’s only speaking for corporate market research though, can’t say the same for academic researchers. I suspect people would use this product as a quick gut check to decide if it is warranted to spend the time and money on a full scale quant study.
- DriverDaily 3y ago
  You want a 90/10: 90% of the benefit, 10% of the effort. This is like a 10/10.
  - hakuseki 3y ago
    ISTM that's just an a priori feeling. The value or lack thereof of the product totally depends on how accurately it predicts human survey responses, which you can't know without looking at the data.
Shrezzing 3y ago
The tool would be useful as a QA step to test for leading questions in survey design. See Yes Minister's[1] explanation for how they can work. A simulation to see if the questions get the same response irrespective of the order they were asked in could improve survey quality. Obviously, the tool could be used in the opposite way too, to help design surveys that say exactly what the company/govt/charity wants it to. [1] https://www.youtube.com/watch?v=G0ZZJXw4MTA https://www.youtube.com/watch?v=G0ZZJXw4MTA
helsinkiandrew 3y ago
> I see the logic here, but I’m highly skeptical about how valid such a tool would be. I see the problem as although you can create lots of examples that are correct/follow real world opinions, you can never prove that a particular question is correct/follows real world opinion. I'm not sure who would trust the output enough to rely on it for decision making.

timshell 3y ago

We're trying to figure out the optimal use case for this, i.e. whether it's internal or client-facing (your example). Internal purposes include stuff like optimally rewording questions and getting priors. A hybrid approach would be something like - hey let's not ask someone 100 questions because we can accurately predict 80%. Let's just ask them the hard-to-estimate 20 questions

tcgv 3y ago
I think it's less about "prediction" and more about mapped cohort behaviors and opinions, especially those that change slowly over time. The LLM model will likely be a picture of how the population and each demographic group behaved and what they believed at a specific time window (i.e. when the data set was collected), and will produce answers that reflect that. It will most likely be lagging behind new trends and how they shape population behaviors and beliefs over time. In any case I think even the most experienced market research professionals would agree that discovering new trends before they become mainstream is really challenging.
quadrature 3y ago
> optimally rewording questions This kind of concerns me because you could use this to bias surveys in different directions. This obviously already happens, so maybe it just part of the status quo.
- ddq 3y ago
  There’s lies, damned lies, statistics, and now fake statistics!

digitcatphd 3y ago

I have been using AI generated surveys using the playground and have found them quite effective in simulating responses. In fact they are incredibly similar to my experience asking the same questions IRL. The challenge is people don’t trust them and AI still have this negative association. So yes I mean to say it’s yet another human error.

famouswaffles 3y ago

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (https://arxiv.org/abs/2301.07543 https://arxiv.org/abs/2301.07543) Out of One, Many: Using Language Models to Simulate Human Samples (https://arxiv.org/abs/2209.06899 https://arxiv.org/abs/2209.06899) There's been some research in this vain. To answer your question, seemingly very valid.

puppy_nap 3y ago
These papers suggest that LLMs do something a lot more specific (when asked to simulate a certain political background, they're able to give responses to questions in a way that's consistent with those political backgrounds). That's not particularly surprising to me as I would expect a human to be also able to simulate this kind of thing pretty accurately. I don't think it implies that LLMs would be good at answering typical business survey questions.

tchock23 3y ago

I’ve worked in this industry for a while and in the ‘faster, cheaper, better - pick two’ trade off, many will select faster and cheaper. That’s only speaking for corporate market research though, can’t say the same for academic researchers. I suspect people would use this product as a quick gut check to decide if it is warranted to spend the time and money on a full scale quant study.

DriverDaily 3y ago
You want a 90/10: 90% of the benefit, 10% of the effort. This is like a 10/10.
- hakuseki 3y ago
  ISTM that's just an a priori feeling. The value or lack thereof of the product totally depends on how accurately it predicts human survey responses, which you can't know without looking at the data.

Shrezzing 3y ago

The tool would be useful as a QA step to test for leading questions in survey design. See Yes Minister's[1] explanation for how they can work. A simulation to see if the questions get the same response irrespective of the order they were asked in could improve survey quality. Obviously, the tool could be used in the opposite way too, to help design surveys that say exactly what the company/govt/charity wants it to. [1] https://www.youtube.com/watch?v=G0ZZJXw4MTA https://www.youtube.com/watch?v=G0ZZJXw4MTA

helsinkiandrew 3y ago

> I see the logic here, but I’m highly skeptical about how valid such a tool would be. I see the problem as although you can create lots of examples that are correct/follow real world opinions, you can never prove that a particular question is correct/follows real world opinion. I'm not sure who would trust the output enough to rely on it for decision making.