5 ms·

oh hey - it's my paper! If anyone is interested in exploring these ideas, feel free to get in touch (@johnjhorton, https://www.john-joseph-horton.com/ https://w

oh hey - it's my paper! If anyone is interested in exploring these ideas, feel free to get in touch (@johnjhorton, https://www.john-joseph-horton.com/ https://www.john-joseph-horton.com/). FWIW - I think it would be really neat to build a Python library w/ some tools for constructing & running experiments of different kinds. I think the paper only scratches surface of what's possible (esp. once GPT4-ish has an API).

anigbrowl 4y ago
You might like to include this experiment in future research: https://www.science.org/doi/10.1126/sciadv.1600451 https://www.science.org/doi/10.1126/sciadv.1600451
- john_horton 4y ago
  that's a great suggestion - thanks!
doctoboggan 4y ago
This is really cool. I had a similar thought that GPT3 could be used to simulate political polling. A few weeks ago I tried telling GPT3 that it was part of a specific demographic (age, gender, race, income, political leaning, etc) and then asked it how it would respond to certain political questions (I tried gun control, immigration, abortion and some other issues). GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be. My thinking was that this could be used as a quick polling test to see how the real population may respond to certain new ideas. More work would need to be done to calibrate it, as without specific demographic details the answers tended to be liberal leaning. But its an interesting idea which could be used to create instant focus tests on any number of topics.
- capitol_ 4y ago
  > GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be. Doesn't this just mean that your own preconceptions about those demographics matches the language models preconceptions? How would we know that is matches reality when presented with novel ideas/concepts that we want to get feedback on?
  - doctoboggan 4y ago
    That's part of what I mean by it needing to be calibrated. Initially some polling could be done with real people and the GPT agents. Whatever calibration factors are needed to make those two line up could then be used when asking the GPT agents novel questions.
- visarga 4y ago
  > I had a similar thought that GPT3 could be used to simulate political polling. There's a good paper for that task. See my other post: https://news.ycombinator.com/item?id=34385489 https://news.ycombinator.com/item?id=34385489

anigbrowl 4y ago

You might like to include this experiment in future research: https://www.science.org/doi/10.1126/sciadv.1600451 https://www.science.org/doi/10.1126/sciadv.1600451

john_horton 4y ago
that's a great suggestion - thanks!

doctoboggan 4y ago

This is really cool. I had a similar thought that GPT3 could be used to simulate political polling. A few weeks ago I tried telling GPT3 that it was part of a specific demographic (age, gender, race, income, political leaning, etc) and then asked it how it would respond to certain political questions (I tried gun control, immigration, abortion and some other issues). GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be. My thinking was that this could be used as a quick polling test to see how the real population may respond to certain new ideas. More work would need to be done to calibrate it, as without specific demographic details the answers tended to be liberal leaning. But its an interesting idea which could be used to create instant focus tests on any number of topics.

capitol_ 4y ago
> GPT3 was able to change its answers in believable ways depending on what demographic I instructed it to be. Doesn't this just mean that your own preconceptions about those demographics matches the language models preconceptions? How would we know that is matches reality when presented with novel ideas/concepts that we want to get feedback on?
- doctoboggan 4y ago
  That's part of what I mean by it needing to be calibrated. Initially some polling could be done with real people and the GPT agents. Whatever calibration factors are needed to make those two line up could then be used when asking the GPT agents novel questions.
visarga 4y ago
> I had a similar thought that GPT3 could be used to simulate political polling. There's a good paper for that task. See my other post: https://news.ycombinator.com/item?id=34385489 https://news.ycombinator.com/item?id=34385489