Show HN: Roundtable – Estimating survey results in seconds

Recent academic work ([1], [2]) has suggested that LLMs can effectively simulate different Internet subpopulations. For example, you may ask ChatGPT to emulate being a high school teacher explaining Newton’s laws of physics. Building upon this, we created Roundtable, a platform that uses LLMs to predict how people will respond to any arbitrary survey question. To do so, we needed to first reduce bias arising from GPT’s training procedure. Because these models are primarily trained on Internet data, they can be heavily skewed towards the demographics of heavy Internet users (e.g., high-income, male). We addressed this by fine-tuning GPT on the GSS (General Social Survey) to ‘de-bias’ the model into emulating a more representative U.S. population. We allow users to ask any multiple-choice question and add conditioning questions and/or descriptions of their target population. Here are some examples: Simulation 1 (General Interest) Are you interested in buying an e-bike? Yes 28%, No 72% ([3]) Are you interested in buying an e-bike? conditioned on "Yes" to "Do you own a Tesla car?" Yes 40%, No 60% ([4]) Simulation 2 (reproducing the Stack Overflow Developer Survey; [5]) Where did you learn to code? conditioned on "Yes" to "Are you 45 years or older?" Books 55%, Online 45% ([6]) Where did you learn to code? conditioned on "No" to "Are you 45 years or older?" Books 26%, Online 74% ([7]) Simulation 3 (USA vs. Stack Overflow Developers vs. Hacker News Users) Do you code? Yes 24%, No 76% ([8]; USA) Do you code? Yes >99%, No 0% ([9]; Stack Overflow Developers) Do you code? Yes 83%, No 17% ([10]; Hacker News Users) — Of course, a natural question is whether we can trust these results. If you click ‘Investigate Results’, we report the most similar (in terms of cosine distance between LLM embeddings) GSS questions as a way of estimating how much extrapolation / interpolation is going on. This doesn’t quite address the accuracy of the subpopulations / conditioning questions (we are working on this), but we thought we are at a sufficiently advanced point to share what we’ve built with you all. Feedback would be greatly appreciated. ---- [1] https://arxiv.org/pdf/2209.06899.pdf https://arxiv.org/pdf/2209.06899.pdf [2] https://openreview.net/pdf?id=eYlLlvzngu https://openreview.net/pdf?id=eYlLlvzngu [3] https://roundtable.ai/sandbox/e02e92a9ad20fdd517182788f4ae7e1f96a849c0 https://roundtable.ai/sandbox/e02e92a9ad20fdd517182788f4ae7e... [4] https://roundtable.ai/sandbox/6b4bf8740ad1945b08c0bf584c84c1202a5fec53 https://roundtable.ai/sandbox/6b4bf8740ad1945b08c0bf584c84c1... [5] https://survey.stackoverflow.co/2023/ https://survey.stackoverflow.co/2023/ [6] https://roundtable.ai/sandbox/d701556248385d05ce5d26ce7fc776bb4d32fad0 https://roundtable.ai/sandbox/d701556248385d05ce5d26ce7fc776... [7] https://roundtable.ai/sandbox/8bd80babad042cf60d500ca28c40f7db413f553a https://roundtable.ai/sandbox/8bd80babad042cf60d500ca28c40f7... [8] https://roundtable.ai/sandbox/4a9d2fd6025459bd73b7798a8b2fdc5640ca8c35 https://roundtable.ai/sandbox/4a9d2fd6025459bd73b7798a8b2fdc... [9] https://roundtable.ai/sandbox/7e41ed16c01de48247bce02700c39893463abb88 https://roundtable.ai/sandbox/7e41ed16c01de48247bce02700c398... [10] https://roundtable.ai/sandbox/13aaa142e87337201601fb4b76d125d2180a1dda https://roundtable.ai/sandbox/13aaa142e87337201601fb4b76d125...