3 ms·

Hi congrats. LLMs model a static distribution, whereas consumer preferences change over time to the point that companies regularly run the same survey at diffe

Hi congrats. LLMs model a static distribution, whereas consumer preferences change over time to the point that companies regularly run the same survey at different points in time. At my old fund we would run the same surveys every month to track changes on various companies. How do you counteract this time effect? Presumably a lot of your training data is from the past. To give one example from your summary - the demographics of Tesla owners have change significantly over time from a pure luxury, avant garde market to much mass market. So info about Tesla from 5 years ago is not that useful

timshell 3y ago
Pasting below answer to niko001 The data we trained on has year, so we can specify the year you ask the question (the default is 2023). You can also see how answers change over time. [1] shows how the distribution for "Do you support the President" changes from 2000 to 2023 (see the 9/11 spike, end of Bush era, Obama era, Trump era, etc.) [1] https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948 https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...
- tempusalaria 3y ago
  Which is logical and kind of what I expected. But raises the obvious question of where does your data come from going forward? The internet is getting more and more polluted with machine generated data, previous big ongoing data sources like Twitter, Reddit, etc. are all full of GPT spam and are trying to monetise their data. I’d also be interested in how much you think your platform is just capturing say reported surveys/data. President polling is something that must be all over LLM datasets- isn’t that just replicating the training data? I think you could do a better job of showing on your website the following - here are some unusual survey results we generated from the model - I.e. stuff definitely not in the training data - and here’s the data we actually got when we did that survey for real
  - timshell 3y ago
    Going forward, the current business model (with caveat that pivots are always likely this early stage) is to train on companies' proprietary survey data so we can estimate how their specific users respond to questions. In the backend, we check to see if the answers are stated in a high-quality survey and just retrieve that. I know we do this for gender, and I'm not sure whether that happens for presidential polling. Great idea, thank you. We're still figuring out whether the business model will be a general-purpose tool that anyone can use or those custom models I referenced above. If the former, your suggestion is spot on.
    - another-dave 3y ago
      > Going forward, the current business model (with caveat that pivots are always likely this early stage) is to train on companies' proprietary survey data so we can estimate how their specific users respond to questions. I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you. Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"?
      - timshell 3y ago
        > I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you. We're in the process of figuring that out. Hopefully that is another use case for LLMs :) > Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"? The latter, i.e. base model + "bring your own research"

timshell 3y ago

Pasting below answer to niko001 The data we trained on has year, so we can specify the year you ask the question (the default is 2023). You can also see how answers change over time. [1] shows how the distribution for "Do you support the President" changes from 2000 to 2023 (see the 9/11 spike, end of Bush era, Obama era, Trump era, etc.) [1] https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948 https://roundtable.ai/sandbox/2dd4e9d32c24e9abff01810695e948...

tempusalaria 3y ago
Which is logical and kind of what I expected. But raises the obvious question of where does your data come from going forward? The internet is getting more and more polluted with machine generated data, previous big ongoing data sources like Twitter, Reddit, etc. are all full of GPT spam and are trying to monetise their data. I’d also be interested in how much you think your platform is just capturing say reported surveys/data. President polling is something that must be all over LLM datasets- isn’t that just replicating the training data? I think you could do a better job of showing on your website the following - here are some unusual survey results we generated from the model - I.e. stuff definitely not in the training data - and here’s the data we actually got when we did that survey for real
- timshell 3y ago
  Going forward, the current business model (with caveat that pivots are always likely this early stage) is to train on companies' proprietary survey data so we can estimate how their specific users respond to questions. In the backend, we check to see if the answers are stated in a high-quality survey and just retrieve that. I know we do this for gender, and I'm not sure whether that happens for presidential polling. Great idea, thank you. We're still figuring out whether the business model will be a general-purpose tool that anyone can use or those custom models I referenced above. If the former, your suggestion is spot on.
  - another-dave 3y ago
    > Going forward, the current business model (with caveat that pivots are always likely this early stage) is to train on companies' proprietary survey data so we can estimate how their specific users respond to questions. I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you. Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"?
    - timshell 3y ago
      > I imagine cleaning customer data to get it to the point that it's inputtable will be a big job for you. We're in the process of figuring that out. Hopefully that is another use case for LLMs :) > Are you then creating individual models per customer? As in, if Coke are an existing customer of yours and Pepsi sign up, do they get access to a model that's partially trained on Coke data, or it's a case of your base model + "bring your own research"? The latter, i.e. base model + "bring your own research"