5 ms·

It's far, far more likely that Grok was just trained on data that includes text generated by GPT. This is super common in the open-source/local AI area. Many m

It's far, far more likely that Grok was just trained on data that includes text generated by GPT. This is super common in the open-source/local AI area. Many models are trained on output from GPT. The better models will filter out anything that mentions GPT or OpenAI. Seems like Grok is not one of the better models.

john-radio 3y ago
I know that's X's story, but why do you consider it "far, far more likely" for that to be true?
- simonw 3y ago
  Because the alternative is that Grok are reselling OpenAI API access fraudulently. OpenAI would know that they were doing this. Reselling like this is against their terms of service: https://openai.com/policies/terms-of-use https://openai.com/policies/terms-of-use
  - john-radio 3y ago
    My assumption was that they simply have a contract with openai.
  - dools 3y ago
    Where does it say that goes against their terms of service? I can’t see anything that would preclude Twitter from just plugging in the API and using it directly
- aprilthird2021 3y ago
  Occam's razor probably
- VirusNewbie 3y ago
  I think it's far more likely they're building off some open source foundational models, rather than doing one from scratch OR using GPT under the hood. If it was the latter, don't you think we'd hear about how "Grok is on par with GPT"? But you don't, people think it's not as good (but funnier?)
deleted 3y ago
[deleted]
3cats-in-a-coat 3y ago
Yes, it does put into question the basic competence of its makers. Like how hard it is to massage at least the explicit brand mentions out of the data.
TradingPlaces 3y ago
"Large language models (LLM) have become state of the art in many benchmarks and conversational LLM applications like ChatGPT are now widely used by the public. Those LLMs can be used to generate large amounts of content which is posted on the internet to various platforms. As LLMs are trained on datasets usually collected from the internet, this LLM-generated content might be used to train the next generation of LLMs. Therefore, a self-consuming training loop emerges in which new LLM generations are trained on the output from the previous generations. We empirically study this self-consuming training loop using a novel dataset to analytically and accurately measure quality and diversity of generated outputs. We find that this self-consuming training loop initially improves both quality and diversity. However, after a few generations the output inevitably degenerates in diversity. We find that the rate of degeneration depends on the proportion of real and generated data.” https://arxiv.org/abs/2311.16822 https://arxiv.org/abs/2311.16822

john-radio 3y ago

I know that's X's story, but why do you consider it "far, far more likely" for that to be true?

simonw 3y ago
Because the alternative is that Grok are reselling OpenAI API access fraudulently. OpenAI would know that they were doing this. Reselling like this is against their terms of service: https://openai.com/policies/terms-of-use https://openai.com/policies/terms-of-use
- john-radio 3y ago
  My assumption was that they simply have a contract with openai.
- dools 3y ago
  Where does it say that goes against their terms of service? I can’t see anything that would preclude Twitter from just plugging in the API and using it directly
aprilthird2021 3y ago
Occam's razor probably
VirusNewbie 3y ago
I think it's far more likely they're building off some open source foundational models, rather than doing one from scratch OR using GPT under the hood. If it was the latter, don't you think we'd hear about how "Grok is on par with GPT"? But you don't, people think it's not as good (but funnier?)

deleted 3y ago

[deleted]

3cats-in-a-coat 3y ago

Yes, it does put into question the basic competence of its makers. Like how hard it is to massage at least the explicit brand mentions out of the data.

TradingPlaces 3y ago

"Large language models (LLM) have become state of the art in many benchmarks and conversational LLM applications like ChatGPT are now widely used by the public. Those LLMs can be used to generate large amounts of content which is posted on the internet to various platforms. As LLMs are trained on datasets usually collected from the internet, this LLM-generated content might be used to train the next generation of LLMs. Therefore, a self-consuming training loop emerges in which new LLM generations are trained on the output from the previous generations. We empirically study this self-consuming training loop using a novel dataset to analytically and accurately measure quality and diversity of generated outputs. We find that this self-consuming training loop initially improves both quality and diversity. However, after a few generations the output inevitably degenerates in diversity. We find that the rate of degeneration depends on the proportion of real and generated data.” https://arxiv.org/abs/2311.16822 https://arxiv.org/abs/2311.16822