4 ms·

I think most labs actively create synthetic data using existing model as part of the mix for the pretraining stage for their next model. Would love to know exa

I think most labs actively create synthetic data using existing model as part of the mix for the pretraining stage for their next model. Would love to know exactly what the latest process is to keep slop out of training data.

martinald 6mo ago
const isAiContent = (str) => str.includes('—');? :)
- AlienRobot 6mo ago
  Latest generation LLM's use en dashes instead of em dashes to avoid detection.
  - jondwillis 6mo ago
    No, they don’t. But obviously GP was tongue–—in-–cheek.
madamelic 6mo ago
I think everyone overblows the whole "AI is poisoning AI!" thing. It could be a problem but the genuine value in Reddit or any other human social media is honestly pretty low from my estimates. It's great for seeing how humans talk but in terms of 'nutritional' value for truth or answers... I am not sold. If I was choosing what to 'feed' AI, I wouldn't even bother with textual social media (besides Github / Gitlab / other source control) There's way more value, if seeking out answers, in following the links to external sources, scraping books, and other sources that aren't "unwashed masses saying whatever they want".
- beepbooptheory 6mo ago
  You can put it in scare quotes all you want, doesn't stop you from sounding like Scrooge McDuck.
- palmotea 6mo ago
  > the genuine value in Reddit or any other human social media is honestly pretty low from my estimates. It's great for seeing how humans talk but in terms of 'nutritional' value for truth or answers... > ... > scraping books, and other sources that aren't "unwashed masses saying whatever they want". The problem is there's a lot of knowledge that only exists as reddit comments, blog posts, or social Q&A.
- SV_BubbleTime 6mo ago
  > genuine value in Reddit or any other human social media Kind of a steep slope to convince me that Reddit has a higher P/E ratio than Nvidia because of its authentic content… Reddit seems extremely overvalued because it is a network of people programmed to accept whatever they see and engage. No doubt that there is good information there, discord too. But Reddit is far more bots and “organic propaganda” than you are thinking.

martinald 6mo ago

const isAiContent = (str) => str.includes('—');? :)

AlienRobot 6mo ago
Latest generation LLM's use en dashes instead of em dashes to avoid detection.
- jondwillis 6mo ago
  No, they don’t. But obviously GP was tongue–—in-–cheek.

madamelic 6mo ago

I think everyone overblows the whole "AI is poisoning AI!" thing. It could be a problem but the genuine value in Reddit or any other human social media is honestly pretty low from my estimates. It's great for seeing how humans talk but in terms of 'nutritional' value for truth or answers... I am not sold. If I was choosing what to 'feed' AI, I wouldn't even bother with textual social media (besides Github / Gitlab / other source control) There's way more value, if seeking out answers, in following the links to external sources, scraping books, and other sources that aren't "unwashed masses saying whatever they want".

beepbooptheory 6mo ago
You can put it in scare quotes all you want, doesn't stop you from sounding like Scrooge McDuck.
palmotea 6mo ago
> the genuine value in Reddit or any other human social media is honestly pretty low from my estimates. It's great for seeing how humans talk but in terms of 'nutritional' value for truth or answers... > ... > scraping books, and other sources that aren't "unwashed masses saying whatever they want". The problem is there's a lot of knowledge that only exists as reddit comments, blog posts, or social Q&A.
SV_BubbleTime 6mo ago
> genuine value in Reddit or any other human social media Kind of a steep slope to convince me that Reddit has a higher P/E ratio than Nvidia because of its authentic content… Reddit seems extremely overvalued because it is a network of people programmed to accept whatever they see and engage. No doubt that there is good information there, discord too. But Reddit is far more bots and “organic propaganda” than you are thinking.