3 ms·

I'm wondering if we are now in the Low Background Steel transition, as far as Internet content goes. I already see material on the open Internet that has very o

I'm wondering if we are now in the Low Background Steel transition, as far as Internet content goes. I already see material on the open Internet that has very obviously been generated by AI. As the next round of Common Crawl or whatever is slurped in for training, AI ends up eating its own output. Does the quality of the material degrade at that point? Maybe we'll end up searching for an Internet that existed before AI started rewriting it. https://en.wikipedia.org/wiki/Low-background_steel?wprov=sfla1 https://en.wikipedia.org/wiki/Low-background_steel?wprov=sfl...

etrautmann 2y ago
Yes - this insight has been repeatedly discovered by many, but remains a great analogy.
- klipt 2y ago
  There are two main types of data for training intelligences (natural or artificial): 1. Self play 2. Data left behind by other intelligences' self play. The Internet is 2, but is generated by the self play of humans - who were themselves trained on the self play of previous humans. That's how civilization is bootstrapped. Once you have bootstrapped sufficiently smart AI, they can possibly bootstrap themselves further on their own self play, instead of continuing to rely on human self play data.
  - 8note 2y ago
    Natural gas at least one more main type of training data, and it's much bigger than self play for natural training: Testing against the universe. Self play and self play ignores the whole of Empiricism
dinfinity 2y ago
Training data is preprocessed, you know. I don't understand why everybody making this argument ignores that. It's not as if the internet before LLMs did not have tons of trash content (which also includes things written by humans who sincerely think they're right, but that is factually incorrect). Of course the input data is preprocessed and curated/weighted. Of course newer training data will also be curated. Think for a second about what you said: "I already see material on the open Internet that has very obviously been generated by AI." Why would any AI company worth its salt _not_ ignore such data?
n_ary 2y ago
With the AI progress, we will probably finally see a slowing down of the “js framework of the month” and other rapidly changing things. To a bunch of very junior devs using LLMs to generate code and build something, speed is now main factor, it is irrelevant whether the new Airbnb prototype or even prod version was being generated in Svelte 3 as long as it works. The fine tuning and fixing up happens later on “if needed” basis. Similar apply for similar knowledge where AI is also being utilized in similar manner. A slightly outdated info is sufficient for the task. Also, internet will continue ti have decent content as there will always be passionate people who want to build and share knowledge, then there is the need to advertise your knowledge(consultant/freelancer) who still need to generate content in decent quality to be discovered. The maximum slop will be in sales/marketing/social media sphere where content quality is irrelevant and more clicks/engagement brings profit. Anything else will be continuously get locked behind paywall, so either exclusive contracts with AI companies to supply training data and sell on open market with strong copyright or drm to general consumers.

etrautmann 2y ago

Yes - this insight has been repeatedly discovered by many, but remains a great analogy.

klipt 2y ago
There are two main types of data for training intelligences (natural or artificial): 1. Self play 2. Data left behind by other intelligences' self play. The Internet is 2, but is generated by the self play of humans - who were themselves trained on the self play of previous humans. That's how civilization is bootstrapped. Once you have bootstrapped sufficiently smart AI, they can possibly bootstrap themselves further on their own self play, instead of continuing to rely on human self play data.
- 8note 2y ago
  Natural gas at least one more main type of training data, and it's much bigger than self play for natural training: Testing against the universe. Self play and self play ignores the whole of Empiricism

dinfinity 2y ago

Training data is preprocessed, you know. I don't understand why everybody making this argument ignores that. It's not as if the internet before LLMs did not have tons of trash content (which also includes things written by humans who sincerely think they're right, but that is factually incorrect). Of course the input data is preprocessed and curated/weighted. Of course newer training data will also be curated. Think for a second about what you said: "I already see material on the open Internet that has very obviously been generated by AI." Why would any AI company worth its salt _not_ ignore such data?

n_ary 2y ago

With the AI progress, we will probably finally see a slowing down of the “js framework of the month” and other rapidly changing things. To a bunch of very junior devs using LLMs to generate code and build something, speed is now main factor, it is irrelevant whether the new Airbnb prototype or even prod version was being generated in Svelte 3 as long as it works. The fine tuning and fixing up happens later on “if needed” basis. Similar apply for similar knowledge where AI is also being utilized in similar manner. A slightly outdated info is sufficient for the task. Also, internet will continue ti have decent content as there will always be passionate people who want to build and share knowledge, then there is the need to advertise your knowledge(consultant/freelancer) who still need to generate content in decent quality to be discovered. The maximum slop will be in sales/marketing/social media sphere where content quality is irrelevant and more clicks/engagement brings profit. Anything else will be continuously get locked behind paywall, so either exclusive contracts with AI companies to supply training data and sell on open market with strong copyright or drm to general consumers.