3 ms·

AI winter will arrive if we don’t get the models to depend on ‘just’ more training data to get better. There is no more training data. We need models the same o

AI winter will arrive if we don’t get the models to depend on ‘just’ more training data to get better. There is no more training data. We need models the same or better than gpt-4 but trained on roughly what an 18 year old >100 iq human would digest to get to that point. Which is vastly less than what gpt 4 gets fed. If advancing means ever larger and more expensive systems and ever more data, we will enter a cold winter soon.

pixl97 4y ago
There is nearly an unlimited amount of training data. As of so far models have been eating up text. We still have sound, image, video, temperature, gravimetrics, and other sources that we can feed multimodal models. And that's not even including training models from learning from the world itself.
- anonzzzies 4y ago
  Sound, image and video we’ll ravish through (already happening); is temp and gravimetrics valuable? (I don’t know; it’s a question) Games might be a source too. But yes, the world itself is a good source. So maybe that won’t freeze it over; computing power/energy?
  - pixl97 4y ago
    >computing power/energy? Energy is "probably" not a big deal. If you're looking at long term oversupply from green energy sources, it's probably not hard to train with bursty, but very cheap power like this. Compute is currently the biggest limitation, and will remain so far a long time, as long as scaling continues.
- akiselev 4y ago
  > gravimetrics Calling it now: humanity will seal its fate the day we hook up LIGO to ChatGPT and it gets corrupted by gravitational waves from the beginning of time when the Great Old Ones freely roamed the universe.
stevenhuang 4y ago
You are confusing the entirety of all text corpus to _all data_. The next step is multi-modal data. Think videos, sounds, touch, etc. An 18 year old in fact "trained" on much more input data than GPT4, orders of magnitudes more, considering the complete informational content of our 5 senses and implicit structure of our brains "learned" through evolution. Not to mention recent research in LLM scaling (chinchilla) reveals how our current models are undertrained and over-parameterized. We have not plateaued yet.
- PaulDavisThe1st 4y ago
  > The next step is multi-modal data. Think videos, sounds, touch, etc. Remember that the data must be annotated. This puts limits on what can be usefully ingested.
  - stevenhuang 4y ago
    Not necessarily, see the developments in self supervised learning. It's what's relied on for the majority of the RLHF phase in current LLMs.
  - mlboss 4y ago
    What if it can annotate itself ? With some feedback loop it can learn the correct annotation by experiments like a child. Earlier there was a comment of an agent learning to run a python program by itself and in the process of writing code to file and install python interpreter.

pixl97 4y ago

There is nearly an unlimited amount of training data. As of so far models have been eating up text. We still have sound, image, video, temperature, gravimetrics, and other sources that we can feed multimodal models. And that's not even including training models from learning from the world itself.

anonzzzies 4y ago
Sound, image and video we’ll ravish through (already happening); is temp and gravimetrics valuable? (I don’t know; it’s a question) Games might be a source too. But yes, the world itself is a good source. So maybe that won’t freeze it over; computing power/energy?
- pixl97 4y ago
  >computing power/energy? Energy is "probably" not a big deal. If you're looking at long term oversupply from green energy sources, it's probably not hard to train with bursty, but very cheap power like this. Compute is currently the biggest limitation, and will remain so far a long time, as long as scaling continues.
akiselev 4y ago
> gravimetrics Calling it now: humanity will seal its fate the day we hook up LIGO to ChatGPT and it gets corrupted by gravitational waves from the beginning of time when the Great Old Ones freely roamed the universe.

stevenhuang 4y ago

You are confusing the entirety of all text corpus to _all data_. The next step is multi-modal data. Think videos, sounds, touch, etc. An 18 year old in fact "trained" on much more input data than GPT4, orders of magnitudes more, considering the complete informational content of our 5 senses and implicit structure of our brains "learned" through evolution. Not to mention recent research in LLM scaling (chinchilla) reveals how our current models are undertrained and over-parameterized. We have not plateaued yet.

PaulDavisThe1st 4y ago
> The next step is multi-modal data. Think videos, sounds, touch, etc. Remember that the data must be annotated. This puts limits on what can be usefully ingested.
- stevenhuang 4y ago
  Not necessarily, see the developments in self supervised learning. It's what's relied on for the majority of the RLHF phase in current LLMs.
- mlboss 4y ago
  What if it can annotate itself ? With some feedback loop it can learn the correct annotation by experiments like a child. Earlier there was a comment of an agent learning to run a python program by itself and in the process of writing code to file and install python interpreter.