5 ms·

There's no more text to train on. Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show tha

by fizx 4y ago

There's no more text to train on. Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show that football players learned musical instruments 50% quicker than non-athletes. That would be amazing! But LLMs can write music and training them on music makes them better writers. It's almost like intelligence & skills are completely general!

CuriouslyC 4y ago
That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).
osigurdson 4y ago
The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.
- MattRix 4y ago
  It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.
  - osigurdson 4y ago
    I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.
- ihatepython 4y ago
  > The fact that AI requires so much data does make me think that we are somehow on the wrong track. I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.
- IIAOPSW 4y ago
  Not just that. Language acquisition. Teacher never stands in front of class and says "here's what the rules of grammar aren't. You today learn want do not how." There is a paucity of negative examples in the "training". Same goes for so many other tasks we learn quicker than can be explained with a Hebbian punishment/reward model. Most people can play a board game somewhat competently after just being told the rules. Some are even good at it. You don't need to crash your car dozens of time to figure out how to drive. All over we see something fundamental missing, and the industry is trying to hammer the screw into the wall while pretending that's the best way to do it.

by fizx 4y ago

CuriouslyC 4y ago

That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).

osigurdson 4y ago

The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.

MattRix 4y ago
It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.
- osigurdson 4y ago
  I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.
ihatepython 4y ago
> The fact that AI requires so much data does make me think that we are somehow on the wrong track. I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.
IIAOPSW 4y ago
Not just that. Language acquisition. Teacher never stands in front of class and says "here's what the rules of grammar aren't. You today learn want do not how." There is a paucity of negative examples in the "training". Same goes for so many other tasks we learn quicker than can be explained with a Hebbian punishment/reward model. Most people can play a board game somewhat competently after just being told the rules. Some are even good at it. You don't need to crash your car dozens of time to figure out how to drive. All over we see something fundamental missing, and the industry is trying to hammer the screw into the wall while pretending that's the best way to do it.