Look at the scaling laws. We found that extrapolating the performance given a few data points with smaller models is actually very accurate. That's how they de

Look at the scaling laws. We found that extrapolating the performance given a few data points with smaller models is actually very accurate. That's how they determined hyper parameters, by tuning them on multiple smaller scale models and then extrapolating. So far, all those predictions were quite good. Together with a bigger model, we also need more data to get better performance. If we add video and audio to the text data, we have still a lot more data we can use, so this is also not really a problem. It would be very unexpected that those scaling laws are suddenly not true anymore for the next order of magnitude in model and data size.