3 ms·
That we are running out of data makes continued improvement more likely, in the medium/long term, because it means we have found close to the maximum amount of
by zarzavat 4y ago
That we are running out of data makes continued improvement more likely, in the medium/long term, because it means we have found close to the maximum amount of static data that models need to be trained on. This means that one of the factors for compute requirements (static training data size) has reached its upper bound while still within our computational capacity today, and further improvements must come from elsewhere.
So yes, easy data scaling is coming to an end and may or may not lead to a short term winter. But this also means that, all being equal, training a model will be cheaper compared to a situation where we still had orders of magnitude more data to go.