When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still

When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money. In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go. We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.