At the MIT event, Altman was asked if training GPT-4 cost $100 million; he replied, “It's more than that.” Training costs are decreasing, but whenever there's

At the MIT event, Altman was asked if training GPT-4 cost $100 million; he replied, “It's more than that.” Training costs are decreasing, but whenever there's an update to GPT-4 (e.g. training data cutoff updated to December 2023), it means the model has been retrained. The compute costs of training a model like Claude 2 are significant. Also, keep in mind that not every trained model becomes ready for production. Some are discarded, similar to how you might burn a few cookies while baking.