The usefulness of data here is that you can retrain the model after making changes to its architecture, e.g. seeing if it works better with a different activati

The usefulness of data here is that you can retrain the model after making changes to its architecture, e.g. seeing if it works better with a different activation function. Of course this is most useful for models small enough that you could train it within a few days on a consumer GPU. When it comes to LLMs only the richest companies would have the adequate resources to retrain.