4 ms·
I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (as
by dmarchand90 3y ago
I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (aside from good/ unique data) they just have a good set of hyperparameters used in training and it's too expensive for a competitor to find equal or better settings
- porompompero 3y agoSorry for my ignorance: why does each experiment cost millions?
- Jensson 3y agoBecause training a model costs millions, so each time you experiment with trying to create a new kind of model it costs millions.
- bart_spoon 3y agoIt’s the cost of compute hardware required to train a model of that size
- jwuphysics 3y agoI would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.
- dmarchand90 3y agoAre they really "laws" my impression is its all just a bunch of empirical trends. We cannot know truly how these parameters interact at large scale and also how they interact with each other. Is it really the case that openai has data that Google doesn't?