3 ms·
This all makes me wonder: how reproducible is the final output model?
by amelius 4y ago
This all makes me wonder: how reproducible is the final output model?
- dekhn 4y agothe details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.
- screye 4y agoNot this one, but Google's PaLM (which is 4x OPT3) trains semi-deterministically. These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.
- ur-whale 4y ago> how reproducible is the final output model? Not much, but it also depends on what you mean by "reproducible". Do you expect a similar internal representation (weights) or a similar behavior?