> You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replica

> You have no idea what fraction of their techniques they're not publishing, and how much they contribute to their model performance, because you cannot replicate the models, because they don't publish their training data. Why would you be concerned about THEIR model performance ?! Surely if you are an ML researcher and read about a new technique, you are interested in how YOU may be able to use it. If you are Ilya Sutskever sitting at OpenAI in 2017, and happen upon Google's "attention" (Transformer architecture) paper, then what you do is go and implement it for yourself, get yourself some training data, and try it. What you are NOT going to do is whine about not being given their source code, or their training data, or their training harness, or a dump of Google's corporate secrets. You take the research that has been shared and evaluate it for yourself.