Similar to Moore's law, "transformer" deep neural nets have been found to following scaling laws[0]. This means the faster and more VRAM your GPU's have, the be

Similar to Moore's law, "transformer" deep neural nets have been found to following scaling laws[0]. This means the faster and more VRAM your GPU's have, the better a model you can train "for free". Training models from scratch only works with massive (labeled) datasets covering a massive data distribution. With language models, the datasets being used are quickly approaching "all known written text" sizes. Training a model from scratch on Microsoft's internal code, with not only its precious intellectual property, but also its technical debt. Code at Microsoft is not going to get close to covering the broad range of styles that a coder could possibly use. The model will possibly diverge without enough data, as it needs to see a given "example usage" in multiple different contexts before it can learn it. My current understanding is that deep NN's are quite good at modeling an underlying distribution of data without needing any priors hard-coded about that dataset. But! They need to see a whole lot more of it than an adult human would. Several orders of magnitude more. - and they need to see accurate labels about 75-80% of the time. [0] https://www.lesswrong.com/tag/scaling-laws https://www.lesswrong.com/tag/scaling-laws