"Microsoft will release the code and pre-trained models.", though there is no pointer to where the release will happen. Training gargantuan language models is

"Microsoft will release the code and pre-trained models.", though there is no pointer to where the release will happen. Training gargantuan language models is getting quite expensive, so releasing code + pre-trained models is significant. The architecture is a derivation of PyTorch BERT [0], with an MTL loss function on top. [0] https://github.com/huggingface/pytorch-pretrained-BERT https://github.com/huggingface/pytorch-pretrained-BERT