7 ms·
Microsoft’s New MT-DNN Outperforms Google BERT
- braindead_in 8y agoWas the code released?
- phowon 8y agoIt's relatively small modification of BERT with multi-task fine-tuning and slightly different output heads. It should be easy for any NLP researcher to replicate.
- riku_iki 8y agoexcept you need significant GPU/TPU resources to pretrain language model.
- bitL 8y agoYou can't even train BERT_large on a 12/16GB GPU, and on a single 15TFlops GPU it might take a year to train. GPUs are too slow :-(
- riku_iki 8y agoTPU is also slow, they used pod with 64 TPUs for training BERT. You probably can achieve similar result using distributed training on multiple GPU machines.
- solomatov 8y agoYou can but it will be really slow. You can load just parts of the model, and store them on the disk/in memory :-)
- solomatov 8y agoThe authors of the paper didn't pretrain the language model. They used an existing BERT and fine tuned it in a novel way.
- riku_iki 8y agoCould you provide citation? I tried to find this but couldn't.
- solomatov 8y ago>The training procedure of MT-DNN consists of two stages: pretraining and multi-task fine-tuning. The pretraining stage follows that of the BERT model (Devlin et al., 2018). The parameters of the lexicon encoder and Transformer encoder are learned using two unsupervised prediction tasks: masked language modeling and next sentence pre- diction.3 and this: >Our implementation of MT-DNN is based on the PyTorch implementation of BERT4. We used Adamax (Kingma and Ba, 2014) as our optimizer with a learning rate of 5e-5 and a batch size of 32. The maximum number of epochs was set to 5. A linear learning rate decay schedule with warm-up over 0.1 was used, unless stated otherwise. Fol- lowing (Liu et al., 2018a), we set the number of steps to 5 with a dropout rate of 0.1. To avoid the exploding gradient problem, we clipped the gradi- ent norm within 1. All the texts were tokenized using wordpieces, and were chopped to spans no longer than 512 tokens. You won't be able to train BERT in 3 epochs.
- solomatov 8y agoHere's the quote from BERT: >We train with batch size of 256 sequences (256 sequences * 512 tokens = 128,000 tokens/batch) for 1,000,000 steps, which is approximately 40 epochs over the 3.3 billion word corpus.
- danielcampos93 8y agoCan confirm from conversations I had with the authors.
- pacala 8y ago"Microsoft will release the code and pre-trained models.", though there is no pointer to where the release will happen. Training gargantuan language models is getting quite expensive, so releasing code + pre-trained models is significant. The architecture is a derivation of PyTorch BERT [0], with an MTL loss function on top. [0] https://github.com/huggingface/pytorch-pretrained-BERT https://github.com/huggingface/pytorch-pretrained-BERT
- TaylorAlexander 8y agoCompare also to OpenAI’s recent post on their own NLP work. https://blog.openai.com/better-language-models/ https://blog.openai.com/better-language-models/
- sp332 8y agoIs it weird I can't find anything on a Microsoft domain about this? Why does only Synced Review have this news?
- manojlds 8y agoProbably the official sources are still preparing the write up and code / pretrained models release.
- yorwba 8y agoThe paper has been on arXiv for a while already. https://arxiv.org/abs/1901.11504 https://arxiv.org/abs/1901.11504 Maybe Microsoft doesn't feel like making a big splash about their research?
- solomatov 8y agoTheir model is based on BERT, so essentially it's a BERT fine tuned in a novel way.
- riku_iki 8y agoAnd BERT is OpenAI transformer, finetuned in a novel way, and OpenAI transformer is Tensor2Tensor transformer finetuned in a novel way )
- solomatov 8y agoTo summarize the achievements: * Attention is all you need transformer created a non recurrent architecture for NMT (https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762) * OpenAI GPT modified the original transformer by changing architectutre (one net instead of encoder/decoder pair), and using different hyperparameters which seems to work the best (https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://s3-us-west-2.amazonaws.com/openai-assets/research-co...) * BERT used GPT's architecture but trained in a different way. Instead of training a language model, they forced the model predict holes in a text and predicting whether two sentences go one after another. (https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805) * OpenAI GPT2 achieved a new state of the art in language models (https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...) * The paper in the top post found out that if we fine tune several models in the same way as in BERT, we get improvement in each of the fine tuned models.
- riku_iki 8y agoAlso: * OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model. * BERT created bigger model (16 layers in GPT vs 24 layers in BERT), proving that larger Transformer models increase performance
- 8y ago
- mr_toad 8y agoIf we have vendors touting benchmarks, it can’t be long before Gartner starts putting ML models in magic quandrants.
- dav43 8y agoI’m interested in this space and the technical aspects. However, can someone enlighten me as to specific problems these models solve? Any real world implementations? Eg we had this problem, we used this tool, these are the outcomes?
- karlakush 8y agoIf there were real world problems to be solved with this technology, I can't imagine Microsoft and Google would be telling the world how it works.
- solomatov 8y agoTransformer model which this paper was baed on was used in Starcraft 2 bot by DeepMind.
- tedivm 8y agoThese models are really good at translating from one language to another and summarizing text.
- danielcampos93 8y agoDocument ranking, question answering, caption creation, keyword generation. Basically these model are super useful to search engines or any kind of engine that performs some kind of reasoning with text.
- aboutruby 8y agoLink to the paper: https://arxiv.org/pdf/1901.11504.pdf https://arxiv.org/pdf/1901.11504.pdf