4 ms·

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805

by make3 7y ago

deleted 7y ago
[deleted]
The_rationalist 7y ago
Except that BERT is now obscoleted by https://github.com/zihangdai/xlnet https://github.com/zihangdai/xlnet (but xlnet would never have existed without BERT)
- ivalm 7y ago
  Kind of, there are a bunch of transformers that might perform better than BERT (Ernie 2.0 being stronger than xlnet, for example), but often this is a function of training size (xlnet trained on 10x more data than original BERT). Realistically there are now BERTs released finetuned for special corpa (biobert, clinical bert, etc) so if you want to work on those kind of texts you are better off starting with a BERT that was previously fine tuned to something close to your task (and then fine tune it more yourself).
  - The_rationalist 7y ago
    Well you comment was really interesting to me because I didn't know ERNIE 2.0 and it's concept of continual learning seems to be really a step forward! But some of you statements seems incorrect: Ernie 2.0 being stronger than xlnet XLnet is the neural net with the biggest number of first places on benchmarck leaderboards. Cf: https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... While ernie 2.0 has currently 0 first place on paperswithcode.com https://paperswithcode.com/paper/ernie-20-a-continual-pre-training-framework https://paperswithcode.com/paper/ernie-20-a-continual-pre-tr... xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) Well your point on finetuned bert vs non finetuned xlnet is interesting. ROBERTA is so fine tuned it beat XLnet on some tasks. But generally xlnet non finetuned beat BERT finetuned and there are more and more xlnet finetuned each week. But your point does apply for Roberta, and for the few tasks where bert as been applyed but xlnet hasn't yet.
    - ivalm 7y ago
      > xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read the XLNet paper https://arxiv.org/pdf/1906.08237.pdf https://arxiv.org/pdf/1906.08237.pdf they clearly state in section 3.1: "Following BERT [10], we use the BooksCorpus [41] and English Wikipedia as part of our pretraining data, which have 13GB plain text combined. In addition, we include Giga5 (16GB text) [23], ClueWeb 2012-B (extended from [5]), and Common Crawl [6] for pretraining. We use heuristics to aggressively filter out short or low-quality articles for ClueWeb 2012-B and Common Crawl, which results in 19GB and 78GB text respectively. After tokenization with SentencePiece [16], we obtain 2.78B, 1.09B, 4.75B, 4.30B, and 19.97B subword pieces for Wikipedia, BooksCorpus, Giga5, ClueWeb, and Common Crawl respectively, which are 32.89B in total" If you compare to BERT paper https://arxiv.org/pdf/1810.04805.pdf https://arxiv.org/pdf/1810.04805.pdf training data for "pretraining data" section in also section 3.1: "Pre-training data The pre-training procedure largely follows the existing literature on language model pre-training. For the pre-training corpus we use the BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2,500M words). For Wikipedia we extract only the text passages and ignore lists, tables, and headers. It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to extract long contiguous sequences." So 32.89B words for XLNet vs 3.3B words for BERT. We've also run fine tuning experiments supplementing additional private medical corpus (~10B words) and felt starting from clinical-bert was better than xlnet (for our rather specific use cases).
- make3 7y ago
  then Facebook's roberta came out which beat xlnet and is essentially a more intelligently trained. I included XLNet as being a bc descendant of BERT, but I guess they are all descendants of GPT1
wyldfire 7y ago
Gee, it seems like the state of the art moves very quickly. These are less than twelve months old?!

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805

by make3 7y ago

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805

deleted 7y ago

[deleted]

The_rationalist 7y ago

Except that BERT is now obscoleted by https://github.com/zihangdai/xlnet https://github.com/zihangdai/xlnet (but xlnet would never have existed without BERT)

ivalm 7y ago
Kind of, there are a bunch of transformers that might perform better than BERT (Ernie 2.0 being stronger than xlnet, for example), but often this is a function of training size (xlnet trained on 10x more data than original BERT). Realistically there are now BERTs released finetuned for special corpa (biobert, clinical bert, etc) so if you want to work on those kind of texts you are better off starting with a BERT that was previously fine tuned to something close to your task (and then fine tune it more yourself).
- The_rationalist 7y ago
  Well you comment was really interesting to me because I didn't know ERNIE 2.0 and it's concept of continual learning seems to be really a step forward! But some of you statements seems incorrect: Ernie 2.0 being stronger than xlnet XLnet is the neural net with the biggest number of first places on benchmarck leaderboards. Cf: https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... While ernie 2.0 has currently 0 first place on paperswithcode.com https://paperswithcode.com/paper/ernie-20-a-continual-pre-training-framework https://paperswithcode.com/paper/ernie-20-a-continual-pre-tr... xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) Well your point on finetuned bert vs non finetuned xlnet is interesting. ROBERTA is so fine tuned it beat XLnet on some tasks. But generally xlnet non finetuned beat BERT finetuned and there are more and more xlnet finetuned each week. But your point does apply for Roberta, and for the few tasks where bert as been applyed but xlnet hasn't yet.
  - ivalm 7y ago
    > xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read the XLNet paper https://arxiv.org/pdf/1906.08237.pdf https://arxiv.org/pdf/1906.08237.pdf they clearly state in section 3.1: "Following BERT [10], we use the BooksCorpus [41] and English Wikipedia as part of our pretraining data, which have 13GB plain text combined. In addition, we include Giga5 (16GB text) [23], ClueWeb 2012-B (extended from [5]), and Common Crawl [6] for pretraining. We use heuristics to aggressively filter out short or low-quality articles for ClueWeb 2012-B and Common Crawl, which results in 19GB and 78GB text respectively. After tokenization with SentencePiece [16], we obtain 2.78B, 1.09B, 4.75B, 4.30B, and 19.97B subword pieces for Wikipedia, BooksCorpus, Giga5, ClueWeb, and Common Crawl respectively, which are 32.89B in total" If you compare to BERT paper https://arxiv.org/pdf/1810.04805.pdf https://arxiv.org/pdf/1810.04805.pdf training data for "pretraining data" section in also section 3.1: "Pre-training data The pre-training procedure largely follows the existing literature on language model pre-training. For the pre-training corpus we use the BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2,500M words). For Wikipedia we extract only the text passages and ignore lists, tables, and headers. It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to extract long contiguous sequences." So 32.89B words for XLNet vs 3.3B words for BERT. We've also run fine tuning experiments supplementing additional private medical corpus (~10B words) and felt starting from clinical-bert was better than xlnet (for our rather specific use cases).
make3 7y ago
then Facebook's roberta came out which beat xlnet and is essentially a more intelligently trained. I included XLNet as being a bc descendant of BERT, but I guess they are all descendants of GPT1

wyldfire 7y ago

Gee, it seems like the state of the art moves very quickly. These are less than twelve months old?!