2 ms·
It depends on what the BERT was pre-trained on. If it was pre-trained on Google News Collection like most Transformer models are, fine-tuning on something like
by vinni2 5y ago
It depends on what the BERT was pre-trained on. If it was pre-trained on Google News Collection like most Transformer models are, fine-tuning on something like scholarly data won’t perform so well due to vocabulary mismatch. Besides as the author mentions training models like LightGBM are efficient and perform reasonably.
- spratzt 5y agoI agree with this. In my experience, LightGBM is often good enough. The additional improvement provided by neural net models is not worth the substantial increase in development time.
- dunefox 5y ago> pre-trained on Google News Collection like most Transformer models are BERT for example was trained on Wikipedia and Book corpora. Do you have a source for this claim?