3 ms·
Hm. I read this expecting a more in-depth discussion about best practices for fine-tuning massive transformers while avoiding catastrophic forgetting, ie. * Ho
by whimsicalism 5y ago
Hm. I read this expecting a more in-depth discussion about best practices for fine-tuning massive transformers while avoiding catastrophic forgetting, ie.
* How should you select the learning rate?
* What tasks are best for fine-tuning on small amounts of data?
etc.
Instead, this seems mostly to just be running through the implementation of ML/DL 101: loss function for binary classification, helper functions to load data, etc.
- jpulliam 5y agoAuthor here! This tutorial was mainly setup for people new to ML/DL and walk them through usage of a pre-trained model, creating the dataloader, writing the training loop, etc. In the future we definitely plan to dive further into the details and touch on some of the things you mentioned! This was one of the primary reasons why we chose the "Sentiment Analysis" task as it's fairly simply to get a model trained quickly with good performance.