This is impressive indeed. I trained BERT from scratch on Colab's K80s spending a few hours (not 100h as in the article) on a much smaller corpus. My results

This is impressive indeed. I trained BERT from scratch on Colab's K80s spending a few hours (not 100h as in the article) on a much smaller corpus. My results were understandably rather horrible.