Overfitting happens when you have more parameters than datapoints, what's currently happening would be closer to underfitting, but the reality is that we were v

Overfitting happens when you have more parameters than datapoints, what's currently happening would be closer to underfitting, but the reality is that we were very far from underfitting, we just didn't want to invest the compute necessary to train smaller models for a long time because we thought they weren't at the optimal position on the fitting curve. I'd say we are just sliding a little bit closer to underfitting, but are still far from it because we still have many parameters to work with.