4 ms·

Liquid does amazing work, but I kinda feel like they are overtraining their models. 38T tokens seems like a lot for an 8B model

• andai 4mo ago
  What's the downside? Don't they stop when they hit diminishing returns?
  • Ifkaluva 4mo ago
    You’d think so, but I haven’t seen it explicitly discussed in their papers, and nobody else that I know of trains on that many tokens
  • hgoel 4mo ago
    Wouldn't the model start overfitting at some point? Degrading generalization for accuracy on the training set.