I think it became apparent when mixtral came out. I've noticed too during training that my model overwrites useful information so it makes sense for these types

I think it became apparent when mixtral came out. I've noticed too during training that my model overwrites useful information so it makes sense for these types of models to have emerged.