I think it would be interesting to deflate out to a huge dataset and see where this happens. Certainly it will occur as the generated data exceeds the original

I think it would be interesting to deflate out to a huge dataset and see where this happens. Certainly it will occur as the generated data exceeds the original, eg after 1-10T tokens. I think you could also do this faster by moving down the tree in a depth first manner. Typically I use this for knowledge transfer, style transfer, catastrophic forgetting mitigation, etc and so I don’t go very far. I usually manually review the data samples before using it.