On true for all - you’d need to split it by era I think During the early Llama 1 days The Pile dataset was in heavy use by many. Bit later people figured out t

On true for all - you’d need to split it by era I think During the early Llama 1 days The Pile dataset was in heavy use by many. Bit later people figured out that a subset of it - Books 3 - was especially problematic. I’m guessing all the big houses threw that piece out in later models since it’s extra radioactive