OpenAI hasn’t trained on The Pile, as far as I know. I think you mean "Did Meta buy a copy of every book in books3?" since llama wasn’t trained on The Pile eith

OpenAI hasn’t trained on The Pile, as far as I know. I think you mean "Did Meta buy a copy of every book in books3?" since llama wasn’t trained on The Pile either. And the answer is no. It seems important for the answer to remain no, otherwise the only entities that can afford to train on sufficient numbers of books will be big companies. No one can afford 190,000 books except huge corporations, so we’ll be surrendering our ability to train our own useful models if the price of training is that high. 190k books isn’t even that much. OpenAI probably trained on millions.