3 ms·
Would you be able to tell if it did? There could be an obscure document in the training set that contains the idea. It seems like a very hard problem to definit
by pfannkuchen 2y ago
Would you be able to tell if it did? There could be an obscure document in the training set that contains the idea. It seems like a very hard problem to definitively detect whether a concept came from the training set.
- valine 2y agoA single document doesn’t contribute much to the loss during base training. LLMs can absolutely memorize text if it’s duplicated in the training set, not so much if the document in question is obscure.