3 ms·
The dataset is so large they only do one single pass, and the content is deduplicated. A study showed that it's usually duplicated content that is overfitted an
by visarga 4y ago
The dataset is so large they only do one single pass, and the content is deduplicated. A study showed that it's usually duplicated content that is overfitted and later regurgitated. Examples that are repeated in many places might have slight variations that help them pass the filter.
- Kim_Bruning 4y agoInteresting! Do you have a link(s) to the study, or more details on training?
- visarga 4y agoDeduplicating Training Data Makes Language Models Better https://arxiv.org/pdf/2107.06499 https://arxiv.org/pdf/2107.06499 Quantifying Memorization Across Neural Language Models https://arxiv.org/pdf/2202.07646.pdf https://arxiv.org/pdf/2202.07646.pdf