3 ms·
Except the corpus itself is fractional of all media. This is like saying Twitter is sufficiently representative of all human history.
by krainboltgreene 2y ago
Except the corpus itself is fractional of all media. This is like saying Twitter is sufficiently representative of all human history.
- dleeftink 2y agoWe are truly dealing with immense sizes here, see [0]. I am not saying it is representative of all things, but that large corpora are sufficiently representative of words and phrases in a language (and by extension, the words and phrases assoiciated with topics and ideas). In written form you may miss some slang, pidgins and dialects, but those would be underrepresented in other media as well. [0]: https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...