3 ms·
Maybe not 'median' but rather 'sufficiently representative', as with all distributional semantics, given a large enough corpus we can approach the 'true' distri
by dleeftink 2y ago
Maybe not 'median' but rather 'sufficiently representative', as with all distributional semantics, given a large enough corpus we can approach the 'true' distribution of word/phrases in a given language.
- krainboltgreene 2y agoExcept the corpus itself is fractional of all media. This is like saying Twitter is sufficiently representative of all human history.
- dleeftink 2y agoWe are truly dealing with immense sizes here, see [0]. I am not saying it is representative of all things, but that large corpora are sufficiently representative of words and phrases in a language (and by extension, the words and phrases assoiciated with topics and ideas). In written form you may miss some slang, pidgins and dialects, but those would be underrepresented in other media as well. [0]: https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...