3 ms·
Word embeddings work because they reflect co-occurrences. I don't know whether that counts as an explanation in terms of psychology, but humans tend to put rela
by yorwba 8y ago
Word embeddings work because they reflect co-occurrences. I don't know whether that counts as an explanation in terms of psychology, but humans tend to put related things together. In a newspaper the articles aren't jumbled together, but there are sections on different topics, and in each section the articles are clearly delineated instead of mixing their sentences and each sentence represents a single unit instead of giving partial information on a dozen unrelated things.
It might seem obvious that things should be done that way, but if you consider servers hosting lots of different websites on the same physical machine, or data structures spread out over several memory allocations held together by pointers, it's clear that there are other possibilities. So it does seem to be specific to the way humans use language.
And because human language has this property of co-occurrences corresponding to relatedness in meaning, you can represent the meaning of a word by building a model that only predicts the probability that two words occur together.
- PeterisP 8y agoIn linguistics, there's a classic principle "You shall know a word by the company it keeps" (Firth, J. R. 1957) - colocations (a sequence of words or terms that co-occur more often than would be expected by chance) are very informative about what a word means.