3 ms·
Word frequencies follow a power law, so nearly all words are used very infrequently. So a randomly chosen word is likely to be very uncommon. Also there are man
by wongarsu 5y ago
Word frequencies follow a power law, so nearly all words are used very infrequently. So a randomly chosen word is likely to be very uncommon. Also there are many more specific concepts than general ones (because each general concept encompasses many specific ones). So a randomly chosen word is more likely to be specific than general.
Random sampling is likely to give you unusual, highly specific words. I don't see how that wouldn't do very good. Sure, you could do better by mapping each word to a concept, then selecting ten random concepts and selecting a word for each concept. But why do that when you get 99% of the way there with just random choice.
To show this experientially, I let python choose 10 random words from the first English word list I found. I got "epilithic, codirects, serologic, capsian, digestibility, yasna, samsara, supergun, hepatoptosis, kyurin". Sure, capsian and kyurin are somewhat related (names of places/people), as are serologic and hepatoptosis (both medical terms), but it's a pretty good spread.
- rich_sasha 5y agoNot poster but I think the point is you could do better, not that this doesn’t work. You can imagine having some discrete optimisation maximising the dissimilarity of words, though that’s hard to do, in particular in a discrete space.
- PeterisP 5y agoThe example you got was the exact thing I had in mind - there are "terminology-heavy" fields with lots of terms, and a random draw is quite likely to draw two items from a single field which IMHO should count as quite related. The automated scoring method they're using seems to say that all obscure words are unrelated to each other, though, but I'd use that as a criticism for that evaluation algorithm not as praise for the evaluated method. On the other hand, probably there are many humans who would pick two local animals or two office-related objects that happen to be in their room, so that cuts both ways.