4 ms·
Yes, presumably the larger corpus would tend to contain more "rare" words than the smaller. I think you may want to weigh individual terms by their inverse freq
by troels 12y ago
Yes, presumably the larger corpus would tend to contain more "rare" words than the smaller. I think you may want to weigh individual terms by their inverse frequency in the full corpus or a representative sample (Look up tf-idf if you haven't already). I'm not sure this is a magic bullet, but it might.