4 ms·
Case in point, collocation detection over very large corpora: https://github.com/RaRe-Technologies/bounter#example-on-the-english-wikipedia https://github.com/
by Radim 9y ago
Case in point, collocation detection over very large corpora:
https://github.com/RaRe-Technologies/bounter#example-on-the-english-wikipedia https://github.com/RaRe-Technologies/bounter#example-on-the-...
Reduces RAM requirements some 31x times when automatically detecting common multiword expressions (e.g. "New York", "network license" or "Apache Hadoop"), which is significant.
- 3131s 9y agoYep, I load an enormous list of lexemes (including multiword expressions) into a bloom filter and use it for text segmentation.