5 ms·

What's the memory consumption like? A node for each letter seems like it would result in substantial memory consumption. I was thinking an object pool might hel

What's the memory consumption like? A node for each letter seems like it would result in substantial memory consumption. I was thinking an object pool might help here, but I'm not sure if its possible to use it given the nesting of the nodes Also, if the library removes stop words, does that mean its limited in which languages it supports? I'm assuming only English stop words are filtered Not being critical here, just genuinely curious and don't have time to look through the code ATM

deleted 9y ago
[deleted]
olivernn 9y ago
Lunr.js [1] used to use a trie data structure and memory usage was a concern. The problem is that although the data structure compresses prefixes, it does nothing for suffixes. Depending on the corpus this can lead to a lot of duplication. Switching to a trie-like structure that compresses prefixes and suffixes can lead to significant savings. Building the structure can be a bit more burdensome, so there is a trade off there. There is a paper describing the approach [2] and, if you're interested, my JavaScript implementation [3][4]. [1] https://lunrjs.com https://lunrjs.com [2] http://www.aclweb.org/anthology/J00-1002.pdf http://www.aclweb.org/anthology/J00-1002.pdf [3] https://github.com/olivernn/lunr.js/blob/master/lib/token_set.js https://github.com/olivernn/lunr.js/blob/master/lib/token_se... [4] https://github.com/olivernn/lunr.js/blob/master/lib/token_set_builder.js https://github.com/olivernn/lunr.js/blob/master/lib/token_se...
- gandreani 9y ago
  Cheers! Thanks for sharing. EDIT: I like the site design too!
kbr 9y ago
Yeah, the memory is the only downside of Wade. I'm still looking into ways to optimize the trie used while still keeping the size around 1-2kb. The stop words that are being removed can be configured via `Wade.config.stopWords`, which is an array of stop words that Wade will remove.
- gandreani 9y ago
  I appreciate your efforts at keeping the library footprint small. It seems like less and less people are concerned with this nowadays Good thinking on making them configurable!

deleted 9y ago

[deleted]

olivernn 9y ago

Lunr.js [1] used to use a trie data structure and memory usage was a concern. The problem is that although the data structure compresses prefixes, it does nothing for suffixes. Depending on the corpus this can lead to a lot of duplication. Switching to a trie-like structure that compresses prefixes and suffixes can lead to significant savings. Building the structure can be a bit more burdensome, so there is a trade off there. There is a paper describing the approach [2] and, if you're interested, my JavaScript implementation [3][4]. [1] https://lunrjs.com https://lunrjs.com [2] http://www.aclweb.org/anthology/J00-1002.pdf http://www.aclweb.org/anthology/J00-1002.pdf [3] https://github.com/olivernn/lunr.js/blob/master/lib/token_set.js https://github.com/olivernn/lunr.js/blob/master/lib/token_se... [4] https://github.com/olivernn/lunr.js/blob/master/lib/token_set_builder.js https://github.com/olivernn/lunr.js/blob/master/lib/token_se...

gandreani 9y ago
Cheers! Thanks for sharing. EDIT: I like the site design too!

kbr 9y ago

Yeah, the memory is the only downside of Wade. I'm still looking into ways to optimize the trie used while still keeping the size around 1-2kb. The stop words that are being removed can be configured via `Wade.config.stopWords`, which is an array of stop words that Wade will remove.

gandreani 9y ago
I appreciate your efforts at keeping the library footprint small. It seems like less and less people are concerned with this nowadays Good thinking on making them configurable!