4 ms·
Excellent read. Was searching for a full text search engine but not finding any suitable one. Plan to implement one just this way.
by habibur 6y ago
Excellent read. Was searching for a full text search engine but not finding any suitable one. Plan to implement one just this way.
- bartdegoede 6y agoAuthor here: you may want to check out something like Whoosh https://whoosh.readthedocs.io/en/latest/intro.html https://whoosh.readthedocs.io/en/latest/intro.html (it's like a clone of Lucene but in pure Python). I've used this to build some basic search for a small Python website and it was more than fast enough for my purposes :-)
- rakoo 6y agoSQLite has a pretty good built-in fts engine: https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html
- habibur 6y agoProblem is FTS5 isn't included in the most default installation through package managers [I use Fedora]. And recompiling from source breaks a lot of things, as sqlite libraries are generally linked with all apps that use it.
- rakoo 6y agoI admit I only used sqlite through the go driver (https://github.com/mattn/go-sqlite3 https://github.com/mattn/go-sqlite3) where using fts5 amounts to one flag during the compile phase.
- edwinyzh 6y agoBut SQLite's FTS5 has no support for the `offsets` function...
- dmitriid 6y agoMany years ago I ran into this paper "Self-indexing inverted files for fast text retrieval" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.18.8282&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.18.... It's short and to the point. And then I implemented all that ... in PHP and MySQL :) It feels daunting at first, but once you understand what it wants you to do, it's actually not that hard (for this particular paper, and this particular approach). However, you do want to employ a stemming library to normalize word forms.