4 ms·
It looks like you're running searches using LIKE: https://github.com/NotJoeMartinez/yt-fts/blob/050981c0519a966593d2e2405bf2116997a00909/db_scripts.py#L81 https
by simonw 4y ago
It looks like you're running searches using LIKE: https://github.com/NotJoeMartinez/yt-fts/blob/050981c0519a966593d2e2405bf2116997a00909/db_scripts.py#L81 https://github.com/NotJoeMartinez/yt-fts/blob/050981c0519a96...
SQLite has a really power full-text search mechanism built in - FTS5. It can handle things like stemming and stop words and relevance ranking.
My sqlite-utils Python library includes helper methods for setting that up: https://sqlite-utils.datasette.io/en/stable/python-api.html#full-text-search https://sqlite-utils.datasette.io/en/stable/python-api.html#...
- notjoemartinez 3y agoThank you! I was able to integrate this into the project[1]. I'm also looking into using your openai-to-sqlite[2] library for semantic search. [1]https://github.com/NotJoeMartinez/yt-fts/pull/25 https://github.com/NotJoeMartinez/yt-fts/pull/25 [2]https://github.com/simonw/openai-to-sqlite https://github.com/simonw/openai-to-sqlite
- lfconsult 4y agoYou're right. Thanks for sharing the link to your full-text search helper, really neat.