3 ms·

if you need to do really complicated file search, duckdb has an extension that lets you run SQL on your file system could be an option. https://duckdb.org/comm

if you need to do really complicated file search, duckdb has an extension that lets you run SQL on your file system could be an option. https://duckdb.org/community_extensions/extensions/hostfs.html https://duckdb.org/community_extensions/extensions/hostfs.ht...

ramonverse 2y ago
Duckdb can't even hanlde edge cases in a csv (pandas does it no problem) so I wouldn't trust it to do anything useful at the filesystem level
thatgurjot 2y ago
Thanks! Search wasn’t the most difficult problem here. I found SQLite’s FTS5 to be quite powerful. Later combined with Tantivy, I think it’s a superb combination to tackle pretty much any local search needs.
- serial_dev 2y ago
  What were the most difficult problems you faced, if it wasn’t search?
  - thatgurjot 2y ago
    Getting everything to work reliably. This was my first time handling a software with so many moving parts. If I could get the search query to work, I would break the date range filter, if that worked, the document indexing flow would stutter, or I’d discover a new bug caused by an edge-case. Since I built this application in an exploratory manner (branching out without a larger architectural design), it often became unwieldy for a solo hobbyist like myself. Plus, I wasn’t able to really get going with user feedback. Reached out to family, friends and colleagues but faced the Windows roadblock and got demotivated. In the end, built it for myself and used it by myself. I am happy with that!
- AlexErrant 2y ago
  What issues did you have with FTS5 that made you switch to Tantivy? Does using Tantivy mean you have to load the entire Sqlite db into memory? I ask because in my own project I have my own search query grammar [0] written in Lezer that more or less "transpiles" [1] to SQL+FTS5 that's been uh, tedious to build. [0] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query.grammar [1] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query2sql.ts

ramonverse 2y ago

Duckdb can't even hanlde edge cases in a csv (pandas does it no problem) so I wouldn't trust it to do anything useful at the filesystem level

thatgurjot 2y ago

Thanks! Search wasn’t the most difficult problem here. I found SQLite’s FTS5 to be quite powerful. Later combined with Tantivy, I think it’s a superb combination to tackle pretty much any local search needs.

serial_dev 2y ago
What were the most difficult problems you faced, if it wasn’t search?
- thatgurjot 2y ago
  Getting everything to work reliably. This was my first time handling a software with so many moving parts. If I could get the search query to work, I would break the date range filter, if that worked, the document indexing flow would stutter, or I’d discover a new bug caused by an edge-case. Since I built this application in an exploratory manner (branching out without a larger architectural design), it often became unwieldy for a solo hobbyist like myself. Plus, I wasn’t able to really get going with user feedback. Reached out to family, friends and colleagues but faced the Windows roadblock and got demotivated. In the end, built it for myself and used it by myself. I am happy with that!
AlexErrant 2y ago
What issues did you have with FTS5 that made you switch to Tantivy? Does using Tantivy mean you have to load the entire Sqlite db into memory? I ask because in my own project I have my own search query grammar [0] written in Lezer that more or less "transpiles" [1] to SQL+FTS5 that's been uh, tedious to build. [0] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query.grammar [1] https://github.com/AlexErrant/Pentive/blob/main/shared-dom/src/language/query2sql.ts