5 ms·

Has anyone used sqlite for storing embeddings? Are there any extensions or tips for making it easier? I have a small command line python app that uses sqlite f

by nico 3y ago

Has anyone used sqlite for storing embeddings? Are there any extensions or tips for making it easier? I have a small command line python app that uses sqlite for a db. Postgres would be a huge overkill for the app PS: is sqlite-vas good? https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss

berkes 3y ago
Personally, I'd use meilisearch in your case (though, obv, I know next to nothing about all the requirements and details of your case). Stuffing everything in PG makes sense, because PG is Good Enough in most situations that one throws at it, but hardly "very good". It's just "the best" because it is already there¹. PG truly is "very good", maybe even "the best" at relational data. But all other cases: there are better alternatives out there. Meilisearch is small. It's not embeddable (last time I looked) so It does run as separate service; but does so with minimal effort and overhead. And aside from being blazingly fast in searching (often faster than elasticsearch; definitely much easier and lighter), it has stellar vector storage. ¹ I wrote e.g. github.com/berkes/postgres_key_value a Ruby (so, slow) hashtable-alike interface that uses existing Postgres tech for a situation where Redis and Memcached would've been a much better candidate, but where we already had Postgres, and with this library, could postpone the introduction, complexity and management of a redis server. Edit: forgot this footnote.
- ComputerGuru 3y ago
  Meilisearch is embeddable as of recently, but you have to do some legwork since it’s not the intended use case.
- esafak 3y ago
  Have you used it for hybrid search? What kind of performance do you observe?
ComputerGuru 3y ago
We do, from years ago, with embedding stored as blobs (well before sqlite_vss). No SQLite extension but some user-defined functions for some distance operations and for others we are able to load all weights into memory and perform the ops there (data is small enough). Another comment mentioned meilisearch. You might be interested in the fact that a recent rearchitecture of their codebase split it into a library that can be embedded into your app without needing a separate process.
wiredfool 3y ago
Duckdb seems pretty reasonable for embeddings, as there's a native vector type and you can insert it from a dataframe. For similarity, there's array_cosine and array_dot_product.

by nico 3y ago

berkes 3y ago

Personally, I'd use meilisearch in your case (though, obv, I know next to nothing about all the requirements and details of your case). Stuffing everything in PG makes sense, because PG is Good Enough in most situations that one throws at it, but hardly "very good". It's just "the best" because it is already there¹. PG truly is "very good", maybe even "the best" at relational data. But all other cases: there are better alternatives out there. Meilisearch is small. It's not embeddable (last time I looked) so It does run as separate service; but does so with minimal effort and overhead. And aside from being blazingly fast in searching (often faster than elasticsearch; definitely much easier and lighter), it has stellar vector storage. ¹ I wrote e.g. github.com/berkes/postgres_key_value a Ruby (so, slow) hashtable-alike interface that uses existing Postgres tech for a situation where Redis and Memcached would've been a much better candidate, but where we already had Postgres, and with this library, could postpone the introduction, complexity and management of a redis server. Edit: forgot this footnote.

ComputerGuru 3y ago
Meilisearch is embeddable as of recently, but you have to do some legwork since it’s not the intended use case.
esafak 3y ago
Have you used it for hybrid search? What kind of performance do you observe?

ComputerGuru 3y ago

We do, from years ago, with embedding stored as blobs (well before sqlite_vss). No SQLite extension but some user-defined functions for some distance operations and for others we are able to load all weights into memory and perform the ops there (data is small enough). Another comment mentioned meilisearch. You might be interested in the fact that a recent rearchitecture of their codebase split it into a library that can be embedded into your app without needing a separate process.

wiredfool 3y ago

Duckdb seems pretty reasonable for embeddings, as there's a native vector type and you can insert it from a dataframe. For similarity, there's array_cosine and array_dot_product.