4 ms·

The "Do you need a Dedicated Vector Database?" slide is quite interesting, but doesn't really answer its own question! This is something I've been wondering mys

The "Do you need a Dedicated Vector Database?" slide is quite interesting, but doesn't really answer its own question! This is something I've been wondering myself, if anyone has any guidelines or rules of thumb I would appreciate it. I've recently been using Simon Willison's (hi simonw) excellent `llm` tool that can help with embeddings, and it takes the simplest approach possible: just store embeddings in SQLite with a few UDFs for calculating distance etc. The simplicity of that approach is very appealing, but presumably at some level of traffic+data an application will outgrow it and need a more specialized database. Does anyone have a good intuition for where that cutoff might be?

summarity 3y ago
I’ve had some success scaling a quick and dirty engine (that powers findsight.ai) to tens of millions of vectors, details in the talk here: https://youtu.be/elNrRU12xRc?t=1556 https://youtu.be/elNrRU12xRc?t=1556 That’s maybe 1kLOC so I didn’t need an external one after all.
jide_tracc 3y ago
Thanks for the feedback! I'll make this the topic of my next article and do a deeper dive on this
darkteflon 3y ago
YMMV, of course, but this[1] article - posted here a few weeks back - really helped us to focus on how to think about what we actually needed: in short, a search engine. [1] https://news.ycombinator.com/item?id=38703943 https://news.ycombinator.com/item?id=38703943
tristanho 3y ago
Honestly just loading all vectors in-memory (and stuff like sqlite, pgvector) is totally fine when you're dealing with O(100k) vectors, but beyond that all the workable options like pinecone get gnarly, slow, and ridiculously expensive. The best option by far I know of is turbopuffer.com , which is like 100x cheaper than pinecone and seems to actually scale. Since it's not listed in the suggested vector dbs section of the slides, wanted to lob it in as a solid suggestion :)
- tmoertel 3y ago
  There are options such as Google's ScaNN that may let you go farther before needing to consider specialized databases. https://github.com/google-research/google-research/blob/master/scann%2FREADME.md https://github.com/google-research/google-research/blob/mast...
estebarb 3y ago
Some years ago I wrote a vector index that used LSH. Search was implemented in the most trivial way: scanning everything and comparing using hamming distance (xor and popcount). I was able to scan 200k hashes and find the most similar in less than 10ms, in a single core of a 2011 MBP.
- winrid 3y ago
  That's cool, but 200k is pretty low. Also 10ms with logn and indexes hot in memory should be able to sort through a billion or so index keys at least

summarity 3y ago

I’ve had some success scaling a quick and dirty engine (that powers findsight.ai) to tens of millions of vectors, details in the talk here: https://youtu.be/elNrRU12xRc?t=1556 https://youtu.be/elNrRU12xRc?t=1556 That’s maybe 1kLOC so I didn’t need an external one after all.

jide_tracc 3y ago

Thanks for the feedback! I'll make this the topic of my next article and do a deeper dive on this

darkteflon 3y ago

YMMV, of course, but this[1] article - posted here a few weeks back - really helped us to focus on how to think about what we actually needed: in short, a search engine. [1] https://news.ycombinator.com/item?id=38703943 https://news.ycombinator.com/item?id=38703943

tristanho 3y ago

Honestly just loading all vectors in-memory (and stuff like sqlite, pgvector) is totally fine when you're dealing with O(100k) vectors, but beyond that all the workable options like pinecone get gnarly, slow, and ridiculously expensive. The best option by far I know of is turbopuffer.com , which is like 100x cheaper than pinecone and seems to actually scale. Since it's not listed in the suggested vector dbs section of the slides, wanted to lob it in as a solid suggestion :)

tmoertel 3y ago
There are options such as Google's ScaNN that may let you go farther before needing to consider specialized databases. https://github.com/google-research/google-research/blob/master/scann%2FREADME.md https://github.com/google-research/google-research/blob/mast...

estebarb 3y ago

Some years ago I wrote a vector index that used LSH. Search was implemented in the most trivial way: scanning everything and comparing using hamming distance (xor and popcount). I was able to scan 200k hashes and find the most similar in less than 10ms, in a single core of a 2011 MBP.

winrid 3y ago
That's cool, but 200k is pretty low. Also 10ms with logn and indexes hot in memory should be able to sort through a billion or so index keys at least