8 ms·

Do we think about vector storage wrong?

139 points by softwaredoug 3y ago

quickthrower2 3y ago
Forget RAGs and LLMs, Vim search and a little regex is all you need
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
ankit219 3y ago
Been following this space for a bit. In my view, the literature on the internet is distorted about these things because of the need for certain OSS project creators and (non OSS) companies to promote their own products. It typically goes like this: [Insert technique name] is a good technique, but it can be further improved if you enable [some complex workflow]. [Some example of how it improved a particular case]. We at [Company] already do this at scale and you can use it out of the box if you use it. Go make better things. Nothing against people doing this stuff. It just makes me think how difficult is it to promote open source projects today. Regarding Vector DBs, most have advocated for weaviate and pinecone. Incidentally, all run into quantization problems at scale (1M+ vectors) if the clusters are to be adjusted, and handle it differently. i tried it for a 6000 vector product[1] and got the best results by using a pkl file rather than any of the dbs. [1] Not sharing the product here as such. We just scraped a govt tax website to create a search engine on top of that. Helps as the data is not indexed by google, and we had no interest from customers in wanting to chat. They were interested in original source along w a summary.
- Palmik 3y ago
  You likely won't see performance (speed) benefits if you have just tens of thousands of vectors and of course brute force (vector matrix multiplication) will have better precision & recall. Vector databases make sense when you have a lot of data or some other requirements (e.g. persistence, online updates, complex filtering, etc) that you would rather not implement yourself.
  - Uncroyable 3y ago
    exactly. brute force methods work great for thousands pieces of data of (almost) any kind. for vector data no vector db is needed; for structured data even a simple csv file with a python script will do the job and a sql db is not necessary.
    - brigadier132 3y ago
      The thing is, depending on the dimensions of the vectors your using the growth rate can be pretty drastic. Each openai vector is 5kb and 1500 dimensions. So sure searching 1k vectors, just brute force it. But what if you have 100k vectors and a ton of users hammering at the index searching through it? Each search is now 150,000,000 dot products. Go up one more order of magnitude and you definitely need an index. And I don't think 1 million vectors is that much.
      - ankit219 3y ago
        Number of vectors are determined by (other than original dataset ofc) how you chose to chunk the data you have available. Bigger chunks work better in terms of search (empirically) and they also keep the number of vectors down. For openai, based on prevalant norms and their cookbooks, 1M vectors likely mean 1M (more like 700K) pdf pages of text (at a token size of 1000 per embedding). That is a lot of textual data for a decent size company. Enterprises might reach that stage. Consulting firms definitely would - though they already trained and announced their own models.
        brigadier132 3y ago
        700k pdf pages is not a lot. Also, you might be a business serving other businesses and indexing their documents and at reasonable (again, not google scale), 700k pages is again not a lot. Another way to look at 700k pages is 2333 300 page books.
  - deleted 3y ago
    [deleted]
- keyle 3y ago
  Bit sad how it's going. People go straight to product and venture scaled growth. Build protocols, not products! I tell people. It's a terrible advice actually, unless you like eating noodles and scraping by. But it's still the right way. We're still waiting for the antirez' Redis version of Vector DBs. Maybe he should do one! :)
  - vkb 3y ago
    The good news is that this already exists in the Redis search module [1], which allows you to do similarity search against indexed embeddings, among other features, and offers comparable performance against other ANN libraries [2], depending on your performance criteria. I've been using it for a side project to do semantic search on books[3] and have been really happy with its performance. (Not affiliated with any of this, was mostly interested in exploring existing well-performing, fairly standard tools with low latency) [1] https://redis.io/docs/interact/search-and-query/search/vectors/ https://redis.io/docs/interact/search-and-query/search/vecto... [2] https://ann-benchmarks.com/#redisearch https://ann-benchmarks.com/#redisearch [3] https://viberary.pizza/how https://viberary.pizza/how
    - neeleshs 3y ago
      Bit of a side note on Redis Search. While performance is pretty good, it is not production quality yet, mostly around monitoring indexing errors and quirky query results. Silent indexing errors pretty much make it impossible to use it in production, especially for structured data.
  - manishsharan 3y ago
    Redis Enterprise edition does have Vector database.i haven't used it but am curious about other people's experiences. https://redis.com/solutions/use-cases/vector-database/ https://redis.com/solutions/use-cases/vector-database/
- gk1 3y ago
  > Regarding Vector DBs, most have advocated for weaviate and pinecone. Incidentally, all run into quantization problems at scale (1M+ vectors)... Can you elaborate? We (at Pinecone) have dozens of customers with over a billion embeddings, still serving queries in under 300ms (p95) with index updates in <1s. If you saw performance issues with just 1M embeddings then something else is going on that we may be able to help with.
  - ankit219 3y ago
    It's a specific case. Most customers won't run into it. It's to do with quantization or specifically what happens when you add enough vectors that the original clustering changes (use cases could be hit when you onboard a IM/consulting customer which has so much data to alter clusters amongst others). It is not a performance issue to be fair. Just that altering a cluster takes time. I was just testing to see if there is a DB which does not run into such problems, before putting a business case. I am not discouraging the use, just saying that most cases do not need a vector DB. The reason we did not choose one was because our business case's number of vectors was small.
    - jamesbriggs 3y ago
      I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken). Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index) So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.
redwood 3y ago
This space is thr ultimate in premature optimization and obsession or implementation details (including the classic "it's written in programming language X so it has to be better") with relatively little real world experiential knowledge in the mix
dmezzetti 3y ago
The focus on the top 10 in vector search is a product of wanting to prove value over keyword search. Keyword search is going to miss some conceptual matches. You can try to work around that with tokenization and complex queries with all variations but it's not easy. Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy https://github.com/spotify/annoy) has been around since 2014. It was one of the first open source approximate nearest neighbor libraries. Recommendations have always been a good use case for vector similarity. Recommendations are a natural extension of search and transformers models made building the vectors for natural language possible. To prove the worth of vector search over keyword search, the focus was always on showing how the top N matches include results not possible with keyword search. In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) operates in the sweet spot. Once again this is validated through the same benchmarks which focus on the top 10. On top of all this, there is also the reality that the vector database space is very crowded and some want to use their performance benchmarks for marketing. Disclaimer: I am the author of txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai), an open source embeddings database
- marginalia_nu 3y ago
  Amusingly, vector search is conceptually nearly as old as keyword search. Most older IR textbooks promote the idea of viewing keyword search as a maximization problem of the weighted bit vector product between the query and the document, where the weight matrix is something like BM-25.
  - PaulHoule 3y ago
    I was interested in dense vector search in 2004 and a bit depressed at how the indexing algorithms weren’t that good, Around 2013 I was involved with an actual product which was a search engine for patents. The thing about vector search today is that it is an absolute gold rush, Pinecone had the right idea but the wrong product (sorry SAAS is toxic), many of the latecomers are… late, and will find the whole startup and VC process will cost precious months when customers want to build products right now. Note for that patent search engine we had a quite a different idea than is fashionable today, that is, like very old TREC we care about the quality of results you got 1000 or 2000 results in in the assumption that a patent researcher wanted to be comprehensive. When I was first into IR it was in the context of “how is Google so much better than other search engines?” and one part of it was that based on the way search engines were being evaluated at TREC, Google was not a better search engine* because it was so focused on the first page. It has come full circle now, I have talked to people recently who complain that current TREC evaluations are too focused on the first few results.
Andrew018 3y ago
It's possible that our current perspective on vector storage could be flawed or incomplete, and exploring alternative approaches may yield valuable insights and improvements in data management and computation.
jamesliaamm 3y ago
[flagged]
9dev 3y ago
Anyone have insights on using Elasticsearch as a vector database as opposed to a specialised vector DB? We have a complex keyword search already, and would like to introduce knn query ability to find similar documents. ES offers that in recent versions, but I’m wondering whether that’s a good idea.
- softwaredoug 3y ago
  Jimmy Lin, a researcher a U Waterloo, recently published "Lucene is All You Need" https://arxiv.org/abs/2308.14963 https://arxiv.org/abs/2308.14963 I know, FWIW, Elasticsearch has been investing a lot in this space lately. It's not perfect, but its bound to get better.
  - peterstjohn 3y ago
    That paper does a terrible job of making Lucene look useful, though. 10qps from a server with 1TB of RAM is not great (and I know Lucene HNSW can perform better than that in the real world, so I am somewhat mystified that this paper is being pushed by the community).
- jillesvangurp 3y ago
  I looked at this a few months ago. Elasticsearch has some limitations with respect to the vector length (1024, I think); which rules out using a lot of the popular off the shelf models. A key insight is that the performance of off the shelf models isn't great compared to a hand tuned query and bm25 (the Lucene ranking algorithm). I've seen multiple people make the point that the built in ranking is pretty hard to beat unless you specialize your models to your use case. A key consideration for the vector size limitation is that storing and working with large amounts of huge vectors gets expensive quickly. Simply storing lots of huge embeddings can take up a lot of space. And of course using knn with huge result sets is very expensive, especially if the vectors are large. Having the ability to filter down the result set with a regular query and then ranking the candidates with a vector query helps keep searches responsive and cost low. If you are interested in this, you might want to look at Opensearch as well. They implemented vector search independently from Elasticsearch. Their implementation supports a few additional vector storage options and querying options via native libraries. I haven't used any of that extensively but it looks interesting.
pedrovhb 3y ago
I'm not sure I understand. Isn't vector storage about storage and retrieval strictly, being a separate thing from the embedding generation itself? My mental model is that the job of vector storage is to just fetch the closest vectors to a target vector for some proximity metric, with the model (or method , more generally) that generates embeddings being its own distinct problem. You may later do things like passage re-ranking and maximal marginal relevance to refine it, but again, different task. I understand vector search against tons of other vectors is an approximation endeavor due to practical concerns, so it doesn't seem strange for vector store accuracy to be measured in terms of top-k retrieval. Post processing and the quality of the vectors themselves is a different thing to judge.
Double_a_92 3y ago
For starters I don't believe that people even understand or agree what "vector" even means in this context.
- fodkodrasz 3y ago
  Yes, the word is a bit overloaded... I guess from skimming through the article that here it is vector in the sense of word2vec algorithms.
- simonw 3y ago
  A great deal of the current excitement around vector search is related to embeddings. I tried to provide a very clear explanation of embeddings here, since I know lots of people haven't yet learned what they are or why they are interesting: https://simonwillison.net/2023/Sep/4/llm-embeddings/ https://simonwillison.net/2023/Sep/4/llm-embeddings/
Androider 3y ago
Remember the "Mongo DB is web scale" craze from now over a decade ago? That's the current state of vector DBs, despite vector DBs having been around forever. Do you have more than a terabyte of embeddings? No? Do yourself a favor and use pg_vector with a HNSW index, it'll do just fine. Operationally, it is very hard to beat Postgres.
- drittich 3y ago
  And if you just have tens of thousands, straight up SQL can work pretty well, too.
  - pornel 3y ago
    If you have tens of thousands, a `for` loop over an array in memory is plenty fast. I've worked on a project that searched through 50 million vectors with linear brute-force search, just with a sprinkle of AVX (and it used the brute-force search, because the curse of dimensionality killed all the smarter approaches).
- osigurdson 3y ago
  An OpenAI embeddings vector is 1536 4 byte floats. 1 TiB is roughly 174K such embeddings vectors.
  - brigadier132 3y ago
    Your math is wrong. 100k 32 bit vectors is 600mb I think your point is right though. Searching through these is requires an index of some sort at any reasonable scale (not google scale).
    - osigurdson 3y ago
      Hmmm, yeah, I used 2^30 instead of 2^40. Should not comment before caffeinating.
osigurdson 3y ago
One thing I have been wondering about is how, concretely, in practice embeddings will be used. Will companies create embeddings for most / all content that they have? Will they create an embedding for every sentence, paragraph or page of text (or all of those). Will they store hierarchies of embeddings? Do they then store the original text so they can invert the process (that seems obvious)?
- WinLychee 3y ago
  User/Item/Query embeddings are the most common. That way you can generate per-user recommendations, or search results for a given query (with personalization using side information). Video will be interesting, once we have video embeddings (maybe this exists already). It depends on the use-case but a few of your ideas are certainly possible. Generally I've seen them at a coarse rather than fine level, but I'm sure that's out there too. This looks like a good overview if you want to read about it: https://recsysml.substack.com/p/two-tower-models-for-retrieval-of https://recsysml.substack.com/p/two-tower-models-for-retriev...
chaxor 3y ago
Wow! Not the point of the article, but it is striking to see how mastodon is way better in terms of usability/aesthetic and general UX than Twitter or 'x'. I've been mildly interested in mastodon for quite some time from a political perspective (remove large support of big companies where possible), but I didn't realize it was just sooo much better to use.
- sp332 3y ago
  And there are tons of third party clients. I think Tusky is the best one I’ve seen for Android, and there’s an interesting web-based one called Elk that’s very nice. You load up https://elk.zone https://elk.zone and then use it as a front-end to sign in to your server.
  - jameshart 3y ago
    Also the mastodon web client installs great as a PWA. Just add your home server as a shortcut to your home screen - no client install needed.
- softwaredoug 3y ago
  I really enjoy Mastodon and believe it has staying power. Not rocket ship “going to explode and burn out” power. There’s enough there to keep people engaged. The decentralized nature means there’s always corners of it alive and growing.
rvrs 3y ago
This is tangential, but another issue I've had with existing vector databases is write throughput. Weaviate and Pinecone build an index immediately after the creation of a "table" (or rather, the analogue thereof) and this makes write throughput abysmal at times. We have around 2 million 768-length embeddings + metadata -- writing too fast to Weaviate quickly results in OOMs. We circumvented this by very slowly trickling our data through to Weaviate, but this causes many-hours-long writes. I'm experimenting with dumping our data into postgres and building an hnsw index with pgvector once all writes are done. This has reduced our time to maybe 2 hours at most (the bulk of that being index building time, of course).
- rectang 3y ago
  I once solved worst-case write performance in an open source MVCC data store (Apache Lucy, a “loose C” port of Lucene) like so: 1) Limit the maximum size of an update. This causes gradual degradation of the index, which is ordinarily dealt with through periodic consolidation — but now this this consolidation will be disabled in the main write mechanism. 2) Enable a background consolidation process which runs concurrently. At the end of the consolidation process a short write lock will need to be obtained while you replay all deletions that have occurred since consolidation started, but the time required isn’t problematic. This fix required major surgery to the Lucy indexer core code, so it’s something that that would only be feasible if the data store itself supported it.
- aslPine 3y ago
  @rvrs, Have you tried Pinecone's S1 pods? Curious which indexing type you went with when experimenting (https://docs.pinecone.io/docs/indexes https://docs.pinecone.io/docs/indexes) Disclaimer: I'm from Pinecone, hello!
- mish15 3y ago
  Yeah basically all the vector "database" solutions in market have chosen data-dependent indexes, so you need the data upfront. Imagine if regular databases needed all data upfront before they could build indexes. It's kind of crazy...
- trengrj 3y ago
intalentive 3y ago
1. Check out this cool chat-with-my-docs app I made with LangChain! 2. Okay, let's productize! 3. Ouch, the results aren't quite production-ready... 4. Huh, the bottleneck seems to be the search & retrieval process... 5. Oh, I guess I don't know anything about search, time to catch up on the field... 6. Ok, now I need to build an inverse index or knowledge graph and augment it with embeddings. 7. Hm, actually, language models (including BERT-based, Splade) could be pretty useful in building structure out of my dataset... IMO the best way to incorporate transformers into search has yet to be discovered, although whatever OpenEvidence.com is doing seems to work well. If I were still in grad school I would look into some kind of PageRank / graph NN approach to the problem, or maybe hyperbolic embeddings. Reason being that for many corpora there exists some kind of internal structure or linking that can be exploited for speed and accuracy, which you don't quite get from nearest neighbors in a Euclidean space.
yujian 3y ago
Who's focusing on top 10 recall? I recently saw someone ask if we (I work at Zilliz) can update Milvus' recall to 10 million nearest neighbors lol
hintymad 3y ago
Eh... Because keyword and BM25 do not work that well, especially for enterprise search? Many of us surely complained how bad enterprise search was, right? A company usually has a hodgepodge of documents. They don't link to each other that much so page rank does not work well. They have fewer than 1 click per day, if there is any at all. So signals based on user interactions will not be useful or will be expensive to build. We often search with keywords that show up in enough number of documents but not enough to make BM25 rank the matches high. Query understanding hardly exists in enterprise search, because building query autocompletion, query correction, query reformulation and etc all requires some level of semantic understanding. If it's not embedding based, it has to be stats-based, which requires enough volume of clicks and user evaluation that enterprise search can't afford. The list can go on and on. There is a reason that Google's search appliance never took off, and people are still complaining about quality of enterprise search. Amazingly, a true semantic search would work well. And to do semantic search in scale, you'll need a vector database.
- aaroninsf 3y ago
  I work in an adjacent area, and this is the most useful and information-dense summary I've read in any forum in at least a week. Word!
- code_biologist 3y ago
  Dear god this is accurate. A product I worked on a few years ago wanted to add a feature searching various PDFs that were a slurry of the same words. After being disappointed with the results of keyword search, we deemed semantic search too complicated to implement and scrapped the feature. Great to see it's getting easier.
aslPine 3y ago
Do we "obsess over perfect accuracy in top 10 for vector dbs"? I don't see a ton of data that points towards us (vector DB ppl) building towards traditional TREC/BEIR #s. I do see lots of us in the vector DB space demonstrating on real data the speed and semantic retrieval capabilities of vector DBs. Not many of us publish traditional IR benchmark #s simply because vector search is a pretty different ballgame than traditional keyword/BM25 search (as you know). However, since most search practitioners have developed their mental models around things like TFIDF and BM25 + Learning To Rank, many vector DBs tailor their language to fit that heuristic. If you look at vector DBs on their own -- I like to look at them as a piece of infra to use in conjunction with pre/post processing + RAG + BM25 + whatever else you want -- you'll find that measuring their accuracy against traditional IR benchmarks is a bit like fitting a square peg into a round hole. It almost fits! But not quite. I think one of the starkest differences between the vector search and traditional search when it comes to retrieval evaluation is the great difficulty involved in creating judgment lists to measure vector search performance. Since vector search inherently retrieves documents that are approximately conceptually similar to a query, the initial recall set, especially with an IRL dataset (e.g. Reddit, Shopify, etc.) is a moving target. Language evolves. Idiosyncrasies pop up. Etc. Not to say that those ^ issues don't plague keyword search as well. But the powerful thing about vector databases IMO is that unlike traditional indices that need lots of fine tuning and IR experts, a vector DB will capture these new additions to your domain's vocabulary immediately, based on the surrounding context in the document. Of course you can do similar things with traditional search engines by incorporating curated lists of synonyms, antonyms, stop words, etc. but that takes time (despite engines like Solr having built-in lists of these kinds, search engineers often have to curate them further, manually, because their product's domain is so niche) and lots and lots of query-stream analysis (I remember personally doing lots of the latter at Reddit, hehe). TLDR: - I think pure vector search platforms should be evaluated differently than traditional keyword search platforms - I think vector search is a tool -- simply one of many -- that search engineers should use to make their search engine results more relevant. Disclaimer: I'm from Pinecone (and hi, Doug!)
- J_Shelby_J 3y ago
  Speaking of tools, I want to add keyword search to my CEQ project. I’m currently just using pinecone for retrieval, but I’m not always getting results for specific niche technical terms. Do you have any advice?
rsrsrs86 3y ago
Asking as a newbie in database programming. Are vector databases built on top of rd trees?
mish15 3y ago
Yes it's all wrong, because: a) recall is designed to measure binary relevance, but vector scores are not good relevance judgments and they aren't binary. b) most models optimise purely for distance, which makes nDCG look great, but causes content to clump together. This loses local ranking precision and the noise from embedding order is significantly greater than the approximation in the ANN system c) bi-encoders have significantly greater error than cross-encoders. Basically every vector DB is blowing at least one order of magnitude more resources than they need to to optimise bi-encoding efficiency which is wrong anyway. Disclaimer: I work at Algolia.

139 points by softwaredoug 3y ago

quickthrower2 3y ago

Forget RAGs and LLMs, Vim search and a little regex is all you need

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

ankit219 3y ago

Been following this space for a bit. In my view, the literature on the internet is distorted about these things because of the need for certain OSS project creators and (non OSS) companies to promote their own products. It typically goes like this: [Insert technique name] is a good technique, but it can be further improved if you enable [some complex workflow]. [Some example of how it improved a particular case]. We at [Company] already do this at scale and you can use it out of the box if you use it. Go make better things. Nothing against people doing this stuff. It just makes me think how difficult is it to promote open source projects today. Regarding Vector DBs, most have advocated for weaviate and pinecone. Incidentally, all run into quantization problems at scale (1M+ vectors) if the clusters are to be adjusted, and handle it differently. i tried it for a 6000 vector product[1] and got the best results by using a pkl file rather than any of the dbs. [1] Not sharing the product here as such. We just scraped a govt tax website to create a search engine on top of that. Helps as the data is not indexed by google, and we had no interest from customers in wanting to chat. They were interested in original source along w a summary.

Palmik 3y ago
You likely won't see performance (speed) benefits if you have just tens of thousands of vectors and of course brute force (vector matrix multiplication) will have better precision & recall. Vector databases make sense when you have a lot of data or some other requirements (e.g. persistence, online updates, complex filtering, etc) that you would rather not implement yourself.
- Uncroyable 3y ago
  exactly. brute force methods work great for thousands pieces of data of (almost) any kind. for vector data no vector db is needed; for structured data even a simple csv file with a python script will do the job and a sql db is not necessary.
  - brigadier132 3y ago
    The thing is, depending on the dimensions of the vectors your using the growth rate can be pretty drastic. Each openai vector is 5kb and 1500 dimensions. So sure searching 1k vectors, just brute force it. But what if you have 100k vectors and a ton of users hammering at the index searching through it? Each search is now 150,000,000 dot products. Go up one more order of magnitude and you definitely need an index. And I don't think 1 million vectors is that much.
    - ankit219 3y ago
      Number of vectors are determined by (other than original dataset ofc) how you chose to chunk the data you have available. Bigger chunks work better in terms of search (empirically) and they also keep the number of vectors down. For openai, based on prevalant norms and their cookbooks, 1M vectors likely mean 1M (more like 700K) pdf pages of text (at a token size of 1000 per embedding). That is a lot of textual data for a decent size company. Enterprises might reach that stage. Consulting firms definitely would - though they already trained and announced their own models.
      - brigadier132 3y ago
        700k pdf pages is not a lot. Also, you might be a business serving other businesses and indexing their documents and at reasonable (again, not google scale), 700k pages is again not a lot. Another way to look at 700k pages is 2333 300 page books.
- deleted 3y ago
  [deleted]
keyle 3y ago
Bit sad how it's going. People go straight to product and venture scaled growth. Build protocols, not products! I tell people. It's a terrible advice actually, unless you like eating noodles and scraping by. But it's still the right way. We're still waiting for the antirez' Redis version of Vector DBs. Maybe he should do one! :)
- vkb 3y ago
  The good news is that this already exists in the Redis search module [1], which allows you to do similarity search against indexed embeddings, among other features, and offers comparable performance against other ANN libraries [2], depending on your performance criteria. I've been using it for a side project to do semantic search on books[3] and have been really happy with its performance. (Not affiliated with any of this, was mostly interested in exploring existing well-performing, fairly standard tools with low latency) [1] https://redis.io/docs/interact/search-and-query/search/vectors/ https://redis.io/docs/interact/search-and-query/search/vecto... [2] https://ann-benchmarks.com/#redisearch https://ann-benchmarks.com/#redisearch [3] https://viberary.pizza/how https://viberary.pizza/how
  - neeleshs 3y ago
    Bit of a side note on Redis Search. While performance is pretty good, it is not production quality yet, mostly around monitoring indexing errors and quirky query results. Silent indexing errors pretty much make it impossible to use it in production, especially for structured data.
- manishsharan 3y ago
  Redis Enterprise edition does have Vector database.i haven't used it but am curious about other people's experiences. https://redis.com/solutions/use-cases/vector-database/ https://redis.com/solutions/use-cases/vector-database/
gk1 3y ago
> Regarding Vector DBs, most have advocated for weaviate and pinecone. Incidentally, all run into quantization problems at scale (1M+ vectors)... Can you elaborate? We (at Pinecone) have dozens of customers with over a billion embeddings, still serving queries in under 300ms (p95) with index updates in <1s. If you saw performance issues with just 1M embeddings then something else is going on that we may be able to help with.
- ankit219 3y ago
  It's a specific case. Most customers won't run into it. It's to do with quantization or specifically what happens when you add enough vectors that the original clustering changes (use cases could be hit when you onboard a IM/consulting customer which has so much data to alter clusters amongst others). It is not a performance issue to be fair. Just that altering a cluster takes time. I was just testing to see if there is a DB which does not run into such problems, before putting a business case. I am not discouraging the use, just saying that most cases do not need a vector DB. The reason we did not choose one was because our business case's number of vectors was small.
  - jamesbriggs 3y ago
    I may be misunderstanding, but I'll try to answer — quantization typically means retrieval will be slower (if referring to techniques like product quantization), but that is the case whether you're at 10K vectors or 1B vectors, afaik it doesn't really make a difference because you're only quantizing the query vector at query time (it has been awhile since I read anything on quantization, so I could be mistaken). Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index) So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.

redwood 3y ago

This space is thr ultimate in premature optimization and obsession or implementation details (including the classic "it's written in programming language X so it has to be better") with relatively little real world experiential knowledge in the mix

dmezzetti 3y ago

The focus on the top 10 in vector search is a product of wanting to prove value over keyword search. Keyword search is going to miss some conceptual matches. You can try to work around that with tokenization and complex queries with all variations but it's not easy. Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy https://github.com/spotify/annoy) has been around since 2014. It was one of the first open source approximate nearest neighbor libraries. Recommendations have always been a good use case for vector similarity. Recommendations are a natural extension of search and transformers models made building the vectors for natural language possible. To prove the worth of vector search over keyword search, the focus was always on showing how the top N matches include results not possible with keyword search. In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) operates in the sweet spot. Once again this is validated through the same benchmarks which focus on the top 10. On top of all this, there is also the reality that the vector database space is very crowded and some want to use their performance benchmarks for marketing. Disclaimer: I am the author of txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai), an open source embeddings database

marginalia_nu 3y ago
Amusingly, vector search is conceptually nearly as old as keyword search. Most older IR textbooks promote the idea of viewing keyword search as a maximization problem of the weighted bit vector product between the query and the document, where the weight matrix is something like BM-25.
- PaulHoule 3y ago
  I was interested in dense vector search in 2004 and a bit depressed at how the indexing algorithms weren’t that good, Around 2013 I was involved with an actual product which was a search engine for patents. The thing about vector search today is that it is an absolute gold rush, Pinecone had the right idea but the wrong product (sorry SAAS is toxic), many of the latecomers are… late, and will find the whole startup and VC process will cost precious months when customers want to build products right now. Note for that patent search engine we had a quite a different idea than is fashionable today, that is, like very old TREC we care about the quality of results you got 1000 or 2000 results in in the assumption that a patent researcher wanted to be comprehensive. When I was first into IR it was in the context of “how is Google so much better than other search engines?” and one part of it was that based on the way search engines were being evaluated at TREC, Google was not a better search engine* because it was so focused on the first page. It has come full circle now, I have talked to people recently who complain that current TREC evaluations are too focused on the first few results.

Andrew018 3y ago

It's possible that our current perspective on vector storage could be flawed or incomplete, and exploring alternative approaches may yield valuable insights and improvements in data management and computation.

jamesliaamm 3y ago

[flagged]

9dev 3y ago

Anyone have insights on using Elasticsearch as a vector database as opposed to a specialised vector DB? We have a complex keyword search already, and would like to introduce knn query ability to find similar documents. ES offers that in recent versions, but I’m wondering whether that’s a good idea.

softwaredoug 3y ago
Jimmy Lin, a researcher a U Waterloo, recently published "Lucene is All You Need" https://arxiv.org/abs/2308.14963 https://arxiv.org/abs/2308.14963 I know, FWIW, Elasticsearch has been investing a lot in this space lately. It's not perfect, but its bound to get better.
- peterstjohn 3y ago
  That paper does a terrible job of making Lucene look useful, though. 10qps from a server with 1TB of RAM is not great (and I know Lucene HNSW can perform better than that in the real world, so I am somewhat mystified that this paper is being pushed by the community).
jillesvangurp 3y ago
I looked at this a few months ago. Elasticsearch has some limitations with respect to the vector length (1024, I think); which rules out using a lot of the popular off the shelf models. A key insight is that the performance of off the shelf models isn't great compared to a hand tuned query and bm25 (the Lucene ranking algorithm). I've seen multiple people make the point that the built in ranking is pretty hard to beat unless you specialize your models to your use case. A key consideration for the vector size limitation is that storing and working with large amounts of huge vectors gets expensive quickly. Simply storing lots of huge embeddings can take up a lot of space. And of course using knn with huge result sets is very expensive, especially if the vectors are large. Having the ability to filter down the result set with a regular query and then ranking the candidates with a vector query helps keep searches responsive and cost low. If you are interested in this, you might want to look at Opensearch as well. They implemented vector search independently from Elasticsearch. Their implementation supports a few additional vector storage options and querying options via native libraries. I haven't used any of that extensively but it looks interesting.

pedrovhb 3y ago

I'm not sure I understand. Isn't vector storage about storage and retrieval strictly, being a separate thing from the embedding generation itself? My mental model is that the job of vector storage is to just fetch the closest vectors to a target vector for some proximity metric, with the model (or method , more generally) that generates embeddings being its own distinct problem. You may later do things like passage re-ranking and maximal marginal relevance to refine it, but again, different task. I understand vector search against tons of other vectors is an approximation endeavor due to practical concerns, so it doesn't seem strange for vector store accuracy to be measured in terms of top-k retrieval. Post processing and the quality of the vectors themselves is a different thing to judge.

Double_a_92 3y ago

For starters I don't believe that people even understand or agree what "vector" even means in this context.

fodkodrasz 3y ago
Yes, the word is a bit overloaded... I guess from skimming through the article that here it is vector in the sense of word2vec algorithms.
simonw 3y ago
A great deal of the current excitement around vector search is related to embeddings. I tried to provide a very clear explanation of embeddings here, since I know lots of people haven't yet learned what they are or why they are interesting: https://simonwillison.net/2023/Sep/4/llm-embeddings/ https://simonwillison.net/2023/Sep/4/llm-embeddings/

Androider 3y ago

Remember the "Mongo DB is web scale" craze from now over a decade ago? That's the current state of vector DBs, despite vector DBs having been around forever. Do you have more than a terabyte of embeddings? No? Do yourself a favor and use pg_vector with a HNSW index, it'll do just fine. Operationally, it is very hard to beat Postgres.

drittich 3y ago
And if you just have tens of thousands, straight up SQL can work pretty well, too.
- pornel 3y ago
  If you have tens of thousands, a `for` loop over an array in memory is plenty fast. I've worked on a project that searched through 50 million vectors with linear brute-force search, just with a sprinkle of AVX (and it used the brute-force search, because the curse of dimensionality killed all the smarter approaches).
osigurdson 3y ago
An OpenAI embeddings vector is 1536 4 byte floats. 1 TiB is roughly 174K such embeddings vectors.
- brigadier132 3y ago
  Your math is wrong. 100k 32 bit vectors is 600mb I think your point is right though. Searching through these is requires an index of some sort at any reasonable scale (not google scale).
  - osigurdson 3y ago
    Hmmm, yeah, I used 2^30 instead of 2^40. Should not comment before caffeinating.

osigurdson 3y ago

One thing I have been wondering about is how, concretely, in practice embeddings will be used. Will companies create embeddings for most / all content that they have? Will they create an embedding for every sentence, paragraph or page of text (or all of those). Will they store hierarchies of embeddings? Do they then store the original text so they can invert the process (that seems obvious)?

WinLychee 3y ago
User/Item/Query embeddings are the most common. That way you can generate per-user recommendations, or search results for a given query (with personalization using side information). Video will be interesting, once we have video embeddings (maybe this exists already). It depends on the use-case but a few of your ideas are certainly possible. Generally I've seen them at a coarse rather than fine level, but I'm sure that's out there too. This looks like a good overview if you want to read about it: https://recsysml.substack.com/p/two-tower-models-for-retrieval-of https://recsysml.substack.com/p/two-tower-models-for-retriev...

chaxor 3y ago

Wow! Not the point of the article, but it is striking to see how mastodon is way better in terms of usability/aesthetic and general UX than Twitter or 'x'. I've been mildly interested in mastodon for quite some time from a political perspective (remove large support of big companies where possible), but I didn't realize it was just sooo much better to use.

sp332 3y ago
And there are tons of third party clients. I think Tusky is the best one I’ve seen for Android, and there’s an interesting web-based one called Elk that’s very nice. You load up https://elk.zone https://elk.zone and then use it as a front-end to sign in to your server.
- jameshart 3y ago
  Also the mastodon web client installs great as a PWA. Just add your home server as a shortcut to your home screen - no client install needed.
softwaredoug 3y ago
I really enjoy Mastodon and believe it has staying power. Not rocket ship “going to explode and burn out” power. There’s enough there to keep people engaged. The decentralized nature means there’s always corners of it alive and growing.

rvrs 3y ago

This is tangential, but another issue I've had with existing vector databases is write throughput. Weaviate and Pinecone build an index immediately after the creation of a "table" (or rather, the analogue thereof) and this makes write throughput abysmal at times. We have around 2 million 768-length embeddings + metadata -- writing too fast to Weaviate quickly results in OOMs. We circumvented this by very slowly trickling our data through to Weaviate, but this causes many-hours-long writes. I'm experimenting with dumping our data into postgres and building an hnsw index with pgvector once all writes are done. This has reduced our time to maybe 2 hours at most (the bulk of that being index building time, of course).

rectang 3y ago
I once solved worst-case write performance in an open source MVCC data store (Apache Lucy, a “loose C” port of Lucene) like so: 1) Limit the maximum size of an update. This causes gradual degradation of the index, which is ordinarily dealt with through periodic consolidation — but now this this consolidation will be disabled in the main write mechanism. 2) Enable a background consolidation process which runs concurrently. At the end of the consolidation process a short write lock will need to be obtained while you replay all deletions that have occurred since consolidation started, but the time required isn’t problematic. This fix required major surgery to the Lucy indexer core code, so it’s something that that would only be feasible if the data store itself supported it.
aslPine 3y ago
@rvrs, Have you tried Pinecone's S1 pods? Curious which indexing type you went with when experimenting (https://docs.pinecone.io/docs/indexes https://docs.pinecone.io/docs/indexes) Disclaimer: I'm from Pinecone, hello!
mish15 3y ago
Yeah basically all the vector "database" solutions in market have chosen data-dependent indexes, so you need the data upfront. Imagine if regular databases needed all data upfront before they could build indexes. It's kind of crazy...
trengrj 3y ago

intalentive 3y ago

1. Check out this cool chat-with-my-docs app I made with LangChain! 2. Okay, let's productize! 3. Ouch, the results aren't quite production-ready... 4. Huh, the bottleneck seems to be the search & retrieval process... 5. Oh, I guess I don't know anything about search, time to catch up on the field... 6. Ok, now I need to build an inverse index or knowledge graph and augment it with embeddings. 7. Hm, actually, language models (including BERT-based, Splade) could be pretty useful in building structure out of my dataset... IMO the best way to incorporate transformers into search has yet to be discovered, although whatever OpenEvidence.com is doing seems to work well. If I were still in grad school I would look into some kind of PageRank / graph NN approach to the problem, or maybe hyperbolic embeddings. Reason being that for many corpora there exists some kind of internal structure or linking that can be exploited for speed and accuracy, which you don't quite get from nearest neighbors in a Euclidean space.

yujian 3y ago

Who's focusing on top 10 recall? I recently saw someone ask if we (I work at Zilliz) can update Milvus' recall to 10 million nearest neighbors lol

hintymad 3y ago

Eh... Because keyword and BM25 do not work that well, especially for enterprise search? Many of us surely complained how bad enterprise search was, right? A company usually has a hodgepodge of documents. They don't link to each other that much so page rank does not work well. They have fewer than 1 click per day, if there is any at all. So signals based on user interactions will not be useful or will be expensive to build. We often search with keywords that show up in enough number of documents but not enough to make BM25 rank the matches high. Query understanding hardly exists in enterprise search, because building query autocompletion, query correction, query reformulation and etc all requires some level of semantic understanding. If it's not embedding based, it has to be stats-based, which requires enough volume of clicks and user evaluation that enterprise search can't afford. The list can go on and on. There is a reason that Google's search appliance never took off, and people are still complaining about quality of enterprise search. Amazingly, a true semantic search would work well. And to do semantic search in scale, you'll need a vector database.

aaroninsf 3y ago
I work in an adjacent area, and this is the most useful and information-dense summary I've read in any forum in at least a week. Word!
code_biologist 3y ago
Dear god this is accurate. A product I worked on a few years ago wanted to add a feature searching various PDFs that were a slurry of the same words. After being disappointed with the results of keyword search, we deemed semantic search too complicated to implement and scrapped the feature. Great to see it's getting easier.

aslPine 3y ago

Do we "obsess over perfect accuracy in top 10 for vector dbs"? I don't see a ton of data that points towards us (vector DB ppl) building towards traditional TREC/BEIR #s. I do see lots of us in the vector DB space demonstrating on real data the speed and semantic retrieval capabilities of vector DBs. Not many of us publish traditional IR benchmark #s simply because vector search is a pretty different ballgame than traditional keyword/BM25 search (as you know). However, since most search practitioners have developed their mental models around things like TFIDF and BM25 + Learning To Rank, many vector DBs tailor their language to fit that heuristic. If you look at vector DBs on their own -- I like to look at them as a piece of infra to use in conjunction with pre/post processing + RAG + BM25 + whatever else you want -- you'll find that measuring their accuracy against traditional IR benchmarks is a bit like fitting a square peg into a round hole. It almost fits! But not quite. I think one of the starkest differences between the vector search and traditional search when it comes to retrieval evaluation is the great difficulty involved in creating judgment lists to measure vector search performance. Since vector search inherently retrieves documents that are approximately conceptually similar to a query, the initial recall set, especially with an IRL dataset (e.g. Reddit, Shopify, etc.) is a moving target. Language evolves. Idiosyncrasies pop up. Etc. Not to say that those ^ issues don't plague keyword search as well. But the powerful thing about vector databases IMO is that unlike traditional indices that need lots of fine tuning and IR experts, a vector DB will capture these new additions to your domain's vocabulary immediately, based on the surrounding context in the document. Of course you can do similar things with traditional search engines by incorporating curated lists of synonyms, antonyms, stop words, etc. but that takes time (despite engines like Solr having built-in lists of these kinds, search engineers often have to curate them further, manually, because their product's domain is so niche) and lots and lots of query-stream analysis (I remember personally doing lots of the latter at Reddit, hehe). TLDR: - I think pure vector search platforms should be evaluated differently than traditional keyword search platforms - I think vector search is a tool -- simply one of many -- that search engineers should use to make their search engine results more relevant. Disclaimer: I'm from Pinecone (and hi, Doug!)

J_Shelby_J 3y ago
Speaking of tools, I want to add keyword search to my CEQ project. I’m currently just using pinecone for retrieval, but I’m not always getting results for specific niche technical terms. Do you have any advice?

rsrsrs86 3y ago

Asking as a newbie in database programming. Are vector databases built on top of rd trees?

mish15 3y ago

Yes it's all wrong, because: a) recall is designed to measure binary relevance, but vector scores are not good relevance judgments and they aren't binary. b) most models optimise purely for distance, which makes nDCG look great, but causes content to clump together. This loses local ranking precision and the noise from embedding order is significantly greater than the approximation in the ANN system c) bi-encoders have significantly greater error than cross-encoders. Basically every vector DB is blowing at least one order of magnitude more resources than they need to to optimise bi-encoding efficiency which is wrong anyway. Disclaimer: I work at Algolia.