5 ms·

The article says: "We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ran

The article says: "We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection......This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure." Curious why stop there, why even use OpenAI embeddings and not use, say, LLaMA embeddings and create a truly open stack.

PheonixPharts 3y ago
I'm guessing they choose to use OpenAI embeddings since this is the dominant use case for most people. The embeddings are just the "data" that's in the database. Swapping out getting embeddings from OpenAI with Llama is as trivial as putting information about your own customers in your own database as opposed to using info on someone else's customers.
spullara 3y ago
This is completely obvious and I am surprised that they had to write a paper about it but with $100m investments in vector databases I guess it needed to be officially said. For those companies to be successful they will have to also become either a better search engine or a better database in addition to vector search and compete either with folks like elastic/redis/opensearch/vespa or with postgres/mongodb/oracle/mysql. An independent vector only search system doesn't make sense. Also, all embeddings are basically equivalent for this use case.
- TuringNYC 3y ago
  >> but with $100m investments in vector databases I also do not understand why VCs are investing in this space. The base case is that they are almost completely interchangeable c/o langchain and other intermediaries. I do understand that some vector databases have strengths over others w/r/t scaling OUT, however, they do not have stickiness and time erodes the scaling advantages via both competitors catching up and via machines getting cheaper allow for scaling UP The history of commercial DBs was usually supported by a variety of use cases, proprietary hooks to keep customers, choices on the CAP theorem, etc. Almost none of that applies here given the minimal interaction modes we have with vector DBs. Could anyone speak to the case to invest in vector DBs? (and to address the parent take, "elastic/redis/opensearch/vespa or with postgres/mongodb/oracle/mysql" - this is one of the most crowded spaces in the marketplace and i have no idea why customers would choose an upstart for the sake of consolidation rather than clear winners in a best of breed solution.
- beachy 3y ago
  History repeats itself endlessly in the database world. Late last century Oracle felt urgency to compete with the new hotness back then, object databases. The guys in charge of the Oracle database itself pushed back, not too much was done, and in the end those competitors all flamed and died. Object databases gave way to graph databases. They broke through a little but RDBMS continued to rule. Then came the NoSQL movement. RDBMS vendors ended up adding json columns and the pure NoSQL vendors flamed and died (excepting Mongo DB which is web scale (https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs)). Thus it will continue forever.
- ofermend 3y ago
  Yes, plus we certainly have too many independent vector DB products as it is.
BoorishBears 3y ago
LLaMA embeddings would perform terribly. MS MARCO is a well known benchmark that mirrors a pretty common use case: https://www.sbert.net/docs/pretrained-models/msmarco-v3.html https://www.sbert.net/docs/pretrained-models/msmarco-v3.html https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard To intentionally oversimplify, embeddings fall into 2 main categories if you do a cursory search: - Good at semantic similarity: Most common case, check if two strings have similar meaning, even if the words don't match exactly. - Good at Q/A: Finds text that can answer a question. Sound similar to semantic, but "What is a dog" and "What is a cat" are very similar sentences but very different questions. These models cluster questions closer to their answers and further from other questions. They also handle the difference in length between questions and answers better. (Some leading embedding models use prefixes during training to let you adjust performance between those two tasks on the fly) LLaMA embeddings weren't optimized to any specific task, you'd just be hoping that they tangentially align with some arbitrary goal. OpenAI had to fine tune their embeddings model, and despite being massively oversized you can see it's not at the top of the leaderboard compared to much smaller models.

PheonixPharts 3y ago

I'm guessing they choose to use OpenAI embeddings since this is the dominant use case for most people. The embeddings are just the "data" that's in the database. Swapping out getting embeddings from OpenAI with Llama is as trivial as putting information about your own customers in your own database as opposed to using info on someone else's customers.

spullara 3y ago

This is completely obvious and I am surprised that they had to write a paper about it but with $100m investments in vector databases I guess it needed to be officially said. For those companies to be successful they will have to also become either a better search engine or a better database in addition to vector search and compete either with folks like elastic/redis/opensearch/vespa or with postgres/mongodb/oracle/mysql. An independent vector only search system doesn't make sense. Also, all embeddings are basically equivalent for this use case.

TuringNYC 3y ago
>> but with $100m investments in vector databases I also do not understand why VCs are investing in this space. The base case is that they are almost completely interchangeable c/o langchain and other intermediaries. I do understand that some vector databases have strengths over others w/r/t scaling OUT, however, they do not have stickiness and time erodes the scaling advantages via both competitors catching up and via machines getting cheaper allow for scaling UP The history of commercial DBs was usually supported by a variety of use cases, proprietary hooks to keep customers, choices on the CAP theorem, etc. Almost none of that applies here given the minimal interaction modes we have with vector DBs. Could anyone speak to the case to invest in vector DBs? (and to address the parent take, "elastic/redis/opensearch/vespa or with postgres/mongodb/oracle/mysql" - this is one of the most crowded spaces in the marketplace and i have no idea why customers would choose an upstart for the sake of consolidation rather than clear winners in a best of breed solution.
beachy 3y ago
History repeats itself endlessly in the database world. Late last century Oracle felt urgency to compete with the new hotness back then, object databases. The guys in charge of the Oracle database itself pushed back, not too much was done, and in the end those competitors all flamed and died. Object databases gave way to graph databases. They broke through a little but RDBMS continued to rule. Then came the NoSQL movement. RDBMS vendors ended up adding json columns and the pure NoSQL vendors flamed and died (excepting Mongo DB which is web scale (https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs)). Thus it will continue forever.
ofermend 3y ago
Yes, plus we certainly have too many independent vector DB products as it is.

BoorishBears 3y ago

LLaMA embeddings would perform terribly. MS MARCO is a well known benchmark that mirrors a pretty common use case: https://www.sbert.net/docs/pretrained-models/msmarco-v3.html https://www.sbert.net/docs/pretrained-models/msmarco-v3.html https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard To intentionally oversimplify, embeddings fall into 2 main categories if you do a cursory search: - Good at semantic similarity: Most common case, check if two strings have similar meaning, even if the words don't match exactly. - Good at Q/A: Finds text that can answer a question. Sound similar to semantic, but "What is a dog" and "What is a cat" are very similar sentences but very different questions. These models cluster questions closer to their answers and further from other questions. They also handle the difference in length between questions and answers better. (Some leading embedding models use prefixes during training to let you adjust performance between those two tasks on the fly) LLaMA embeddings weren't optimized to any specific task, you'd just be hoping that they tangentially align with some arbitrary goal. OpenAI had to fine tune their embeddings model, and despite being massively oversized you can see it's not at the top of the leaderboard compared to much smaller models.