8 ms·

Are we at peak vector database?

235 points by softwaredoug 3y ago

danielmarkbruce 3y ago
We are past it, it was months ago :) The points are basically right, but a lot of folks realize all this.
charcircuit 3y ago
>Real-time recommendations, but driven by vector (and other kinds of) retrieval that looks more like a search engine - not batch computed, nightly jobs common these days. This is already the case. Recommendations are just a fancy search where the query is a vector representing the user. Whether the learning is batched or not doesn't change the fact that it will use vector search for at least candidate generation.
dmezzetti 3y ago
I'll add txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) to the list. There is still plenty of room for innovation in this space. Just need to focus on the right projects that are innovating and not the ones (re)working on problems solved in 2020/2021.
- zeroCalories 3y ago
  I agree. Honestly even the fundamentals of vector databases aren't really "solved" in the way they are for other databases. Vector indexing, embedding generation, horizontal scaling, etc. can probably still improve a lot. And don't forget, even if Postgres and MySQL are the only traditional databases in town, every tech company had their own SQL database once. Many of them are still around too. No need to get pissy about these companies.
  - dmezzetti 3y ago
    Agreed. For example, here is a post about integrating vector search results with semantic graphs for RAG - https://news.ycombinator.com/item?id=39141420 https://news.ycombinator.com/item?id=39141420 And here's a post on an alternative way to integrate vectors with traditional databases (Postgres, MySQL) - https://neuml.hashnode.dev/external-database-integration https://neuml.hashnode.dev/external-database-integration As others have said in this thread, cosine similarity on arrays of vectors isn't novel. But there are many possibilities past that, many we haven't thought of yet too.
monero-xmr 3y ago
Someday enterprises will actually pay someone for LLM tech and infra. Someday…
- esafak 3y ago
  Lots of companies are paying OpenAI, so someday is yesterday? https://www.reuters.com/technology/openai-annualized-revenue-tops-16-billion-information-2023-12-30/ https://www.reuters.com/technology/openai-annualized-revenue...
  - monero-xmr 3y ago
    Most of that is recycled. They don’t break it down because it would make the obvious, obvious. Microsoft pays OpenAI but requires them to use Azure, and OpenAI pays Microsoft the same money back. This is why they continually need billions in investment, because they are far from profitable. The same principle applies to defense. The US gives Israel and Ukraine tens of billions, but that’s a credit to buy from US defense firms. That money gets recycled right back to US weaponry.
    - infecto 3y ago
      Your logic seems highly flawed. I get what you are saying in the example, yes the government provides weapons which are paid for by the government but produced by defense companies. But in the Microsoft example it is customers who are paying Microsoft to use OpenAI via Azure. Thats a free market of money inflows. Same with all the people using OpenAI directly. Not sure how you would even think of the money being recycled in this scenario. Yes of course there is some back scratching in the sense that Microsoft invested in OpenAI with a large portion of that investment in Azure credits which makes the investment quite nice from MSFT's side but there is still real demand for Azure services to use OpenAI apis.
      - monero-xmr 3y ago
        For big enterprises it is either included in existing licenses (i.e. Microsoft Word has ChatGPT embedded) or pilots. No one is cutting massive checks to Microsoft specifically for OpenAI services. This is obvious, but if you need some journalist to validate what is already logically clear: https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-early-products-a-hard-sell-bdd29b9f https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-ea... https://www.wsj.com/tech/ai/ai-deals-microsoft-google-amazon-7f624054 https://www.wsj.com/tech/ai/ai-deals-microsoft-google-amazon...
LispSporks22 3y ago
I believe you've reached peak anything when it's been incorporated into PostgreSQL.
- spootze 3y ago
  pgvector has you covered: https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
  - hot_gril 3y ago
    Wow. Back in the day, I had to do cosine similarity indexing with pg-cube. It only did euclidean distance, so I had to store a separate column with normalized vectors.
    - BiteCode_dev 3y ago
      What's cosine similarity, what do you use it for and why is it good to have it into your db instead of somewhere else (like a lib)?
      - pxndxx 3y ago
        Euclidean distance stops "making sense" as the number of dimensions goes up: https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions https://stats.stackexchange.com/questions/99171/why-is-eucli... Cosine similarity measures the angle between two vectors instead, and doesn't suffer from the curse of dimensionality. I guess it's important to have this in your DB, so you make "nearby" queries (give me text that's similar to this other text) in an efficient way.
        hot_gril 3y ago
        Exactly. Some other context implied here, these vectors were ML embeddings. In that case, like 100 dimensions that vaguely represented our input data in compact form. There was probably a better solution out there, this was just the most readily available for us.
dbCoop3r 3y ago
Why not a "vector filesystem" for Linux? I know it's subjective, but databases have started to feel like running a window manager and desktop on a server. I feel like software needs to take a step back and rethink itself after years of putting chimps at typewriters searching for Shakespeare. Why not a Linux kernel with a module(s) to provide the same assurances, SQL operations? Write directly to the filesystem? Why is all the mathematical concept that we derive software from packaged into endless conceptual blobs of black box state?
feverzsj 3y ago
Investors don't really care if it actually creates more value. They only care if the story can attract the public. They just want to profit by taking next investors' money.
lumost 3y ago
So, there have been gnaw plugins for elastic search for some time. Is there really a need for a new search platform? If there is, what api differentiates it, and why can’t this be expressed in either elasticsearch or Postgres?
reissbaker 3y ago
IMO we are well past peak cosine-similarity-search as a service. Most people I talk to in the space don't bother using specialized vector DBs for that. I think there's space for a much more interesting product that is longer-lived (since it's harder to implement than just cosine-similarity-search on vectors), which is: 1. Fine-tuning OSS embedding models on your real-world query patterns 2. Storing and recomputing embeddings for your data as you update the fine-tuned models. MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I'd bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else's: code retrieval vs document Q&A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system). And if you're fine-tuning new models regularly, you need storage + management, since you'll need to recompute the embeddings every time you deploy a new model. I would pay for a service that made (1) and (2) easy.
- latchkey 3y ago
  Question, does this require specialized hardware at all? GPUs?
  - ipsum2 3y ago
    It doesn't require it in theory, but in practice its required bc CPUs are too slow at fine-tuning and computing embeddings.
- phreeza 3y ago
  Are you aware of any service or OSS solution for this?
- deleted 3y ago
  [deleted]
- jn2clark 3y ago
mirekrusin 3y ago
We're at the peak of blog posts listing vector databases.
visarga 3y ago
Embeddings are good at capturing surface level information but can't match implicit/deeper/conclusion level information. Say you have a collection of 100,000 math problems, and you want to embed them to search problems that give result "0". Any number of problems can give this result and it is not explicit in the problem statement. But if you solve the problems you can see the data was in there, just not apparent. In general you can see the raw text as a simulation premise that will generate inferences when "executed". The inferenced part is like the hidden part of the iceberg, you don't see it but it is there, implicit in the source text. Not just in math, but in all fields. Embeddings are only good at superficial retrieval. The text needs to be fully analyzed with LLMs before embedding. Thus my conclusion is that we still have a long way to go, we haven't peaked.
- deleted 3y ago
  [deleted]
- wodenokoto 3y ago
  What do you mean by fully analyzed? It’s the LLM that does the embedding.
  - visarga 3y ago
    Oh the embedding LLMs are usually lightweight BERT models with few layers and <<1B weights, while LLMs are easily 10-100x larger. The idea is to ingest the text in a LLM to extract the facets you are going to search and add those extra tokens to the original text. Then you do regular RAG.
    - sroussey 3y ago
      How?
    - wodenokoto 3y ago
      You want to enrich the base text before embedding? With what kind of prompt? It’s a pretty simple thing to add to a pipeline. Have you tried?
- 3y ago
deleted 3y ago
[deleted]
physicsguy 3y ago
> In the same way NoSQL forced us to rethink databases. Did it? After using Mongo in my current job (not my choice), I'd choose Postgres again for my next project.
- cloverich 3y ago
  The thing to know about Mongo is, every database involves design choices that balance ergonomics, performance, and reliability. Every one, except Mongo which, according to their sales team, is the best at everything and has no faults, unless your technical choices are incorrect. In fact I just learned (in a lunch and learn with their team) that when you de-normalize data, inconsistency issues aren't really a problem, and joins are so unusably slow in ALL use cases anyways. Went ahead and just threw my DDIA book in the trash, as they nodded approvingly.
ig1 3y ago
I think https://vespa.ai/ https://vespa.ai/ has the right approach in this space by focusing on being hybrid - vectors alone aren't great for production use cases, it's the combining of vectors+text that lets you use ranking to get meaningful result. (I'm an investor so I'm biased; but it's also the reason why I invested)
zcw100 3y ago
This is a ridiculous rant. “ oh no! We have choices”. Then you list out every choice available for what is a new space people are exploring and the list is barely a half dozen long? It’s more like this is peak “claiming everything is peak”.
- dimatura 3y ago
  Yeah, I'm happy there's a lot of development in this area - even if it's fueled by the LLM frenzy, good nearest neighbor search solutions are useful in a lot of domains. Though I worked a little bit on this problem over 10 years ago (with an application to visual SLAM), and it is a bit amusing to see that a lot of the ideas and even the libraries are still the same!
- softwaredoug 3y ago
  Author here, well yeah, I agree its probably ridiculous. Sort of testing the waters to see if I'm way off base. I think what I mean to say is that, in my experience, practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve. In fact, that's a teeny tiny part of it. Hence "peak vector DB". So I think the market needs some education that its harder than that. That part is my rant :). I've spoken / worked on enough problems now to see that disconnect between market and reality. Though I think "vector DB" is actually a place for capital/brainpower to concentrate to solve these other problems. And I think we'll see the vector DB vendors pivot there. It's just taking a while for the market and investors to see this...
  - brookst 3y ago
    It sounds like you’ve conflated “gold rush” with “peak”. All sorts of novel technologies had mad rushes when they’re new, but that does not mean they have peaked. The dot bomb era with its ridiculous overvalued useless startups was a gold rush, but it was in no way peak Internet.
  - phillipcarter 3y ago
    > practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve I agree, and as one who does exactly and only this on the search side, it's also something that falls flat on its face if you don't think a little more about the data and tasks involved. I wrote about it here[0], but the gist of it for our use case is that if we don't intentionally include what may be considered "less relevant" data then we stand a good chance at failing our main generative task. [0]: https://phillipcarter.dev/2024/01/15/three-properties-of-data-for-making-llms-awesome/ https://phillipcarter.dev/2024/01/15/three-properties-of-dat...
wahnfrieden 3y ago
Is there a good choice available for running inside a browser, client-side? Without a server to create or run inferences
FrustratedMonky 3y ago
""“how can so many vector databases need to exist?”."" Same with languages. Why so many languages. Why can't we all get behind a few, do we need more than 6? For every case/problem? Put all our combined resources towards a smaller set. We need a few DB's, a few languages, a few frameworks. Do we need hundreds? Like everyone rolls their own everything.
ryoshu 3y ago
Don't confuse a feature with a product. Postgres works great and you can layer in cosine similarity along with full-text search in a single query if you need to.
crowcroft 3y ago
The big LLM companies are well positioned to build a lot of what a vector database is used for into their existing APIs and offerings. Both simplifying DX and devops. Then on the other side existing databases will want to add functionality to be used as vector databases as well. I think there’s lots of innovation ahead and it’s too soon to know what the end outcome will be.
macspoofing 3y ago
Not yet. There is excitement for vector databases in some specialized areas but it hasn't really filtered out to the wider rank-and-file software engineering circles. You know it will be 'peak vector database' when you'll see blog posts on migrating your relational data to a vector database (with a follow-up 2 years later about moving back to PostgreSQL due to the shitshow that ensued).
lpgauth 3y ago
"We would say Cassandra is a columnar data store, alongside the Scylla or HBase." Cassandra and Scylla are row based distributed key value stores.
sergiotapia 3y ago
I believe the next step is an "Algolia" of sorts for cosine-similarity search. Why bother with chunking data, synching it, and then tagging metadata to it. DB providers should be smart enough to optimize chunking strategy for the kind of content being indexed and then provide a simple API endpoint to query against their data. "RAG in a can".
- eskibars 3y ago
  That's exactly what Vectara is (full disclosure, I work there)
jhj 3y ago
Speaking as an author of one of the primary libraries for doing this stuff (faiss), it is not because it is still an open ended research problem on how approximate high-dimensional dense or sparse nearest neighbor should work, let alone maximum inner product search where the research story is even worse, or other non-metric space similarity measures. All of the current techniques still have quite unacceptable tradeoffs involved. While traditional database indexing is also still an open-ended research problem (e.g., read amplification/write amplification tradeoffs and the like), it produces exact solutions. That isn't the case at all for vector indexing beyond brute-force search, or exact indexing like k-D/BSP trees which don't work well in high dimensions due to the curse of dimensionality.
- jbellis 3y ago
  Why is the research story for MIPS even worse than for ANN?
  - jhj 3y ago
    There is no good geometry to be exploited, and the query vectors might be (and are usually) distributed quite differently than the indexed vectors. For Euclidean (L2) distance indexes where the vectors are partitioned based on geometry (e.g., pretty much every indexing type, including cell-probe like IVF, most forms of LSH, or graph based indices), query vectors can be naturally associated geometrically with candidate nearest neighbor vectors, so the distribution of queries doesn't matter as much. For inner product, it's hard to do much better than spherical clustering (what one would usually do for cosine similarity, which is to project all vectors to the surface of a unit hypersphere, and searching for nearest neighbors via cosine similarity is exactly equivalent to L2 search). But, in general the maximum inner product in the indexed set may lie nowhere near to the projection of the query vector onto the surface of the hypersphere. The maximum inner product for a query vector might be almost nearly perpendicular to the query vector (e.g., a very, very far out and almost perpendicular) versus a vector that is parallel to the query vector but with tiny norm. In two dimensions, an example could be (1, 0) as a query vector, but (1, 10^6) as a database vector (or vice versa). The inner product is 1 but the two vectors are very far apart in Euclidean distance. If you project the vectors to the unit 1-sphere, the query vector is still (1, 0) but the database vector now becomes (1 / sqrt(10^12 + 1), 10^6 / sqrt(10^12 + 1)) ~= (0.000000999..., 0.99999...) (apologies if there's an error here) which would also be in a very different cell if one were using a graph-based or IVF partitioning. Neural search techniques do show some promise here though (say, using a neural net to predict which vector buckets to look at).
agonz253 3y ago
Are there no distinguishing features between these vector databases? I'm not familiar with them so I was looking for any comment on that in the article, whether some make different tradeoffs than others, are easier to operate or implement, more scalable, etc. That together with their relative novelty might help explain why there are so many.
EGreg 3y ago
I don't know, let's ask an AI about it :)
throwaway201606 3y ago
Hill I am willing to die on: Peak "$XXXXXXXX" database is when your particular flavor of DB is completely consumed into traditional RDBMSes. Vector databases (and all other incremental or transformational improvements) are just features of regular plain traditional RDBMSes that have not been implemented in traditional RDBMSes yet. I have seen every new DB tech subsumed by traditional databases over time as compute capability improved. No exceptions. The list is endless: - object databases (e.g. blobs, JSON) - OLAP - in DB programming ( XX-SQL eg PL/SQL, T-SQL, ANSI-SQL) - column-oriented data stores - key-value - graph databases - No SQL - Cloud, distributed, whatever - statistical analysis databases - document databases All these used to be standalone, very expensive, specialty products but are now just one more checkbox on the Oracles/SQL-Servers/DB2s of this world. All these have been swallowed by the borg of commercial databases without so much as a burp. There is no winning the commercial market long term for these products. Big business buys traditional RDBMSes because they are the kitchen sink. They do EVERYTHING and they will eventually do this new hot thing, the business will just have to pay big dollars for it. Which is not a problem for big business. There is a reason that cartoon about the Oracle org hierarchy was made (bottom right): all the company does is make product (Engineering) and protect that product. And it is very good at making good product. https://i0.wp.com/stratechery.com/wp-content/uploads/2013/07/2011.06.27_organizational_charts.png?resize=646%2C630&ssl=1 https://i0.wp.com/stratechery.com/wp-content/uploads/2013/07...
- senderista 3y ago
  Exactly my take, I see no moat here. If there were a way to short the vector DB startup phenomenon and I had the resources I would do it.
  - thraxil 3y ago
    Literally. We do a lot of vector DB and RAG stuff (who isn't these days, right?) and after a bunch of testing and benchmarking went with pgvector integrated into our existing PostgreSQL database. Operationally simple, performs perfectly adequately. I'm sure there are some niche use-cases where the dedicated vector DBs make sense, but for anyone just getting into it, don't underestimate PostgreSQL and pgvector.
hiddencost 3y ago
Nope
PaulHoule 3y ago
I got interested in vector search around 2004, read a lot of papers about vector search algorithms and was not really impressed with the tradeoffs involved (it's not the clear win that B-Trees are for 1-d indexing) and wound up using full scans unless I had sparse vectors. When Pinecone came out and started blogging heavily it seemed that they'd read the same papers I did but came to the conclusion the glass was half full instead of half empty. I could have missed it but I haven't see anything in the literature that's a huge improvement over 20 year old algos. Circa 2014 I worked on a search engine for patents and related literature that made vectors for 20 million + documents and they decided to use full scan and (i) it performed so well (in terms of accuracy) that we sold a license to the USPTO on day two after we put up the demo, and (ii) there were a lot of things about it that were slow like the build system, index building and model training but vector search wasn't one of them. My YOShInOn RSS reader has about a million documents in 2024 and it uses vectors for classification and clustering. Using vectors for search is a clear extension and I've done some prototyping of searches with full-scan and performance is "good enough" (full scan has 'mechanical sympathy'.) I'd probably stuff my vectors into FAISS if I wanted to do anything more and forget about it. Sending my vectors to some cloud service so they can pay AWS prices to store them? That's for the birds. I respect Pinecone for being early to the party but I think those who jumped in in 2022 were laggards.
- kwillets 3y ago
  My experience comes from around the same time frame -- I spent about a year on an aborted spectral dimension reduction project, and I only recently realized how similar the problem still is today. I'm not sure if that makes me more or less qualified to do vector DB's -- I tend to block out things that I learned a lot about in the past without much result.
- dhruv_anand 3y ago
  Is your YOShInOn RSS reader available as an app or open-source library?
JabavuAdams 3y ago
Why would you need a vector database when your system response time is dominated by calls to off-prem LLMs? Linear search through flat-file of embeddings. Done.
flying_lotus 3y ago
To your point, the market for vector db solutions feels very undifferentiated. I am genuinely curious -- what are the types of ANN use-cases that truly require XXms lookup latency, XXX QPS, and capacity for billions of documents?
formercoder 3y ago
When I prototype RAG systems I don’t use a “vector database.” I just use a pandas dataframe and I do an apply() with a cosine distance function that is one line of code. I’ve done it with up to 1k rows and it still takes less than a second.
- jjtheblunt 3y ago
  What RAG systems do you prototype?
- whalesalad 3y ago
  1k rows? Sounds like kindergarten.
  - visarga 3y ago
    up to 100k rows you don't get faster by using vector store, just use numpy
    - intalentive 3y ago
      And often you have tags that filter it down even further.
- omeze 3y ago
  Everyone is piling on you but Id love to see what their companies are doing. Cosine similarity and loading a few thousand rows sounds trivial but most of the enterprise/b2b chat/copilot apps have a relatively small amount of data whose embeddings can fit in RAM. Combine that with natural sharding by customer ID and it turns out vector DBs are much more niche than an RDBMS. I suspect most people reaching for them haven’t done the calculus :/
  - coffeebeqn 3y ago
    People rushing to slap “AI” on their products don’t really know what they need? Yea that’s absolutely what’s happening now
heyitsguay 3y ago
As someone who has been using pgvector for a while and is vaguely curious about alternatives without having the bandwidth to investigate -- is there anything out there that offers truly differentiated advantages over pgvector? I'm extremely wary of non-OSS solutions in this area, it seems ripe for enshittification and attempts at vendor lock-in.
- MarkMarine 3y ago
  No
- serjester 3y ago
  I use PgVector myself but here's the advantages to a true vector db. - Vectors are massive data wise. In our current production database they take up 95% of the memory - should they be stored separately? - Better support for easily re-embedding, hybrid search, certain RAG workflows - Stronger performance once you're dealing with millions of vectors. I would still stick with PgVector until you're dealing with non trivial scale.
  - nikita 3y ago
    What is taking the most time at scale? Is this ingest, index build or lookups ?
    - visarga 3y ago
      ingest and index build can take time
      - nikita 3y ago
        What volumes are we talking about. There are ways to speed things up dramatically. Index build just became multithreaded (see above). We have ideas on what to do with ingest. Also do you interest from S3 ?
        3y ago
vishnumohandas 3y ago
There are none that run on the edge, yet. Few more miles to go before we "peak".
- 0x6c6f6c 3y ago
  Hmmm. Does pgvector count? That's supported by NeonDB, serverless compute on PostgreSQL. https://neon.tech/docs/extensions/pgvector https://neon.tech/docs/extensions/pgvector
  - nikita 3y ago
    (Neon CEO) It’s about to get a lot better too. Pgvector now supports multi-threaded build https://github.com/pgvector/pgvector/issues/409#issuecomment-1905594721 https://github.com/pgvector/pgvector/issues/409#issuecomment...
    - boomskats 3y ago
      Another very significant contribution to the pg ecosystem. You guys are awesome, thank you for everything you're doing.
      - nikita 3y ago
        Lol rare collaboration between neon, AWS, and supabase. But if Postgres wins we all win!
  - vishnumohandas 3y ago
    By "edge", I was talking about mobile / IOT devices. The closest I can see is the VSS extension[1] for Sqlite. [1]: https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss
- janalsncm 3y ago
  What is the use case for this?

235 points by softwaredoug 3y ago

We are past it, it was months ago :) The points are basically right, but a lot of folks realize all this.

>Real-time recommendations, but driven by vector (and other kinds of) retrieval that looks more like a search engine - not batch computed, nightly jobs common these days. This is already the case. Recommendations are just a fancy search where the query is a vector representing the user. Whether the learning is batched or not doesn't change the fact that it will use vector search for at least candidate generation.

dmezzetti 3y ago

I'll add txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) to the list. There is still plenty of room for innovation in this space. Just need to focus on the right projects that are innovating and not the ones (re)working on problems solved in 2020/2021.

zeroCalories 3y ago
I agree. Honestly even the fundamentals of vector databases aren't really "solved" in the way they are for other databases. Vector indexing, embedding generation, horizontal scaling, etc. can probably still improve a lot. And don't forget, even if Postgres and MySQL are the only traditional databases in town, every tech company had their own SQL database once. Many of them are still around too. No need to get pissy about these companies.
- dmezzetti 3y ago
  Agreed. For example, here is a post about integrating vector search results with semantic graphs for RAG - https://news.ycombinator.com/item?id=39141420 https://news.ycombinator.com/item?id=39141420 And here's a post on an alternative way to integrate vectors with traditional databases (Postgres, MySQL) - https://neuml.hashnode.dev/external-database-integration https://neuml.hashnode.dev/external-database-integration As others have said in this thread, cosine similarity on arrays of vectors isn't novel. But there are many possibilities past that, many we haven't thought of yet too.

monero-xmr 3y ago

Someday enterprises will actually pay someone for LLM tech and infra. Someday…

esafak 3y ago
Lots of companies are paying OpenAI, so someday is yesterday? https://www.reuters.com/technology/openai-annualized-revenue-tops-16-billion-information-2023-12-30/ https://www.reuters.com/technology/openai-annualized-revenue...
- monero-xmr 3y ago
  Most of that is recycled. They don’t break it down because it would make the obvious, obvious. Microsoft pays OpenAI but requires them to use Azure, and OpenAI pays Microsoft the same money back. This is why they continually need billions in investment, because they are far from profitable. The same principle applies to defense. The US gives Israel and Ukraine tens of billions, but that’s a credit to buy from US defense firms. That money gets recycled right back to US weaponry.
  - infecto 3y ago
    Your logic seems highly flawed. I get what you are saying in the example, yes the government provides weapons which are paid for by the government but produced by defense companies. But in the Microsoft example it is customers who are paying Microsoft to use OpenAI via Azure. Thats a free market of money inflows. Same with all the people using OpenAI directly. Not sure how you would even think of the money being recycled in this scenario. Yes of course there is some back scratching in the sense that Microsoft invested in OpenAI with a large portion of that investment in Azure credits which makes the investment quite nice from MSFT's side but there is still real demand for Azure services to use OpenAI apis.
    - monero-xmr 3y ago
      For big enterprises it is either included in existing licenses (i.e. Microsoft Word has ChatGPT embedded) or pilots. No one is cutting massive checks to Microsoft specifically for OpenAI services. This is obvious, but if you need some journalist to validate what is already logically clear: https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-early-products-a-hard-sell-bdd29b9f https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-ea... https://www.wsj.com/tech/ai/ai-deals-microsoft-google-amazon-7f624054 https://www.wsj.com/tech/ai/ai-deals-microsoft-google-amazon...

LispSporks22 3y ago

I believe you've reached peak anything when it's been incorporated into PostgreSQL.

spootze 3y ago
pgvector has you covered: https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- hot_gril 3y ago
  Wow. Back in the day, I had to do cosine similarity indexing with pg-cube. It only did euclidean distance, so I had to store a separate column with normalized vectors.
  - BiteCode_dev 3y ago
    What's cosine similarity, what do you use it for and why is it good to have it into your db instead of somewhere else (like a lib)?
    - pxndxx 3y ago
      Euclidean distance stops "making sense" as the number of dimensions goes up: https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions https://stats.stackexchange.com/questions/99171/why-is-eucli... Cosine similarity measures the angle between two vectors instead, and doesn't suffer from the curse of dimensionality. I guess it's important to have this in your DB, so you make "nearby" queries (give me text that's similar to this other text) in an efficient way.
      - hot_gril 3y ago
        Exactly. Some other context implied here, these vectors were ML embeddings. In that case, like 100 dimensions that vaguely represented our input data in compact form. There was probably a better solution out there, this was just the most readily available for us.

dbCoop3r 3y ago

Why not a "vector filesystem" for Linux? I know it's subjective, but databases have started to feel like running a window manager and desktop on a server. I feel like software needs to take a step back and rethink itself after years of putting chimps at typewriters searching for Shakespeare. Why not a Linux kernel with a module(s) to provide the same assurances, SQL operations? Write directly to the filesystem? Why is all the mathematical concept that we derive software from packaged into endless conceptual blobs of black box state?

feverzsj 3y ago

Investors don't really care if it actually creates more value. They only care if the story can attract the public. They just want to profit by taking next investors' money.

lumost 3y ago

So, there have been gnaw plugins for elastic search for some time. Is there really a need for a new search platform? If there is, what api differentiates it, and why can’t this be expressed in either elasticsearch or Postgres?

reissbaker 3y ago

IMO we are well past peak cosine-similarity-search as a service. Most people I talk to in the space don't bother using specialized vector DBs for that. I think there's space for a much more interesting product that is longer-lived (since it's harder to implement than just cosine-similarity-search on vectors), which is: 1. Fine-tuning OSS embedding models on your real-world query patterns 2. Storing and recomputing embeddings for your data as you update the fine-tuned models. MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I'd bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else's: code retrieval vs document Q&A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system). And if you're fine-tuning new models regularly, you need storage + management, since you'll need to recompute the embeddings every time you deploy a new model. I would pay for a service that made (1) and (2) easy.

latchkey 3y ago
Question, does this require specialized hardware at all? GPUs?
- ipsum2 3y ago
  It doesn't require it in theory, but in practice its required bc CPUs are too slow at fine-tuning and computing embeddings.
phreeza 3y ago
Are you aware of any service or OSS solution for this?
deleted 3y ago
[deleted]
jn2clark 3y ago

mirekrusin 3y ago

We're at the peak of blog posts listing vector databases.

visarga 3y ago

Embeddings are good at capturing surface level information but can't match implicit/deeper/conclusion level information. Say you have a collection of 100,000 math problems, and you want to embed them to search problems that give result "0". Any number of problems can give this result and it is not explicit in the problem statement. But if you solve the problems you can see the data was in there, just not apparent. In general you can see the raw text as a simulation premise that will generate inferences when "executed". The inferenced part is like the hidden part of the iceberg, you don't see it but it is there, implicit in the source text. Not just in math, but in all fields. Embeddings are only good at superficial retrieval. The text needs to be fully analyzed with LLMs before embedding. Thus my conclusion is that we still have a long way to go, we haven't peaked.

deleted 3y ago
[deleted]
wodenokoto 3y ago
What do you mean by fully analyzed? It’s the LLM that does the embedding.
- visarga 3y ago
  Oh the embedding LLMs are usually lightweight BERT models with few layers and <<1B weights, while LLMs are easily 10-100x larger. The idea is to ingest the text in a LLM to extract the facets you are going to search and add those extra tokens to the original text. Then you do regular RAG.
  - sroussey 3y ago
    How?
  - wodenokoto 3y ago
    You want to enrich the base text before embedding? With what kind of prompt? It’s a pretty simple thing to add to a pipeline. Have you tried?
3y ago

deleted 3y ago

[deleted]

physicsguy 3y ago

> In the same way NoSQL forced us to rethink databases. Did it? After using Mongo in my current job (not my choice), I'd choose Postgres again for my next project.

cloverich 3y ago
The thing to know about Mongo is, every database involves design choices that balance ergonomics, performance, and reliability. Every one, except Mongo which, according to their sales team, is the best at everything and has no faults, unless your technical choices are incorrect. In fact I just learned (in a lunch and learn with their team) that when you de-normalize data, inconsistency issues aren't really a problem, and joins are so unusably slow in ALL use cases anyways. Went ahead and just threw my DDIA book in the trash, as they nodded approvingly.

ig1 3y ago

I think https://vespa.ai/ https://vespa.ai/ has the right approach in this space by focusing on being hybrid - vectors alone aren't great for production use cases, it's the combining of vectors+text that lets you use ranking to get meaningful result. (I'm an investor so I'm biased; but it's also the reason why I invested)

zcw100 3y ago

This is a ridiculous rant. “ oh no! We have choices”. Then you list out every choice available for what is a new space people are exploring and the list is barely a half dozen long? It’s more like this is peak “claiming everything is peak”.

dimatura 3y ago
Yeah, I'm happy there's a lot of development in this area - even if it's fueled by the LLM frenzy, good nearest neighbor search solutions are useful in a lot of domains. Though I worked a little bit on this problem over 10 years ago (with an application to visual SLAM), and it is a bit amusing to see that a lot of the ideas and even the libraries are still the same!
softwaredoug 3y ago
Author here, well yeah, I agree its probably ridiculous. Sort of testing the waters to see if I'm way off base. I think what I mean to say is that, in my experience, practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve. In fact, that's a teeny tiny part of it. Hence "peak vector DB". So I think the market needs some education that its harder than that. That part is my rant :). I've spoken / worked on enough problems now to see that disconnect between market and reality. Though I think "vector DB" is actually a place for capital/brainpower to concentrate to solve these other problems. And I think we'll see the vector DB vendors pivot there. It's just taking a while for the market and investors to see this...
- brookst 3y ago
  It sounds like you’ve conflated “gold rush” with “peak”. All sorts of novel technologies had mad rushes when they’re new, but that does not mean they have peaked. The dot bomb era with its ridiculous overvalued useless startups was a gold rush, but it was in no way peak Internet.
- phillipcarter 3y ago
  > practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve I agree, and as one who does exactly and only this on the search side, it's also something that falls flat on its face if you don't think a little more about the data and tasks involved. I wrote about it here[0], but the gist of it for our use case is that if we don't intentionally include what may be considered "less relevant" data then we stand a good chance at failing our main generative task. [0]: https://phillipcarter.dev/2024/01/15/three-properties-of-data-for-making-llms-awesome/ https://phillipcarter.dev/2024/01/15/three-properties-of-dat...

wahnfrieden 3y ago

Is there a good choice available for running inside a browser, client-side? Without a server to create or run inferences

FrustratedMonky 3y ago

""“how can so many vector databases need to exist?”."" Same with languages. Why so many languages. Why can't we all get behind a few, do we need more than 6? For every case/problem? Put all our combined resources towards a smaller set. We need a few DB's, a few languages, a few frameworks. Do we need hundreds? Like everyone rolls their own everything.

ryoshu 3y ago

Don't confuse a feature with a product. Postgres works great and you can layer in cosine similarity along with full-text search in a single query if you need to.

crowcroft 3y ago

The big LLM companies are well positioned to build a lot of what a vector database is used for into their existing APIs and offerings. Both simplifying DX and devops. Then on the other side existing databases will want to add functionality to be used as vector databases as well. I think there’s lots of innovation ahead and it’s too soon to know what the end outcome will be.

macspoofing 3y ago

Not yet. There is excitement for vector databases in some specialized areas but it hasn't really filtered out to the wider rank-and-file software engineering circles. You know it will be 'peak vector database' when you'll see blog posts on migrating your relational data to a vector database (with a follow-up 2 years later about moving back to PostgreSQL due to the shitshow that ensued).

lpgauth 3y ago

"We would say Cassandra is a columnar data store, alongside the Scylla or HBase." Cassandra and Scylla are row based distributed key value stores.

sergiotapia 3y ago

I believe the next step is an "Algolia" of sorts for cosine-similarity search. Why bother with chunking data, synching it, and then tagging metadata to it. DB providers should be smart enough to optimize chunking strategy for the kind of content being indexed and then provide a simple API endpoint to query against their data. "RAG in a can".

eskibars 3y ago
That's exactly what Vectara is (full disclosure, I work there)

jhj 3y ago

Speaking as an author of one of the primary libraries for doing this stuff (faiss), it is not because it is still an open ended research problem on how approximate high-dimensional dense or sparse nearest neighbor should work, let alone maximum inner product search where the research story is even worse, or other non-metric space similarity measures. All of the current techniques still have quite unacceptable tradeoffs involved. While traditional database indexing is also still an open-ended research problem (e.g., read amplification/write amplification tradeoffs and the like), it produces exact solutions. That isn't the case at all for vector indexing beyond brute-force search, or exact indexing like k-D/BSP trees which don't work well in high dimensions due to the curse of dimensionality.

jbellis 3y ago
Why is the research story for MIPS even worse than for ANN?
- jhj 3y ago
  There is no good geometry to be exploited, and the query vectors might be (and are usually) distributed quite differently than the indexed vectors. For Euclidean (L2) distance indexes where the vectors are partitioned based on geometry (e.g., pretty much every indexing type, including cell-probe like IVF, most forms of LSH, or graph based indices), query vectors can be naturally associated geometrically with candidate nearest neighbor vectors, so the distribution of queries doesn't matter as much. For inner product, it's hard to do much better than spherical clustering (what one would usually do for cosine similarity, which is to project all vectors to the surface of a unit hypersphere, and searching for nearest neighbors via cosine similarity is exactly equivalent to L2 search). But, in general the maximum inner product in the indexed set may lie nowhere near to the projection of the query vector onto the surface of the hypersphere. The maximum inner product for a query vector might be almost nearly perpendicular to the query vector (e.g., a very, very far out and almost perpendicular) versus a vector that is parallel to the query vector but with tiny norm. In two dimensions, an example could be (1, 0) as a query vector, but (1, 10^6) as a database vector (or vice versa). The inner product is 1 but the two vectors are very far apart in Euclidean distance. If you project the vectors to the unit 1-sphere, the query vector is still (1, 0) but the database vector now becomes (1 / sqrt(10^12 + 1), 10^6 / sqrt(10^12 + 1)) ~= (0.000000999..., 0.99999...) (apologies if there's an error here) which would also be in a very different cell if one were using a graph-based or IVF partitioning. Neural search techniques do show some promise here though (say, using a neural net to predict which vector buckets to look at).

agonz253 3y ago

Are there no distinguishing features between these vector databases? I'm not familiar with them so I was looking for any comment on that in the article, whether some make different tradeoffs than others, are easier to operate or implement, more scalable, etc. That together with their relative novelty might help explain why there are so many.

EGreg 3y ago

I don't know, let's ask an AI about it :)

throwaway201606 3y ago

Hill I am willing to die on: Peak "$XXXXXXXX" database is when your particular flavor of DB is completely consumed into traditional RDBMSes. Vector databases (and all other incremental or transformational improvements) are just features of regular plain traditional RDBMSes that have not been implemented in traditional RDBMSes yet. I have seen every new DB tech subsumed by traditional databases over time as compute capability improved. No exceptions. The list is endless: - object databases (e.g. blobs, JSON) - OLAP - in DB programming ( XX-SQL eg PL/SQL, T-SQL, ANSI-SQL) - column-oriented data stores - key-value - graph databases - No SQL - Cloud, distributed, whatever - statistical analysis databases - document databases All these used to be standalone, very expensive, specialty products but are now just one more checkbox on the Oracles/SQL-Servers/DB2s of this world. All these have been swallowed by the borg of commercial databases without so much as a burp. There is no winning the commercial market long term for these products. Big business buys traditional RDBMSes because they are the kitchen sink. They do EVERYTHING and they will eventually do this new hot thing, the business will just have to pay big dollars for it. Which is not a problem for big business. There is a reason that cartoon about the Oracle org hierarchy was made (bottom right): all the company does is make product (Engineering) and protect that product. And it is very good at making good product. https://i0.wp.com/stratechery.com/wp-content/uploads/2013/07/2011.06.27_organizational_charts.png?resize=646%2C630&ssl=1 https://i0.wp.com/stratechery.com/wp-content/uploads/2013/07...

senderista 3y ago
Exactly my take, I see no moat here. If there were a way to short the vector DB startup phenomenon and I had the resources I would do it.
- thraxil 3y ago
  Literally. We do a lot of vector DB and RAG stuff (who isn't these days, right?) and after a bunch of testing and benchmarking went with pgvector integrated into our existing PostgreSQL database. Operationally simple, performs perfectly adequately. I'm sure there are some niche use-cases where the dedicated vector DBs make sense, but for anyone just getting into it, don't underestimate PostgreSQL and pgvector.

hiddencost 3y ago

Nope

PaulHoule 3y ago

I got interested in vector search around 2004, read a lot of papers about vector search algorithms and was not really impressed with the tradeoffs involved (it's not the clear win that B-Trees are for 1-d indexing) and wound up using full scans unless I had sparse vectors. When Pinecone came out and started blogging heavily it seemed that they'd read the same papers I did but came to the conclusion the glass was half full instead of half empty. I could have missed it but I haven't see anything in the literature that's a huge improvement over 20 year old algos. Circa 2014 I worked on a search engine for patents and related literature that made vectors for 20 million + documents and they decided to use full scan and (i) it performed so well (in terms of accuracy) that we sold a license to the USPTO on day two after we put up the demo, and (ii) there were a lot of things about it that were slow like the build system, index building and model training but vector search wasn't one of them. My YOShInOn RSS reader has about a million documents in 2024 and it uses vectors for classification and clustering. Using vectors for search is a clear extension and I've done some prototyping of searches with full-scan and performance is "good enough" (full scan has 'mechanical sympathy'.) I'd probably stuff my vectors into FAISS if I wanted to do anything more and forget about it. Sending my vectors to some cloud service so they can pay AWS prices to store them? That's for the birds. I respect Pinecone for being early to the party but I think those who jumped in in 2022 were laggards.

kwillets 3y ago
My experience comes from around the same time frame -- I spent about a year on an aborted spectral dimension reduction project, and I only recently realized how similar the problem still is today. I'm not sure if that makes me more or less qualified to do vector DB's -- I tend to block out things that I learned a lot about in the past without much result.
dhruv_anand 3y ago
Is your YOShInOn RSS reader available as an app or open-source library?

JabavuAdams 3y ago

Why would you need a vector database when your system response time is dominated by calls to off-prem LLMs? Linear search through flat-file of embeddings. Done.

flying_lotus 3y ago

To your point, the market for vector db solutions feels very undifferentiated. I am genuinely curious -- what are the types of ANN use-cases that truly require XXms lookup latency, XXX QPS, and capacity for billions of documents?

formercoder 3y ago

When I prototype RAG systems I don’t use a “vector database.” I just use a pandas dataframe and I do an apply() with a cosine distance function that is one line of code. I’ve done it with up to 1k rows and it still takes less than a second.

jjtheblunt 3y ago
What RAG systems do you prototype?
whalesalad 3y ago
1k rows? Sounds like kindergarten.
- visarga 3y ago
  up to 100k rows you don't get faster by using vector store, just use numpy
  - intalentive 3y ago
    And often you have tags that filter it down even further.
omeze 3y ago
Everyone is piling on you but Id love to see what their companies are doing. Cosine similarity and loading a few thousand rows sounds trivial but most of the enterprise/b2b chat/copilot apps have a relatively small amount of data whose embeddings can fit in RAM. Combine that with natural sharding by customer ID and it turns out vector DBs are much more niche than an RDBMS. I suspect most people reaching for them haven’t done the calculus :/
- coffeebeqn 3y ago
  People rushing to slap “AI” on their products don’t really know what they need? Yea that’s absolutely what’s happening now

heyitsguay 3y ago

As someone who has been using pgvector for a while and is vaguely curious about alternatives without having the bandwidth to investigate -- is there anything out there that offers truly differentiated advantages over pgvector? I'm extremely wary of non-OSS solutions in this area, it seems ripe for enshittification and attempts at vendor lock-in.

MarkMarine 3y ago
No
serjester 3y ago
I use PgVector myself but here's the advantages to a true vector db. - Vectors are massive data wise. In our current production database they take up 95% of the memory - should they be stored separately? - Better support for easily re-embedding, hybrid search, certain RAG workflows - Stronger performance once you're dealing with millions of vectors. I would still stick with PgVector until you're dealing with non trivial scale.
- nikita 3y ago
  What is taking the most time at scale? Is this ingest, index build or lookups ?
  - visarga 3y ago
    ingest and index build can take time
    - nikita 3y ago
      What volumes are we talking about. There are ways to speed things up dramatically. Index build just became multithreaded (see above). We have ideas on what to do with ingest. Also do you interest from S3 ?
      - 3y ago

vishnumohandas 3y ago

There are none that run on the edge, yet. Few more miles to go before we "peak".

0x6c6f6c 3y ago
Hmmm. Does pgvector count? That's supported by NeonDB, serverless compute on PostgreSQL. https://neon.tech/docs/extensions/pgvector https://neon.tech/docs/extensions/pgvector
- nikita 3y ago
  (Neon CEO) It’s about to get a lot better too. Pgvector now supports multi-threaded build https://github.com/pgvector/pgvector/issues/409#issuecomment-1905594721 https://github.com/pgvector/pgvector/issues/409#issuecomment...
  - boomskats 3y ago
    Another very significant contribution to the pg ecosystem. You guys are awesome, thank you for everything you're doing.
    - nikita 3y ago
      Lol rare collaboration between neon, AWS, and supabase. But if Postgres wins we all win!
- vishnumohandas 3y ago
  By "edge", I was talking about mobile / IOT devices. The closest I can see is the VSS extension[1] for Sqlite. [1]: https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss
janalsncm 3y ago
What is the use case for this?