16 ms·
Vector Databases: A Technical Primer [pdf]
- jide_tracc 3y agoA few months ago, I taught a class on Vector Databases for a TGE Data private client and then decided to record it into a short course for a wider audience. The course is a mix of theory and demos discussing some of the underlying concepts of Vectors, Vector Databases, Indexing, Search Similarity and ending with demos specifically for Pinecone and Weaviate databases.
- CharlesW 3y agoCool! Is there a video as well, then?
- jide_tracc 3y agoThere is but it's a udemy course unfortunately. I don't want to spam the group with links but the link is on the lat page of the document
- codetrotter 3y agoIf you have a discount code you can share, it would provide a neat excuse for posting the link in a comment along side it ;)
- deleted 3y ago[deleted]
- jide_tracc 3y agoThank you! That's a great idea. I created a discount coupon which is active for the next 5 days up until Jan 17th https://www.udemy.com/course/vector-databases-deep-dive/?couponCode=59670CA7401E60FE031A https://www.udemy.com/course/vector-databases-deep-dive/?cou...
- codetrotter 3y agoThanks :) I bought a copy now. I already have a Udemy account so when I went to checkout and logged in it said the promo was for new users only. But I went back to the course page again and reloaded the page so that I was logged in and then I was able to buy it with the coupon applied.
- AtlasBarfed 3y agoANNOY is essentially a binary space partition tree for points without having to worry about sets of points aka polygons?
- fzliu 3y agoLike IVF, Annoy partitions the entire embedding space into high-dimensional polygons. The difference is how the two algorithms do it - IVF (https://zilliz.com/learn/vector-index https://zilliz.com/learn/vector-index) uses centroids, while Annoy (https://zilliz.com/learn/approximate-nearest-neighbor-oh-yeah-ANNOY https://zilliz.com/learn/approximate-nearest-neighbor-oh-yea...) is basically just one big binary tree.
- ignoramous 3y agoSome more resources: - A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge, https://arxiv.org/abs/2310.11703 https://arxiv.org/abs/2310.11703 - Survey of Vector Database Management Systems, https://arxiv.org/abs/2310.14021 https://arxiv.org/abs/2310.14021 - What are Embeddings, https://raw.githubusercontent.com/veekaybee/what_are_embeddings/main/embeddings.pdf https://raw.githubusercontent.com/veekaybee/what_are_embeddi... --- h/t: https://twitter.com/eatonphil/status/1745524630624862314 https://twitter.com/eatonphil/status/1745524630624862314 and https://twitter.com/ChristophMolnar/status/1745731602682982675 https://twitter.com/ChristophMolnar/status/17457316026829826...
- fzliu 3y agoThrowing a few more on here (mix of beginner and advanced): - Wikipedia article: https://en.wikipedia.org/wiki/Vector_database https://en.wikipedia.org/wiki/Vector_database - Vector Database 101: https://zilliz.com/learn/introduction-to-unstructured-data https://zilliz.com/learn/introduction-to-unstructured-data - ANN & Similarity search: https://vinija.ai/concepts/ann-similarity-search/ https://vinija.ai/concepts/ann-similarity-search/ - Distributed database: https://15445.courses.cs.cmu.edu/fall2021/notes/21-distributed.pdf https://15445.courses.cs.cmu.edu/fall2021/notes/21-distribut...
- rammy1234 3y agoThrowing one more - https://www.pinecone.io/learn/vector-database/ https://www.pinecone.io/learn/vector-database/
- simonw 3y agoHere's my attempt at this: Embeddings: What they are and why they matter https://simonwillison.net/2023/Oct/23/embeddings/ https://simonwillison.net/2023/Oct/23/embeddings/ Available as an annotated presentation + an optional 38 minute video.
- Scorpiion 3y agoThanks for writing this one Simon, I read it some time ago and I just wanted to say thanks and recommend it to folks browsing the comments, it's really good!
- jpasmore 3y agohelpful - thx - building Latimer.ai using Pinecone
- kgeist 3y agoWhen does a simple linear search become an issue? For example, a cos similarity function which runs even over a million of vectors is surprisingly fast. We can further shard vectors by user (i.e. if they can only search in their own data they uploaded), or even by document (for things like "chat with this PDF") then it's likely that most searches will happen over small ranges of vectors. I suspect the main downsides would be increased CPU usage and having to load vectors to RAM all the time. However, for small projects with low RPS you could probably get away without a special database? Does anyone know of any benchmarks where they compare vector databases to a plain linear search? How they scale in regards to increased vector count and RPS, etc.
- rolisz 3y agoDo you mean using brute force (exact search) and not approximate nearest neighbor search? From my unscientific benchmarks, doing an exact search with 100k vectors takes about 1 second. Usually chatbots take much longer to generate the text, so that's still an acceptable time for doing exact search, which also means you will always find the best matches.
- kgeist 3y ago>Do you mean using brute force (exact search) and not approximate nearest neighbor search? Yes. >doing an exact search with 100k vectors takes about 1 second. The speed can depend on the language/runtime of your choice and the number of dimensions. What language/runtime did you use, how many dimensions in a vector? (I heard too many dimensions for vector search is an overkill)
- Radim 3y agoBrute force search is both exact (100% accurate) and pleasantly linear – a predictable algorithm. CPUs and caches like that, so performance is much better than you might otherwise expect. From my https://rare-technologies.com/performance-shootout-of-nearest-neighbours-querying/ https://rare-technologies.com/performance-shootout-of-neares... benchmark of kNN libs: "Brute force doesn’t care about the number of neighbours, so its performance is 679ms/query regardless of “k”. When run in batch mode (issuing all 100 queries at once), average query time drops to 354ms/query." This was 500 dimensions over a 3.7M dataset (the English Wikipedia), in 2014. So, ~700ms/search, or half that if you can batch several searches together at once. YMMV.
- m3kw9 3y agoIs this how most AI services implements RAG?
- gk1 3y agoYes, most RAG is done with a vector database. Or something that approximates a vector database, like a traditional database with a vector-index add-on.
- charcircuit 3y agoKeyword or text matching based search is likely still more popular due to how long its been around, its simplicity, and the tooling built around it. Most companies who have internal search are most likely not using vector / semantic search, but are doing something more basic.
- esafak 3y agoMissing coverage of hybrid search (vector + lexical).
- swyx 3y agothis is quite important. every vector db will provide lexical features, every traditional db will provide vector features. everything trends hybrid.
- cranberryturkey 3y agois surrealdb anygood as a vector database?
- gk1 3y agoThis is really cool, thanks for making this and sharing with the community!
- simonw 3y agoSince digitaloceanspaces.com is an S3-style hosting provider, it would be neat if Hacker News could special-case it to display something like tge-data-web.nyc3.digitaloceanspaces.com as the domain instead of just digitaloceanspaces.com Though it looks like S3 has the same problem: https://news.ycombinator.com/item?id=38876761 https://news.ycombinator.com/item?id=38876761 There's precedent for this elsewhere though - sites on a x.github.io subdomain have special treatment here: https://news.ycombinator.com/from?site=lfranke.github.io https://news.ycombinator.com/from?site=lfranke.github.io
- A_Duck 3y agoYou may be interested in this project: https://publicsuffix.org/ https://publicsuffix.org/
- danjl 3y agoGreat overview, but the final section doesn't address the obvious question about how to decide between using a "vector store", like Postgres+pgvector vs a "vector database", like Pinecone. I'd love to see another presentation which discusses the various tradeoffs, like query speed, insertion/index-building speed, ease-of-use, and others to help guide people trying to decide which of the options is best for their application.
- esafak 3y agoI'd call the former a vector extension. A database is a store with bells and whistles.
- jide_tracc 3y agoThanks for the feedback! I'll explore this topic for upcoming articles for a deeper dive
- Beefin 3y agoi started compiling comparisons between a bunch a while back: http://vectorsearch.dev/ http://vectorsearch.dev/ welcome any PRs
- danjl 3y agoAlthough excellent for vector databases, unfortunately this does not include postgres with pgvector.
- takinola 3y agoWhen I started playing around with AI applications and learnt about RAG techniques, it took me a while to grok vector databases and was a bit of pain to learn how to set one up. So I built my own little pet project, RagTag [1], which abstracts the vector database behind a simple CRUD API. I simply POST documents to RagTag and they are automatically converted into embeddings and made available to be queried for similarity searches. [1] RagTag - https://ragtag.weaveapi.com https://ragtag.weaveapi.com
- stefanha 3y agoIn the table on slide 15 the Indexing & Search Efficiency cells for Traditional Databases and Vector Databases appear to be swapped.
- kevindamm 3y agoYes that last row looks swapped to me as well.
- nostrebored 3y agoFrom looking at this, I think it’s a very risky starting point for an engineer to kick off from. Things like mentioning they’re clustered by meaning and optimized for analytics are questionable. The clustering depends on the embedding you calculate. If you think that the embedding is a good semantic approximation of the data then maybe this is a fine way of thinking about it. But it’s not hard to imagine embeddings that may violate this — eg if I use an audio file and a text file that are identical in meaning through the same embed process, unless it is multimodal they will likely be distant in the embedding vector space. I fully expect to see embeddings that put things close together in the vector space based on utilization rather than semantic similarity. If I’m creating a recommender system, I don’t want to group different varieties of one off purchases closely. For instance, the most semantically similar flight is going to be another fight to the same destination at a different time or a flight to a nearby airport. But I would want to group hotels often purchased by people who have previously bought the flight. Vector databases also allow you to provide extra dimensionality into the data, like time awareness. Nothing is forcing you to use a vector that encodes semantic meaning. And from this, you can see that we’re optimized for lookups or searches based on an input vector. This is not analogous to OLAP queries. This is more akin to elasticsearch than snowflake. If you are using a vector database thinking it’s going to give you reporting or large scale analytics on the vector space afaik there isn’t a readily available offering.
- chasd00 3y agocalculating the embeddings is still a mystery to me. I get going from a picture of an Apple to a vector representing "appleness" and then comparing that vector to other vectors using all the usual math. What I don't get is, who/what takes the image as input and outputs the vector. Same goes for documents, let's say i want to add a dimension (another number in the array) what part of the vector database do I modify to include this dimension in the vector calculation? Or is going from doc/image/whatver to the vector representation done outside the database in some other way? edit: it seems like calculating embeddings would be something an ML algorithm would do but then, again, you have to train that one first. ...it's training all the way down.
- 3y ago
- dmezzetti 3y agoIt's also important to understand why we'd even want vectors in the first place. This article covers the advantages of semantic search over keyword search - https://medium.com/neuml/getting-started-with-semantic-search-a9fd9d8a48cf https://medium.com/neuml/getting-started-with-semantic-searc...
- yagami_takayuki 3y agoFor recognizing features such as hair and skin color, which would do a better job? Machine learning with image classification? Or a vector database? I've had Weaviate return a dog given a human as input when doing an image similarity search, so I was wondering if there's some way to improve the results or whether I'm barking up the wrong tree.
- bfeynman 3y agoI would think you could improve your embedding space to address that issue, partially. Similarity search (as a result of some contrastive loss) definitely suffers at the tails and the OOD is pretty bad. That being said, you're more likely to have higher recall than a more classical technique.
- yagami_takayuki 3y agoThanks for your answer!
- dimatura 3y agoFor those two in particular? You'd definitely get a better result with an ML model such as a convolutional neural net. In some sense, using an image similarity query is a kind of ML model - nearest neighbor - which can work in some scenarios. But for this specifically, I'd recommend a CNN.
- yagami_takayuki 3y agoThanks will experiment with a CNN.
- m00x 3y agoYou don't use vector databases independently, you need to input the embedding from a ML model. For your use-case, it should be pretty simple. You could use a CNN and train it, or use YOLO, Deepface, or other face detection algos, then within the face, find the hair and find the skin. From there you can use a vector database to get the colors that resemble other inputs, or you can use a simple CNN to classify the hair and skin to the closest label.
- dimatura 3y agoAny recommendations for an embedded embedding database (heh)? Embedded, as in sqlite. For smaller-scale problems, but hopefully more convenient than say, LMDB + FAISS.
- ripley12 3y agoFWIW, Simon Willison's `llm` tool just uses SQLite plus a few UDFs. The simplicity of that approach is appealing to me but I don't have a good sense of when+why it becomes insufficient.
- dimatura 3y agoThanks, I'll check it out.
- pjot 3y agoI actually just finished a POC using DuckDB that does similarity search for HN comments. https://github.com/patricktrainer/hackernews-comment-search https://github.com/patricktrainer/hackernews-comment-search
- PhilippGille 3y agoFor Python I believe Chroma [1] can be used embedded. For Go I recently started building chromem-go, inspired by the Chroma interface: https://github.com/philippgille/chromem-go https://github.com/philippgille/chromem-go It's neither advanced nor for scale yet, but the RAG demo works. [1] https://github.com/chroma-core/chroma https://github.com/chroma-core/chroma
- catketch 3y agohttps://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss
- dimatura 3y agoawesome, how did I not find this :D
- 3y ago
- ripley12 3y agoThe "Do you need a Dedicated Vector Database?" slide is quite interesting, but doesn't really answer its own question! This is something I've been wondering myself, if anyone has any guidelines or rules of thumb I would appreciate it. I've recently been using Simon Willison's (hi simonw) excellent `llm` tool that can help with embeddings, and it takes the simplest approach possible: just store embeddings in SQLite with a few UDFs for calculating distance etc. The simplicity of that approach is very appealing, but presumably at some level of traffic+data an application will outgrow it and need a more specialized database. Does anyone have a good intuition for where that cutoff might be?
- summarity 3y agoI’ve had some success scaling a quick and dirty engine (that powers findsight.ai) to tens of millions of vectors, details in the talk here: https://youtu.be/elNrRU12xRc?t=1556 https://youtu.be/elNrRU12xRc?t=1556 That’s maybe 1kLOC so I didn’t need an external one after all.
- jide_tracc 3y agoThanks for the feedback! I'll make this the topic of my next article and do a deeper dive on this
- darkteflon 3y agoYMMV, of course, but this[1] article - posted here a few weeks back - really helped us to focus on how to think about what we actually needed: in short, a search engine. [1] https://news.ycombinator.com/item?id=38703943 https://news.ycombinator.com/item?id=38703943
- tristanho 3y agoHonestly just loading all vectors in-memory (and stuff like sqlite, pgvector) is totally fine when you're dealing with O(100k) vectors, but beyond that all the workable options like pinecone get gnarly, slow, and ridiculously expensive. The best option by far I know of is turbopuffer.com , which is like 100x cheaper than pinecone and seems to actually scale. Since it's not listed in the suggested vector dbs section of the slides, wanted to lob it in as a solid suggestion :)
- ok123456 3y agoAre these dedicated vector databases doing anything more complicated than what can be accomplished using Postgres with the Cube extension?
- m00x 3y agoYou don't need a dedicated vector db, you can use pgvector. You could maybe use Cube for euclidean space search, but you're better off using optimized algorithms for embedding space search. https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- gk1 3y agoYes, see: https://www.pinecone.io/blog/hnsw-not-enough/ https://www.pinecone.io/blog/hnsw-not-enough/
- ok123456 3y agoThis is an unconvincing advertisement.
- Beefin 3y agoyou really don't need much more than hnswlib: https://esteininger.medium.com/building-a-vector-search-engine-using-hnsw-and-cosine-similarity-753fb5268839 https://esteininger.medium.com/building-a-vector-search-engi...
- shreyas44 3y agoFrom a few months ago - https://news.ycombinator.com/item?id=35814381 https://news.ycombinator.com/item?id=35814381
- jhj 3y agoI don't know why PQ is listed as an "indexing strategy". It's a vector compression/quantization technique, not a means of partitioning the search space. You could encode vectors with PQ when using brute-force/flat index, an IVF index, with HNSW (all of which are present in Faiss with PQ encoding as IndexPQ, IndexIVFPQ and IndexHNSWPQ respectively), or even k-D trees or ANNOY if someone wanted to do that. "Use HNSW or Annoy for very large datasets where query speed is more important than precision": Graph-based methods have huge memory overhead and construction cost, and they aren't practical for billion-scale datasets. Also, they will usually be more accurate and faster than IVF techniques (as you would need to visit a large number of IVF cells to get comparable accuracy), though IVF can scale to trillion-sized databases without much overhead yet with reasonable speed/accuracy tradeoffs unlike other techniques. I'd say "use for medium-scale datasets where query speed is important, yet high accuracy is still desired and flat/brute-force indexing is impractical".
- m00x 3y agoIt turns a continuous space into a discrete space. You would first do PQ, then do KNN of the new discrete vector. This way you can compress the vocabulary to a fixed size.
- jhj 3y agoThe implementations that I am aware of, including the one in Faiss which I wrote (described in detail in https://arxiv.org/abs/1702.08734 https://arxiv.org/abs/1702.08734), do not index the vector based on its PQ encoding (e.g., in IVFPQ). The IVF cell chosen in which to put the vector is based on its pre-quantized (full precision floating point) representation. It would lose too much precision to perform all comparisons in the compressed space. Also, distance comparisons are usually of the "ADC" (asymmetric distance comparison) form: the query vector is in full floating point format, and vectors, if quantized/compressed in the database, are effectively decompressed and compared in the full floating point domain. This is true even with PQ, as the distance between the query vector subspaces with each of the 2^n PQ codes for the same subspace are precomputed before comparison (and then the distance computation becomes a lookup-add based on the precomputed distance tables). LSH techniques unlike PQ are more accurately described as an indexing technique, since the buckets into which vectors are placed are based on their encoding (via hashing/binarization) and are fully compared in that space.
- perone 3y agoI find it interesting how everyone ignore EuclidesDB (https://euclidesdb.readthedocs.io https://euclidesdb.readthedocs.io) which came before Milvus and others in 2018, it is free and open-source. Same for all presentations from major DBs.
- koreankid227 3y agoNice resource! One observation: the last row of the table in slide 15 (comparing traditional DBs with Vector DBs) seems flipped?
- PaulDavisThe1st 3y agoA nice introduction. However, I feel that many such introductions, it skips too quickly over the question of feature selection. This is the step that introduces, rather subtly and sometimes almost ignorably, human judgement into what otherwise feels like a very automated and "it's just math" system. Let's take audio. What features will you select from it to make your N-dimensional vector? An easy answer might appear to be "as many as possible". However, your first problem is that you may not have access to feature characterization even for just the features you can trivially name. Your second problem is that without deep knowledge of the domain, you may not even be aware of potential features that you should probably be using. Your third problem is that even with deep knowledge of the domain, you may still be unaware of potential features that you probably should be using. A practical example may help. Perhaps you're a fan of Reichian phase-shifting minimalist music. You (indirectly) interrogate a vector database to find music similar to a canonical piece from this genre (say, Piano Phase). The database uses many audio & music related features, including but not limited to: dominant frequency, note onset intervals, volume, frequency-distribution-based timbral characteristics, apparent root notes and scales etc etc etc. Unfortunately for you, the feature set in the database does not include "note-to-note intervals over time is constant". Your query will manage to find things that are timbrally, harmonically, melodically and rythmically similar, but it will be by sheer good luck that any of them share the key characteristic: steady changes in the relative phase of two different (or the same) melodic lines. This is just one example. It's not hard to cook up equivalents for visual, textual, numerical ... any data at all. Now of course, none of this makes vector databases and feature classification useless. It does mean that when you do or do not find match patterns in a given data set, one of the early questions you need to ask yourself is whether the feature set has any strong guarantee of being complete, and if not, how it may need to be expanded.
- yunyu 3y agoVector databases are oriented around search and retrieval. The usual approach of generating vectors is to fine tune a large pretrained model and extract the inner representations. Because the dataset contains successful queries and retrieval results, all you need to do is optimize a loss function (usually contrastively or approximated versions of common ranking functions) using raw inputs on the similarity objective supported by the vector database. For common modalities like tabular/text/image/audio data, there is basically no human judgement involved in feature selection - just apply attention. Note: current state of the art text to vector models like E5-Mistral (note: very different from original E5) don’t even require human curation in the dataset
- derrickrburns 3y agoWell done!
- itsthecourier 3y agoThe table with index types in the slide "comparison" is inverted
- anotheraccount9 3y agoIf I read organized data from a traditional db, and add new tables based on potential similarities from the existing data/tables, with scores, would that be a vector db?
- cubefox 3y agoWhat are vector databases mainly used for in practice? Similarity search? Showing "related" entries?
- huqedato 3y agoExcellent introduction (brief course)! 10/10! Please make more like this on transformers, LLM, DL, ML, AI etc.
- tluyben2 3y agoI found the code of Qdrant very readable (with nice comments in the top) even if you are not (quite) aware of how vector databases (exactly) work. You need to be a programmer, but with that as a given, it reads surprisingly easy for a production database (that companies like OpenAI etc use).
- dejli 3y agoI watched yesterday talk from PostgresML https://www.youtube.com/watch?v=D8YMMOhAeBs https://www.youtube.com/watch?v=D8YMMOhAeBs and found it very practical and to the point on how you could use Postgres to build vector database.