8 ms·
Vector database built for scalable similarity search
- hobs 4y agoA good set of resources if you wanna check out some other ones, been reviewing some myself. https://github.com/currentslab/awesome-vector-search https://github.com/currentslab/awesome-vector-search I was surprised to see Elastic actually has ok support for some of this stuff, though it appears slower for most of the tasks.
- jabo 4y agoWe added HNSW-based vector search to Typesense as well recently: https://typesense.org/docs/0.24.0/api/vector-search.html https://typesense.org/docs/0.24.0/api/vector-search.html So you can combine attribute-based filters along with nearest-neighbor search. Put together this semantic search + filtering demo just last week: https://github.com/typesense/typesense-instantsearch-semantic-search-demo https://github.com/typesense/typesense-instantsearch-semanti...
- jn2clark 4y agoThanks for the link. Nice to see Marqo on there (disclaimer I am a co-founder of Marqo). For anyone that is interested it includes a really nice api for handling a lot of the manipulations and operations you want to do (adding, updating, patching documents, filtering, embeddings only a subset of fields, multi-modal querying, multi-modal document representations) which are absent from vector db's. It also takes care of inference https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo
- fzliu 4y agoI work at Zilliz (https://zilliz.com https://zilliz.com) and am a part of the Milvus community. Here are some other resources in case anybody's interested in learning more about embeddings, vector search, and vector databases: 1) Embedding crash course https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture https://developers.google.com/machine-learning/crash-course/... 2) What is a vector database? https://zilliz.com/learn/what-is-vector-database https://zilliz.com/learn/what-is-vector-database 3) Introduction to vector similarity search https://zilliz.com/blog/vector-similarity-search https://zilliz.com/blog/vector-similarity-search 4) ANN benchmarks http://ann-benchmarks.com http://ann-benchmarks.com 5) Unstructured data ETL https://github.com/towhee-io/towhee https://github.com/towhee-io/towhee
- jhgg 4y agoDoes Zilliz do BYOC?
- fzliu 4y agoNot yet, but this functionality should be coming soon. We're currently working on adding the capability to call third party embedding APIs directly from a Zilliz Cloud instance.
- jhgg 4y agoI was mulling over the idea of building keyword image search (say, with CLIP based embeddings). However I'm not really sure the cost, or whether or not this is the best solution. Do you have any case studies about large deployments of this software, and what the upper limits of scale might be?
- ttt3ts 4y agoDon't start with mullivus if you're learning. Too much yak shaving. Try https://github.com/criteo/autofaiss https://github.com/criteo/autofaiss. Also, TBH, it is a lot cheaper to run a simple faiss index.
- ywmario 4y agoMilvus standalone is pretty neat. I made it successfully run on the first try.
- kkielhofner 4y agoAs another commenter noted, Milvus is overkill and a "bit much" if you're learning/playing. A good intro to the field with progression towards a full Milvus implementation could be starting with towhee[0] (which is also supported by Milvus). towhee has an example to do exactly what you want with CLIP[1]. For icing on the cake the example notebook includes deployment of the model with Nvidia Triton Server[2], which IMO is hands down the best way to actually deploy an ML model[3]. [0] - https://towhee.io/ https://towhee.io/ [1] - https://github.com/towhee-io/examples/tree/main/image/text_image_search https://github.com/towhee-io/examples/tree/main/image/text_i... [2] - https://github.com/triton-inference-server/server https://github.com/triton-inference-server/server [3] - https://news.ycombinator.com/item?id=35199418#35200266 https://news.ycombinator.com/item?id=35199418#35200266
- m1117 4y agoAh I see, they released Milvus 2.0 which is similar to Pinecone. What are the differences?
- gk1 4y agoBiggest difference is Milvus is self-hosted (manage your own infra) while Pinecone is a managed service. There are other differences but that’s often a driving factor. We (Pinecone) tend to attract customers who want to start, ship, and scale quickly and reliably without worrying about infra and ops overhead. https://www.pinecone.io https://www.pinecone.io
- m1117 4y agoI think this update introduces Milvus 2.0, "Managed Milvus", which seems to be not self-hosted, but in-cloud managed. You can create account and see that it's similar to Pinecone.
- fzliu 4y agoYup! Zilliz Cloud (managed Milvus) provides free credits for you to get started with a full-service (no throttling or other limitations) PoC as well.
- fzliu 4y agoMilvus is completely open source (https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus) and supports a variety of index types (https://milvus.io/docs/overview.md#Index-types https://milvus.io/docs/overview.md#Index-types) and support various consistency levels, scalar/metadata filtering, and time travel. We started working on Milvus back in 2018, with 2.0 being released in January 2022 (https://github.com/milvus-io/milvus/releases/tag/v2.0.0 https://github.com/milvus-io/milvus/releases/tag/v2.0.0). For those interested, here's a comparison with other open source vector databases: https://zilliz.com/comparison https://zilliz.com/comparison. For those who don't want to be burdened with installing and maintaining a local database, there's a managed service available as well: https://zilliz.com/cloud https://zilliz.com/cloud.
- orasis 4y agoCan these databases do fast averaging of nearest neighbors for regression or do you have to retrieve the neighbors and manually compute a mean across them?
- fzliu 4y agoYou'll have to manually compute the mean after retrieving the nearest neighbors. Automatic fast averaging could be an interesting use case for clustering or using a vector database to generate training though.
- orasis 4y agoInteresting. What systems or algorithms are you referring to by “clustering”?
- jamesblonde 4y agoThey are primarily for storing wmbeddings for similarity search - not general purpose nearest neighbor algorithms.
- dang 4y agoRelated: Milvus – An Open-Source Vector Similarity Search Engine - https://news.ycombinator.com/item?id=22012300 https://news.ycombinator.com/item?id=22012300 - Jan 2020 (26 comments) (for curiosity purposes only – reposts are ok after a year or so: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html)
- ronsor 4y agoFunnily enough, I just started working on a vector search extension for SQLite today: https://github.com/Ronsor/VecDex https://github.com/Ronsor/VecDex
- jpcapdevila 4y agoThat is great! I'll keep an eye. I've been playing with this extension: https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss
- simplify 4y agoI only heard about vector databases along with the recent advents of AI. Assuming they've been around for a while, what were the benefits of using them over "normal" search engines (e.g. ElasticSearch)?
- fzliu 4y agoTraditional search such as ES and Lucene rely primarily on bag-of-words retrieval and keyword matching e.g. BM25, TF-IDF, etc. Vector databases such as Milvus allow for _semantic_ search. Here's a highly simplified example: if my query was "fields related to computer science", a semantic engine could return "statistics" and "electrical engineering" while avoiding things such as "social science" and "political science".
- billythemaniam 4y agoES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.
- gk1 4y agoWhich is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/ https://www.pinecone.io/learn/hybrid-search-intro/
- ChocoluvH 4y agoIMO vector databases should not mess with ElasticSearch. The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.
- ChocoluvH 4y ago
- arjvik 4y agoWhat algorithms are these databases using for fast similarity search? Are they guaranteed to return the most optimal similar neighbors or will they be sloppy in return for efficiency?
- fzliu 4y agoOne key differentiator of Milvus is the ability to use a variety of different indexing algorithms. Some will consume very little memory and provide significant speedup at the expense of recall, while others are more powerful at the expensive of compute time or memory consumption. The most commonly used one I've seen is HNSW, which greatly speeds up search at the expensive of some extra memory consumption. Recall is also strong - for many large-scale use cases you can get 95%+ with properly tuned parameters.
- dandiep 4y agoI really don't want another database. I just want to have a solution built in for Postgres, and more specifically, RDS, which we use. I know there will be some extra difficulty that I will have to manage (e.g. reindexing to a new model that is outputting different embeddings), but I really don't want another piece of infrastructure. If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your managed Postgres offerings! 1. https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- fzliu 4y agoTotally with you on that - it's annoying to need multiple pieces of infrastructure, especially for solo developers or small teams. Often times, you want to filter further based on scalar fields/metadata such as timestamps, strings, numeric values, etc... That's why we built attribute filtering into Milvus via a Mongo-esque interface. No SQL and not as performant as an RDBMS, but it's an option: https://milvus.io/docs/hybridsearch.md https://milvus.io/docs/hybridsearch.md
- qwick23 4y agoWith Postgres, you can do almost everything, also a full-text search, but you still have Elasticsearch, Mejlisearch, etc when you need performance and advanced features. The multitool approach is suboptimal in most cases.
- adobrawy 4y agoIn small teams, the infrastructure is often not able to be fully utilized, so performance is not an issue. However, feature richness allows this team to deliver higher-level feature faster. Think early stage startup (one or two engineers) or hairdressers-like business (they use a ready-made framework that targets a popular database and limits its feature to have a wide range of users). As a result, you can have a lot of such business creating a very long tail.
- andre-z 4y agoFor small startups is better just to utilize a managed solution like Pinecone or Qdrant and do not take about infra at all.
- gavi 4y agoHow does this compare to pinecone?
- mikeortman 4y agoPinecone is closed-source AND hosted-infra only... which is a non-starter for many companies.
- jamesblonde 4y agoPinecone offers a free managed tier, which was quite nice until it lost my data last month. They did eventually recover it a few days later, to be fair to them.
- asmyers1793 4y agoCheck out trychroma.com
- andre-z 4y agoQdrant also offers a 1GB free tier https://cloud.qdrant.io https://cloud.qdrant.io
- tommoor 4y agoIt feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.
- spartee 4y agoWe've seen a huge uptick in users of Redis VSS. This thread is great feedback for us so thank you.
- tlrobinson 4y agoOr Postgres with pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- thomasahle 4y agoVector search is still a semi active area of research. You can't get "battle tested" and optimal performance at the same time.
- killthebuddha 4y agoI'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented and cost me ~20 minutes. This is a pretty big problem given item 1. 3. The `@redis/search` TypeScript types didn't allow some valid FS.SEARCH calls. One thing to note is that for some use cases (IMO probably a large fraction of use cases right now), the vectors are derived data, so you don't necessarily need a super robust/battle-tested "database", you just need something with a simple happy-path API.
- leibale1998 4y ago1. node-redis or ioredis? If you are using a "modern IDE" (VS Code/WebStorm/any IDE with support for TS language server) you should get autocomplete for all the commands (in both of them). 2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead. 3. Can you please share the command you are trying to run? BTW, these examples might help: 1. https://github.com/redis/node-redis/blob/master/examples/search-knn.js https://github.com/redis/node-redis/blob/master/examples/sea... 2. https://github.com/redis/node-redis/blob/master/examples/search-json.js https://github.com/redis/node-redis/blob/master/examples/sea... 3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main https://github.com/redis/redis-om-node/tree/main
- qwick23 4y agotbh. Looks like a huge overengineered legacy project. What is the clue to having all these ANN indexes in place? Is it a kinda art collection? What is the sense when you can just have HNSW in memory, with quantization, or on disk, GPU accelerated, etc. There are already better alternatives like Qdrant, which is written in Rust and super performant https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant, or Weaviate with GraphQL interface https://github.com/weaviate/weaviate https://github.com/weaviate/weaviate
- redskyluan 4y agodid you try to serve billion level vectors, on both projects you mentioned? Then you can define what is overengineered. And what about ScaNN and GPU index? the flexibility to support multi index type is a actually a big plus
- elbigbad 4y agoI haven’t tried many alternatives, but I needed a fast, self-hosted vector similarity search that had the ability to cull results based on a second criterion. Milvus has worked really well for me. It does take a ton of memory though such that I can’t run on small VMs, and runs a large number of supporting services.
- leobg 4y agohnswlib? Local. Single file. Allows pre-filtering.
- benibraz 4y agoSurprised to see that no one mentioned https://vespa.ai/ https://vespa.ai/ the best alternative to vector DB imho
- ttt3ts 4y agoLearn faiss first. Mullvus is very complicated in comparison and faiss will work in many cases. I mean look at their system diagram https://milvus.io/static/0bc2e74d0a1b20bbfb91bdbd03f77e5e/bbbf7/architecture_diagram.png https://milvus.io/static/0bc2e74d0a1b20bbfb91bdbd03f77e5e/bb.... Not fun...
- anonymousDan 4y agoLooks pretty standard to me for a high availability cloud setup with disaggregated storage?
- subhrm 4y agoCould you please add a pause button for the animations on your home page? They jump unexpectedly when you are reading the code snippet.
- totalhack 4y agoTried Milvus first and never really got it off the ground. Ended up with Qdrant for its simplicity.
- qwick23 4y agoAlso my experience. And since Qdrant has now cloud service, it is the choice number one. Best of both worlds. Just missing the GCP support.
- ChocoluvH 4y agoDon't start with Milvus clustered version, not unless you have like 100million vectors. Try Milvus standalone instead, much simpler. I also just found their python version (https://github.com/milvus-io/embd-milvus https://github.com/milvus-io/embd-milvus), which is quite neat.
- splatzone 4y agoDon’t laugh at me, but is there a vector database solution for MySQL/Mariadb? It’s been my database of choice for years
- bobvanluijt 4y agoSomebody already mentioned it on this thread, but with Weaviate, we aim to create more end-to-end experience that can be used open source or as (hybrid-)SaaS. For example, you can store the complete data object, ANN and inverted filters (hybrid search), and optional modules for vectorization. More info about the concept behind Weaviate here: https://weaviate.io/developers/weaviate/concepts https://weaviate.io/developers/weaviate/concepts
- vp834 4y agoI am not sure why Milvus is so popular. There are other pretty good options like Weaviate and Vespa - both are open source. One important limitation of Milvus that I cannot ignore is that it does not support pre-filtering like Vespa and a few others do. Pre-filtering is critically important for use cases in e-commerce etc. where filters based on structured metadata are applied e.g. brand, category etc.
- ywmario 4y agoI'm very sure milvus supports it. Milvus calls it hybrid search though: https://milvus.io/docs/hybridsearch.md#Conduct-a-hybrid-vector-search https://milvus.io/docs/hybridsearch.md#Conduct-a-hybrid-vect...