7 ms·
Vector database that can index 1B vectors in 48M
- ge96 1y agoM is minutes
- ikanade 1y agoLegend
- HarHarVeryFunny 1y agoI was starting to think this was impressive, if not impossible. 1B vectors in 48 MB of storage => < 1 bit per vector. Maybe not impossible using shared/lossy storage if they were sparsely scattered over a large space ? But anyways - minutes. Thanks. Edit: Gemini suggested that this sort of (lossy) storage size could be achieved using "Product Quantization" (sub vectors, clustering, cluster indices), giving an example of 256 dimensional vectors being stored at an average of 6 bits per vector, with ANN being one application that might use this.
- l5870uoo9y 1y agoThankfully not months.
- softwaredoug 1y agoOh the horrors of search indexing Ive seen... including weeks / months to rebuild an index.
- stevemk14ebr 1y agoThank you, title needs edited.
- gaogao 1y agoYeah, the SI symbol for minutes is min, if you're going to abbreviate it in a technical context. Super funky using M.
- williamscales 1y agoAgree the correct abbreviation is min. Nitpick: could be wrong but I don’t think minutes is an SI derived unit.
- gaogao 1y agoYeah, it's under "Non-SI units accepted for use with SI", so more ISO than SI, but has standards for how it's used with SI units, esp since M has meaning there
- OutOfHere 1y agoProprietary closed-source lock-in. Nothing to see here.
- CuriouslyC 1y agoSeriously. The amount of lift a SaaS product needs to give me is insane for me to even bother evaluating it, and there's a near zero percent chance I'll use it in my core.
- kcb 1y agoEspecially a product that demands access to large quantities of your most sensitive data to be useful.
- esseph 1y agoI really feel like we're heading down the slope of a large section of the internet dieing off, and if that happens I think it may fracture even more than it already has globally.
- stronglikedan 1y agoNothing for you to see here. Surely you just aren't their target customer.
- OutOfHere 1y agoSo who is? Who really needs to index 1 billion new vectors every 48 minutes, or perhaps equivalently 1 million new vectors every 3 seconds?
- hansvm 1y agoIf HNSW were accurate enough (and if this DB were much faster) then I'd have a use case. I wound up going down a different route to create a differentiable database for ML shenanigans though.
- HEmanZ 1y ago
- softwaredoug 1y agoNot trying to be snarky, just curious -- How is this different from TurboPuffer and other serverless, object storage backed vector DBs?
- hungarianhc 1y agoHey! It's a great question. Co-founder of Vectroid here. Today, the differences are going to be performance, price, accuracy, flexibility, and some intangible UI elegance. Performance: We actually INITIALLY built Vectroid for the use-case of billions of vectors and near single digit millisecond latency. During the process of building and talking to users, we found that there are just not that many use-cases (yet!) that are at that scale and require that latency. We still believe the market will get there, but it's not there today. So we re-focused on building a general purpose vector search platform, but we stayed close to our high performance roots, and we're seeing better query performance than the other serverless, object storage backed vector DBs. We think we can get way faster too. Price: We optimized the heck out of this thing with object storage, pre-emptible virtual machines, etc. We've driven our cost down, and we're passing this to the user, starting with a free tier of 100GB. Actual pricing beyond that coming soon. Accuracy: With our initial testing, we see recall greater or equal to competitors out there, all while being faster. Flexibility: We are going to have a self managed version for users who want to run on their own infra, but admittedly, we don't have that today. Still working on it. Other Product Elegance: My co-founder, Talip, made Hazelcast, and I've always been impressed by how easy it is to use and how the end to end experience is so elegant. As we continue to develop Vectroid, that same level of polish and focus on the UX will be there. As an example, one neat thing we rolled out is direct import of data from Hugging Face. We have lots of other cool ideas. Apologies for the long winded answer. Feel free to ping us with any additional questions.
- f311a 1y agoI’m curious, what’s the tech stack behind this?
- talipozturk 1y ago
- ashvardanian 1y agoVery curious about the hardware setup used for this benchmark!
- talipozturk 1y agoNo special hardware. Google Cloud vms. We use multiple of them during index building.
- ashvardanian 1y agoThe question is how many, and what kind of VMs you use? It greatly affects performance :) I run a lot of search-related benchmarks (https://github.com/ashvardanian https://github.com/ashvardanian) and curious if you’ve compared to other engines on the same hardware setup, tracing recall, NDCG, indexing, and query speeds.
- talipozturk 1y agoWe shard the data and index on about 6 x n2-standard-96 spot instances so the total cost of indexing the entire deep1b is less than $12. We are working on to make it $6. We separate indexing and query VMs. For queries we use dedicated VMs. USearch numbers look great and are better than ours if you run the query and indexing on the same VM/node. We believe design-wise distributed, task-oriented design is the right way to handle vector search for thousands of tenants with different size datasets. Data ingest is also a separate task for us so Ingest, Index and Query are all handled by different cluster of VMs.
- esafak 1y agoBy the creator of the real-time data platform https://en.wikipedia.org/wiki/Hazelcast https://en.wikipedia.org/wiki/Hazelcast.
- 1999-03-31 1y ago1B vectors is nothing. You don’t need to index them. You can hold them in VRAM on a single node and run queries with perfect accuracy in milliseconds
- lyu07282 1y agoShow your math lol
- sailingparrot 1y agoI assume by "node" OP meant something like a DGX node. Which yea, that would work, but not everyone (no one?) wants to buy a 500k system to do vector search. B200 spec: * 8TB/sec HBM bandwidth * 10 PetaOPs assuming int8. * 186GB of VRAM. If we work with 512-dimensional int8 embeddings, then we need 512GB VRAM to hold them, so assuming we have 8xB200 node (~500k$++), we can easily hold them (125M vectors per GPU). It takes about 1000 OPs to do the dot product between two vectors, so we need to do 1000*1B = 1TeraOPs, spread over 8 GPUs, that's 125 GigaOPs per GPU, so a fraction of a ms. Now the bottleneck will be data movement between HBM -> chips, since we have 125M vectors per GPU, aka 64GB, we can move them in ~8 ms. Here you go, the most expensive vector search in history, giving you the same performance as a regular CPU-based vectorDB for only 1000x the price.
- lyu07282 1y agoThanks for doing the math! I suppose if we are charitable in practice we would of course index and only offload partially to VRAM (FAISS does that with IVF/PQ and similar).
- adastra22 1y ago1B x 4096 = 4T scalars. That doesn't fit in anyone's video ram.
- kingstnap 1y agoWell we have AI GPUs now so you could do it. Each MI325x has 256 GB of HBM. So you would need ~32 of em if it was 2 bytes per scalar.
- cluckindan 1y agoHow is this different from running tuned HNSW vector indices on Elasticsearch?
- wwdmaxwell 1y agoAside from being serverless. This is like elasticsearch but with a kind of built in redis-like layer, I think.
- talipozturk 1y agoco-founder of Vectroid: We forked Lucene. Lucene is awesome for search in general, filters, and obviously full-text search. Very mature and well supported by so many big names and amazing engineers. So we take advantage of that but we had to change a few things to make it work perfectly for Vector use-case. We basically think Vector should be the main data type as it is the most difficult one to deal with. For instance, we modified Lucene to use X number of CPU / threads to build a single segment index. As a result, if/when needed, we can utilize hundreds of CPUs to index quicker and generate less number of segments that will enable lower query latency. We also built a custom File System Directory for Lucene to work off of GCS directly (or S3 later on). It can by-pass the kernel, read from network and write directly into the memory... no SSD, no page-cache, no mmap involved. Perhaps I should not say more...
- ashvardanian 1y agoLucene is tough to deal with. About 15 hours ago — right when this comment was posted — I was giving a talk at Databricks comparing the world’s most widely used search engines. I’ve never run into as many issues with any other similar tool as I did with Lucene. To be fair, it’s been around for ~26 years and has aged remarkably well... but it’s the last thing I’d choose today.
- cluckindan 1y agoInteresting, then, that Vectroid would choose to fork it. Elasticsearch is at least good / at hiding the Lucene zoo under the hood.
- 1y ago
- kgeist 1y agoThere was recently this paper: https://arxiv.org/abs/2508.21038 https://arxiv.org/abs/2508.21038 They show that with 4096-dimensional vectors, accuracy starts to fail at 250 mln documents (fundamental limits of embedding models). For 512-dim, it's just 500k. Is 1 bln vectors practical?
- OutOfHere 1y agoI would think that 1 bln refers to the row count, not to a vector's length.
- yorwba 1y agoThose numbers are for the case where you want all possible pairs of two vectors to have a corresponding query that returns those vectors as the top two results. If you mostly just want to find a particular single vector if possible and don't care so much what the second-best result is, you can get away with much smaller embeddings. And if you do want to cover all possible pairs, 6500 dimensions or so should be enough. (Their empirical results roughly fit a cubic polynomial.)
- chatmasta 1y agoI would like to see a “DataFusion for Vector databases,” i.e. an embeddable library that Does One Thing Well – fast embedding generation, index builds, retrieval, etc. – so that different systems can glue it into their engines without reinventing the core vector capabilities every time. Call it a generic “vector engine” (or maybe “embedding engine” to avoid confusion with “vectorized query engine.”) Currently, every new solution is either baked into an existing database (Elastic, pgvector, Mongo, etc) or an entirely separate system (Milvus, now Vectroid, etc.) There is a clear argument in favor of the pgvector approach, since it simply brings new capabilities to 30 years of battle-tested database tech. That’s more compelling than something like Milvus that has to re-invent “the rest of the database.” And Milvus is also a second system that needs to be kept in sync with the source database. But pgvector is still _just for Postgres_. It’s nice that it’s an extension, but in the same way Milvus has to reinvent the database, pgvector needs to reinvent the vector engine. I can’t load pgvector into DuckDB as an extension. Is there any effort to make a pure, Unix-style, batteries not included, “vector engine?” A library with best-in-class index building, retrieval, storage… that can be glued into a Postgres extension just as easily as it can be glued into a DuckDB extension?
- talipozturk 1y agoI think we have so many of those nice open source libraries but the problem is not the library or the algorithm (hsnw or ivf derivatives).. the problem is figuring out the right distributed architecture to balance cost, accuracy (recall) and speed (latency). I believe no single library will give you all that. For instance if you don't separate writes (indexing) from reads (queries) and scale them separately then your indexing will either suck or your indexing will destroy your read latency. You won't be able to scale as easily either. I believe that is why AWS created Aurora and Google Cloud created AlloyDB to scale relational databases (mysql/postgresql) by separating the reads/writes, implementing a scalable storage backend and by offloading a lot of shared works (replication, compaction, indexing) to cluster of machines.
- chatmasta 1y agoYeah, I feel like these libraries are all one level lower than what I’m asking for. We need something that makes more assumptions (e.g. “I’m running as a component of some kind of database”) but… makes less decisions? Is more flexible? Idk. This is the hard part. DataFusion nailed this balance between an embedded query engine and a standalone database system. It brings just the right amount of batteries that it’s not a super generic thing that does nothing useful out of the box, but it doesn’t bring so many that it needs to compete with full database systems. I believe the maintainers refer to it as “the IR of databases” and I’ve always liked that analogy. That’s what I’d like to see for vector engines. Maybe what we need as a pre-requisite is the equivalent of arrow/parquet ecosystem for vectors. DataFusion really leverages those standards for interoperability and performance. This also goes a long way toward the architectural decisions you reference — Arrow and Parquet are a solid, “good enough” choice for in-memory and storage formats that are efficient and flexible and well-supported. Is there something similar for vector storage?
- roflcopter69 1y agoI think the whole field of vector databases is mostly just one huge misunderstanding. Most of you are not Google or any other big tech company so so won't have billions of embeddings. It's crazy how people add bloat and complexity to their stuff just because they want to do medium scale RAG with ca. 2 million embeddings. Here comes the punchline, you do not need a fancy vector database in this case. I stumbled over https://github.com/sqliteai/sqlite-vector https://github.com/sqliteai/sqlite-vector which is a SQLite extension and I wonder why no one else did this before, but it simply implements a highly optimized brute force search over the vectors, so you get sub 100ms queries over millions of vectors with perfect recall. It uses dynamic runtime dispatch that makes use of the available SIMD instructions your CPU has. Turns out this might be all you need. No need for memory a memory hungry search index (like HNSW) or writing a huge index to disk (like DiskANN).
- stevesimmons 1y agoMight be all you need, except an open source licence: > For production or managed service use, please contact SQLite Cloud, Inc for a commercial license.
- roflcopter69 1y agoDamn, you're right. That's a deal breaker for me at least.
- graphGL 1y agoTry https://github.com/asg017/sqlite-vec https://github.com/asg017/sqlite-vec - Apache 2, Mozilla backed!
- roflcopter69 1y agoI'd be cautious. Project seems abandoned. And I wouldn't say it's one of those cases where a piece of software is just finished and doesn't need any changes.
- nojvek 1y ago