4 ms·
ANN v3: 200ms p99 query latency over 100B vectors
- jascha_eng 8mo agoThis is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
- _peregrine_ 8mo agoseems like a good rule of thumb to me! though i would perhaps lump "cost" into the "until it breaks" equation. even with decent perf, pg_vector's economics can be much worse, especially in multi-tenant scenarios where you need many small indexes (this is true of any vector db that builds indexes primarily on RAM/SSD)
- sa-code 8mo agoQdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good
- benesch 8mo agoFor local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.
- sroussey 8mo agoThat’s not local though
- enigmo 8mo agohaving a local simulator (DynamoDB, Spanner, others) helps me a lot for offline/local development and CI. when a vendor doesn't off this I have often end up mocking it out (one way or another) and have to wait for integration or e2e tests for feedback that could have been pushed further to the left. in many CI environments unit tests don't have network access, it's not purely a price consideration. (not a turbopuffer customer but I have been looking at it)
- benesch 8mo ago> in many CI environments unit tests don't have network access, it's not purely a price consideration. I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services. We've considered the possibility of a local simulator before. Let me know if it winds up being a blocker for your use case.
- lambda 8mo ago> how do you install packages/pull images You pre-build the images with packages installed beforehand, then use those image offline.
- benesch 8mo agoMy point is it's enough of a hassle to set up that I've yet to see that level of restriction in practice (across hundreds of CI systems).
- dzbarsky 8mo agoLook into Bazel, a very standard build system used at many large tech companies. It splits fetches from build/test actions and allows blocking network for build/test actions with a single CLI flag. No hassle at all. The fact that you haven't come across this kind of setup suggests that your hundreds of CI systems are not representative of the industry as a whole.
- lambda 8mo ago> although we get the very occasional complaint about wanting a dev environment that works offline It's only occasional because the people who care about dev environments that work offline are most likely to just skip you and move on. For actual developer experience, as well as a number of use cases like customers with security and privacy concerns, being able to host locally is essential. Fair enough if you don't care about those segments of the market, but don't confuse a small number of people asking about it with a small number of people wanting it.
- sa-code 8mo agoCan confirm. With a setup that works offline, one can - start small on a laptop. Going through procurement at companies is a pain - test things in CI reliably. Outages don’t break builds - transition from laptop scale to web scale easily with the same API with just a different backend Otherwise it’s really hard to justify not using S3 vectors here The current dev experience is to start with faiss for PoCs, move to pgvector and then something heavy duty like one of the Lucene wrappers.
- nik9000 8mo agoAs someone who works for a competitor, they are probably right holding off on that segment for a while. Supporting both cloud and local deployments is somewhere between 20% harder and 300% harder depending on the day. I'm watching them with excitement. We all learn from each other. There's so much to do.
- benesch 8mo agoYep, we're well aware of the selection bias effects in product feedback. As we grow we're thinking about how to make our product more accessible to small orgs / hobby projects. Introducing a local dev environment may be part of that. Note that we already have a in-your-own-VPC offering for large orgs with strict security/privacy/regulatory controls.
- sa-code 8mo agoI should have clarified, by local dev and testing I did in fact mean offline usage. Without that it’s unfortunately a non starter
- benesch 8mo agoSo I can note this down on our roadmap, what's the root of your requirement here? Supporting local dev without internet (airplanes, coffee shops, etc.)? Unit test speed? Something else?
- sa-code 8mo agoI listed some reasons in another comment: https://news.ycombinator.com/item?id=46757853 https://news.ycombinator.com/item?id=46757853 I appreciate your responsiveness and open mind
- nostrebored 8mo agoQdrant is one of the few vendors I actively steer people away from. Look at the GitHub issues, look at what their CEO says, look at their fake “advancements” that they pay for publicity on… The number of people I know who’ve had unrecoverable shard failures on Qdrant is too high to take it seriously.
- sa-code 8mo agoI’m curious about this. Could you please point to some things the CEO has said, or reports of shard failures? The bit about paying for publicity doesn’t bother me. Edit: I haven’t found anything egregious that the CEO has said, or anything really sketchy. The shard failure warnings look serious, but the issues look closed https://github.com/qdrant/qdrant/issues/6025 https://github.com/qdrant/qdrant/issues/6025 https://github.com/qdrant/qdrant/issues/4939 https://github.com/qdrant/qdrant/issues/4939
- nostrebored 8mo agohttps://x.com/nils_reimers/status/1809334134088622217?s=46 https://x.com/nils_reimers/status/1809334134088622217?s=46 https://x.com/generall931/status/1809303448837582850?s=46 https://x.com/generall931/status/1809303448837582850?s=46 There used to be a benchmarking issue with a founder that was particularly egregious but I can’t find it anymore. The sharding and consensus issues were from around a year and a half ago, so maybe it’s gotten better. There are just so many options in the space, I don’t know why you’d go with one of the least correct vendors (whether or not the correctness is deception is a different question that I can’t answer)
- bobvanluijt 8mo ago> issue with a founder That would be me
- andre-z 8mo agoWhat do I say? Happy to talk about "fakes". Here is my calendar. Feel free to book a slot. https://qdrant.to/andre-z https://qdrant.to/andre-z
- jauntywundrkind 8mo agoI'd love to know how they compare versus MixedBread, what relative strengths each has. https://www.mixedbread.com/ https://www.mixedbread.com/ I really really enjoy & learn a lot from the mixedbread blog. And they find good stuff to open source (although the product itself is closed). https://www.mixedbread.com/blog https://www.mixedbread.com/blog I feel like there's a lot of overlap but also probably a lot of distinction too. Pretty new to this space of products though.
- shayonj 8mo agov cool and impressive!
- deleted 8mo ago[deleted]
- lmeyerov 8mo agoFun! I was curious given the cloud discussion - a quick search suggests default AWS SSD bandwidth is 250 MB/s, and you can pay more for 1 GB/s. Similar for s3, one http connection is < 100 MB/s, and you can pay for more parallel connections. So the hot binary quantized search index is doing a lot of work to minimize these both for the initial hot queries and pruning later fetches. Very cool!
- kgeist 8mo agoAre there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.
- _peregrine_ 8mo agothe solution described in the blog post is currently in production at 100B vectors
- rahimnathwani 8mo agoFor what/who?
- esafak 8mo agohttps://turbopuffer.com/customers/cursor https://turbopuffer.com/customers/cursor
- _peregrine_ 8mo agothis is actually not how cursor uses turbopuffer, as they index per codebase and thus need many mid-sizes indexes as opposed to one massive index as this post describes
- _peregrine_ 8mo agounfortunately i'm not able to share the customer or use case :( but the metrics that you see in the first charts in the post are from a production cluster
- jasonjmcghee 8mo agoSo many missing details... Different vector indexes have very different recall and even different parameters for each dramatically impact this. HNSW can have very good recall even at high vector counts. There's also the embedding model, whether you're quantizing, if it's pure rag vs hybrid bm25 / static word embeddings vs graph connections, whether you're reranking etc etc
- mmaunder 8mo agoFor those of us who operate on site, we have to add back network latency, which negates this win entirely and makes a proprietary cloud solution like this a nonstarter.
- benesch 8mo agoOften not a dealbreaker, actually! We can spin up new tpuf regions and procure dedicated interconnects to minimize latency to the on-prem network on request (and we have done this). When you're operating at the 100B scale, you're pushing beyond the capacity that most on-prem setups can handle. Most orgs have no choice but to put a 100B workload into the nearest public cloud. (For smaller workloads, considerations are different, for sure.)
- redskyluan 8mo agoUsing Hierarchical Clustering significantly reduces recall; this is a solution we used and abandoned three years ago.
- montroser 8mo agoThis is at 92% recall. Could be worse, but could definitely be much better. Quantization and hierarchical clustering are tricks that lead to awesome performance at the cost of extremely variable quality, depending on the dataset.
- alanwli 8mo agoOut of curiosity, how is the 92% recall calculated? For a given query, is the recall compared to the true topk of all 100B vectors vs. recall at each of N shards compared to the topk of each respective shard?
- nvanbenschoten 8mo ago(author here) The 92% mentioned in this post is showing recall@10 across all 100B vectors, calculated by comparing to the global top_k. turbopuffer will also continuously monitor production recall at the per-shard level (or on-demand with https://turbopuffer.com/docs/recall https://turbopuffer.com/docs/recall). Perhaps counterintuitively, the global recall will actually be better than the per-shard recall if each shard is asked for its own, local top_k!
- vander_elst 8mo ago> 504MiB shared L3 cache What CPU are they using here?
- benesch 8mo agoThe exact CPU depends on the region/cloud provider, but this Granite Rapids CPU is representative: https://www.intel.com/content/www/us/en/products/sku/240777/intel-xeon-6980p-processor-504m-cache-2-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/240777/...
- vander_elst 8mo agoThanks!
- hwspeed 8mo agoThe offline/local dev point is underrated. Being able to iterate without network latency or metered API costs makes a huge difference for prototyping. The challenge is making sure your local setup actually matches prod behavior. I've been burned by pgvector working fine locally then hitting performance cliffs at scale when the index doesn't fit in memory anymore.