4 ms·
Our top_k search in our own product is in Go and it's ~25 lines of code. Super simple stuff, does the job of funding the top_k in 1ms, no index, no framework, n
by phillipcarter 3y ago
Our top_k search in our own product is in Go and it's ~25 lines of code. Super simple stuff, does the job of funding the top_k in 1ms, no index, no framework, nothing. We benefit from the corpus we embed being relatively static and we can just store them in redis, so it's not a hard problem to solve. Anyways, often you don't need some turnkey solution to this stuff.
- nikhilgarg28 3y agoWhat is the size and dimensionality of your corpus?
- phillipcarter 3y agoSize can be up to ~140MB of vectors, so not exactly enormous. But it's nowhere near enough to stress a single node running a for loop on the data and doing a dot product, then sorting by relevancy.
- zainhoda 3y agoSame — I just asked ChatGPT to generate Go code to do cosine similarity and I was surprised when the code it generated worked and the worst performance I get is like 10ms
- phillipcarter 3y agoYep. If you parallelize with goroutines it'll be faster, but we're talking about shaving a few ms. The effectiveness of a for loop crunching floats in a single thread is not to be underestimated.
- huac 3y agoI have gotten 10x speedups with SIMD on modern hardware - goes from fast to blazing. https://github.com/stillmatic/gollum/blob/main/vectorstore.go https://github.com/stillmatic/gollum/blob/main/vectorstore.g... FWIW - Weaviate's pure Go implementation of cosine similarity is probably as fast as you can possibly get in pure go, without SIMD -- https://github.com/weaviate/weaviate/blob/2f44dbee52196656bd6c0eacb588b5f8b926e8e7/usecases/vectorizer/distance.go#L30 https://github.com/weaviate/weaviate/blob/2f44dbee52196656bd...
- phillipcarter 3y agoYep, we'd go with what they did, except OpenAI embedding vectors are already normalized, so you can just do the dot product.
- nearestn 3y agoWeaviate actually does use SIMD for AMD64 with Go assembly (cosine distance is just normalized dot product) https://github.com/weaviate/weaviate/blob/master/adapters/repos/db/vector/hnsw/distancer/asm/dot_amd64.s https://github.com/weaviate/weaviate/blob/master/adapters/re...
- AA-BA-94-2A-56 3y agoDo you need anything special to run a Rust program with SIMD enabled?