7 ms·

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

spartee 4y ago
We've seen a huge uptick in users of Redis VSS. This thread is great feedback for us so thank you.
tlrobinson 4y ago
Or Postgres with pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
thomasahle 4y ago
Vector search is still a semi active area of research. You can't get "battle tested" and optimal performance at the same time.
killthebuddha 4y ago
I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented and cost me ~20 minutes. This is a pretty big problem given item 1. 3. The `@redis/search` TypeScript types didn't allow some valid FS.SEARCH calls. One thing to note is that for some use cases (IMO probably a large fraction of use cases right now), the vectors are derived data, so you don't necessarily need a super robust/battle-tested "database", you just need something with a simple happy-path API.
- leibale1998 4y ago
  1. node-redis or ioredis? If you are using a "modern IDE" (VS Code/WebStorm/any IDE with support for TS language server) you should get autocomplete for all the commands (in both of them). 2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead. 3. Can you please share the command you are trying to run? BTW, these examples might help: 1. https://github.com/redis/node-redis/blob/master/examples/search-knn.js https://github.com/redis/node-redis/blob/master/examples/sea... 2. https://github.com/redis/node-redis/blob/master/examples/search-json.js https://github.com/redis/node-redis/blob/master/examples/sea... 3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main https://github.com/redis/redis-om-node/tree/main
- qwick23 4y ago
  What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/ https://qdrant.tech/benchmarks/
  - killthebuddha 4y ago
    I haven't really looked into either. So far I've tested pinecone, Redis, chroma, elasticsearch, and pgvector. I'm not really considering performance at all, just looking for something dead-simple to deploy and use. At the moment it looks like pgvector on supabase is the winner.
    - qwick23 4y ago
      Yeah, if you don't need performance, then just take a simple ANN library. No need for a database at all. In terms of databases, Qdrant and Pinecone are the simplest I've tried so far. But Pinecone isn't open-source, not an option for on-prem. PGvector is too much imho, do not want to have all the other Sequel stuff if I just need an NN search.
  - elbartus 4y ago
    Qdrant v1.1 was released recently and its quantization feature is just fantastic . See: https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant
- simplify 4y ago
  Not trying to say you're wrong in your decision, but in the context of the post you're replying to, aren't these points arguments against the node client rather than redis itself?
ttt3ts 4y ago
Redis uses too much memory and supports only two NN algorithms FLAT (very slow) and HNSW if you start indexing millions of large vectors you will quickly understand the problem. That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.
- winddude 4y ago
  it uses a lot of memory because it's an in memory db, that's by design.
- spartee 4y ago
  Enterprise has tiered memory. We have customers in the 50-100m range. benchmarked more than that as well. Single query latency will usually be better than most of the field. Also, were working on other variants of our index that will scale to 500m + More to come. https://github.com/redisventures https://github.com/redisventures to keep up with our team
- fzliu 4y ago
  Redis does fairly well single-threaded - iirc it does better than Milvus. The problem is, once you move to larger applications and want higher performance, its architectural limitations make it impossible to scale.
hobofan 4y ago
One positive thing I can say about Weaviate is that it's incredibly simple to get started with it, as it handles talking to the vectorizer(s) for you, and they provide many out of the box container images for them. With that we were able to integrate a similarity search into our product very quickly. From what I can tell, its competitors generally don't do this, and you have to handle generating the embeddings and all the back and forth with the embeddings yourself, setting the barrier for integration much higher. If you are at a level where you want to get a lot more control about embeddings, I agree that there it will often be better to just use the vector search capabilities that your existing solution like Redis/Postgres are getting.
- bobvanluijt 4y ago
  Correct - this is actually a core focus of Weaviate. That's why it's not limited to only ANN but also has an inverted index, optional modules, etc.

spartee 4y ago

We've seen a huge uptick in users of Redis VSS. This thread is great feedback for us so thank you.

tlrobinson 4y ago

Or Postgres with pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector

thomasahle 4y ago

Vector search is still a semi active area of research. You can't get "battle tested" and optimal performance at the same time.

killthebuddha 4y ago

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented and cost me ~20 minutes. This is a pretty big problem given item 1. 3. The `@redis/search` TypeScript types didn't allow some valid FS.SEARCH calls. One thing to note is that for some use cases (IMO probably a large fraction of use cases right now), the vectors are derived data, so you don't necessarily need a super robust/battle-tested "database", you just need something with a simple happy-path API.

leibale1998 4y ago
1. node-redis or ioredis? If you are using a "modern IDE" (VS Code/WebStorm/any IDE with support for TS language server) you should get autocomplete for all the commands (in both of them). 2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead. 3. Can you please share the command you are trying to run? BTW, these examples might help: 1. https://github.com/redis/node-redis/blob/master/examples/search-knn.js https://github.com/redis/node-redis/blob/master/examples/sea... 2. https://github.com/redis/node-redis/blob/master/examples/search-json.js https://github.com/redis/node-redis/blob/master/examples/sea... 3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main https://github.com/redis/redis-om-node/tree/main
qwick23 4y ago
What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/ https://qdrant.tech/benchmarks/
- killthebuddha 4y ago
  I haven't really looked into either. So far I've tested pinecone, Redis, chroma, elasticsearch, and pgvector. I'm not really considering performance at all, just looking for something dead-simple to deploy and use. At the moment it looks like pgvector on supabase is the winner.
  - qwick23 4y ago
    Yeah, if you don't need performance, then just take a simple ANN library. No need for a database at all. In terms of databases, Qdrant and Pinecone are the simplest I've tried so far. But Pinecone isn't open-source, not an option for on-prem. PGvector is too much imho, do not want to have all the other Sequel stuff if I just need an NN search.
- elbartus 4y ago
  Qdrant v1.1 was released recently and its quantization feature is just fantastic . See: https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant
simplify 4y ago
Not trying to say you're wrong in your decision, but in the context of the post you're replying to, aren't these points arguments against the node client rather than redis itself?

ttt3ts 4y ago

Redis uses too much memory and supports only two NN algorithms FLAT (very slow) and HNSW if you start indexing millions of large vectors you will quickly understand the problem. That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.

winddude 4y ago
it uses a lot of memory because it's an in memory db, that's by design.
spartee 4y ago
Enterprise has tiered memory. We have customers in the 50-100m range. benchmarked more than that as well. Single query latency will usually be better than most of the field. Also, were working on other variants of our index that will scale to 500m + More to come. https://github.com/redisventures https://github.com/redisventures to keep up with our team
fzliu 4y ago
Redis does fairly well single-threaded - iirc it does better than Milvus. The problem is, once you move to larger applications and want higher performance, its architectural limitations make it impossible to scale.

hobofan 4y ago

One positive thing I can say about Weaviate is that it's incredibly simple to get started with it, as it handles talking to the vectorizer(s) for you, and they provide many out of the box container images for them. With that we were able to integrate a similarity search into our product very quickly. From what I can tell, its competitors generally don't do this, and you have to handle generating the embeddings and all the back and forth with the embeddings yourself, setting the barrier for integration much higher. If you are at a level where you want to get a lot more control about embeddings, I agree that there it will often be better to just use the vector search capabilities that your existing solution like Redis/Postgres are getting.

bobvanluijt 4y ago
Correct - this is actually a core focus of Weaviate. That's why it's not limited to only ANN but also has an inverted index, optional modules, etc.