14 ms·

Do we really need a specialized vector database?

147 points by gaocegege 3y ago

charcircuit 3y ago
What if your main data is in mysql? Lack of consistency is not a big deal and you often don't have it at scale anyways.
rvba 3y ago
> Search the prompt vector to find the most similar chunk vector. Does this mean that a question is changed to a similar question? Doesnt it decrease the quality of answers significantly? If someone asks "who is the best student in California?" will the question be changed to "what is the best school in California?". This would explain the terrible drops in quality of amswers that we saw. The underlying technology is changed for easier scaling, but is much worse. It's like the current google (what has multiple problems), where it also sometimes doesnt search your keywords - even when they are in quotation marks - because it knows "better"..
- _pdp_ 3y ago
  Without exception, LLM frontends out there that use vector search simply do a query and hope for the best. The reality is that the answer may not be in the result. It is a hard problem to solve. We are at least 1-2 stepping stones away from having a solution that actually works!
  - danielbln 3y ago
    You can do a lot with hybrid approaches, where you mix and match semantic search results with classic text based search. In addition, depending on what kind of knowledge you pull in, the LLM pretraining can often fill the gaps pretty well even if the retrieval augmentation isn't ideal. But yes, it's still pretty fuzzy, though it works more often than not.
    - kordlessagain 3y ago
      Like this: https://GitHub.com/featurebasedb/DoctorGPT https://GitHub.com/featurebasedb/DoctorGPT
- charcircuit 3y ago
  >Does this mean that a question is changed to a similar question? No. It is retrieving the most semantically similar document(s) to work with. For a question about students information about students will be more semantically similar than information about schools. >It's like the current google Google has used vector search for years. It's why you can just type questions in and Google will understand what you are talking about. >doesnt search your keywords - even when they are in quotation marks Quotation marks work. If there are no results it will show results without the quotes. The mobile site doesn't make this clear when it happens though.
danielbln 3y ago
We've gone with Postgres and pg_vector which works really well thus far, especially since we know and like the existing Postgres tooling and vector functionality there is just a (widely supported) pg_vector extension away. Since it now also supports HNSW indices it should perform well on large amounts of vectorized data as well (though we haven't tested this yet). From a consultant perspective, Postgres as a requirement is a much easier sell then some new, hot but unknown dedicated vector DB.
- kiwicopple 3y ago
  > HNSW indices it should perform well on large amounts of vectorized data as well (though we haven't tested this yet) here are a few benchmarks from this week's commit: https://jkatz05.com/post/postgres/pgvector-hnsw-performance/ https://jkatz05.com/post/postgres/pgvector-hnsw-performance/
weinzierl 3y ago
It feels like Postgres is eating the world. In a thread a couple of days ago many argued that with the capability of efficiently storing and processing JSON we can do without MongoDB et al. Now we also replace vector databases. There are extensions that provide columnar storage and to be honest I never completely understood the need for specialized time series databases. It is a bit as if Postgres has become what Oracle wanted to be - a one-stop solution for all your data storage needs.
- rewmie 3y ago
  > (...) and to be honest I never completely understood the need for specialized time series databases. The need for specialized time series databases is explained by the basis of any software development process: reuse ready-made high-level abstractions instead of having to reinvent the wheel, poorly. There are a few key features of time series databases that you either go without (i.e., data compression) or have to reinvent the wheel (querying, resolution-based retention periods, statistical summary of time periods,etc). You might live without those features,but in some applications there's a hefty price tag tied to not having them. If you adopt a time series database, you benefit from all those features without having to implement a single one of them. Perhaps that has no value to you, but it does have for the world.
  - starfallg 3y ago
    I think what OP was getting at is why it has to be specialized. Time-series features can be implemented in a general purpose DB engine like postgres or mongo.
    - nonameiguess 3y ago
      How? I just tried to look for time series in Postgres and it looks like TimescaleDB is the plugin you'd want? At least web search seems to love it. Their landing page is just marketing fluff and says next to nothing about their feature set. In a database that isn't column-oriented, how can they do automatic downsampling? Or do they just not?
Roark66 3y ago
Is there a better article explaining how vector DBs are used with LLMs exactly? This 4 point description raises more questions than it answers for me. For example what is the difference between a "chunk vector" and a "prompt vector"? Aren't they essentially the same thing (a vector representation of text)? How do we "search the prompt vector for the most similar chunk vector"? A short code snippet is shown that queries the DB, but it's not shown what is done with what comes back. What format is the output in? I suspect this works by essentially replacing chunks of input text by shorter chunks of "roughly equivalent" text found in the vector DB and sending that as the prompt instead, but based on this description I can't be sure.
- dcastm 3y ago
  The article is referring to the problem of having a limited context length in LLMs. That is you can only pass X tokens in the prompt. For example, let’s say you have a prompt that lets you answer questions about a book. If the book is long enough, you won’t be able to include it as is in the prompt, so you have to figure out what are the most relevant passages you must include to answer a given question. What you usually do is find the passages that are the most semantically similar to your question. Chunk vectors are the vectorized passages of the book (i.e., a numerical vector that represents a passage), and the prompt vector is usually the vectorized question. To find the most similar vectors you need a distance measure, cosine similarity being the most popular. The output of finding the most similar vectors is the vectors + it’s metadata (chunk, page, chapter, etc)
  - Roark66 3y ago
    Thank you for explaining it. I was aware of the problem of too short context length. I'd love to be able to pass an entire programming project with my prompt for example (or a book in your example). I think I understand how it works now for many kinds of prompts where the information to be extracted is contained in one(or more) of the chunks of much bigger whole. I'm not sure about prompts where it is required to "understand" entire input to answer properly. For example summarising a book. Although even with this vector search could perhaps help by looking for things not near "please provide a summary", but certain hand crafted values such as "important to the plot" etc. I guess I need to do some experimenting with It. I found some open source alternatives to (not at all)OpenAI in form of "Sentence Transformers" to create embeddings. However, what would be really neat is to have a large open source dataset of embeddings already created on some general purpose collection of texts, to try searches etc.
presentation 3y ago
Neon also made a new pg_embedding extension to add to the Postgres mix https://neon.tech/blog/pg-embedding-extension-for-vector-search https://neon.tech/blog/pg-embedding-extension-for-vector-sea...
- ssabev 3y ago
  Any first hand experience? We are using pg_vector and it works really well
  - presentation 3y ago
    Nope, just saw their blog post about it recently.
blackcat201 3y ago
I have been following the vector database trend back in 2020 and I ended up with the conclusion: vector search features are a nice to have features which adds more value on existing database (postgres) or text search services (elasticsearch) than using an entirely new framework full of hidden bugs. You could get way higher speedup when you are using the right embedding models and encoding way than just using the vector database with the best underlying optimization. And the bonus side is that you are using a stack which was battle tested (postgres, elasticsearch) vs new kids (pinecone, milvus ... )
ThinkBeat 3y ago
The work with deciding what words closely related and and which ones are not seems like a hugly difficult problem where there is no single "right" or "wrong". I would think it fits within linguistics.
mark_l_watson 3y ago
Author makes a good argument. I wanted to experiment with local do-it-myself in-memory vector embeddings to use with OpenAI APIs in Common Lisp and Swift. Simple to do, and I added short examples to both my Common Lisp and Swift books. I was also experimenting with simply persisting the data stores to SQLite. Anyway, really simple stuff, and fun to play with.
pmcf 3y ago
The Cassandra project recently[1] added vector search. Relative to a lot of other features, it was fairly simple. A new 'vector' type and an extension of the existing indexing system using the Lucene HNSW library. Now we'll be finding ways to optimize and improve performance with better algorithms and query schemes. What we won't be doing is figuring out how to scale to petabytes of data distributed across multiple data centers in a massive active-active cluster. We've spent the last 14 years perfecting that, and still have work to do. With the benefit of hindsight, if you have a database that is less than 10 years old, all I have to say is good luck. You have some challenging days ahead. 1. https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30...
lampington 3y ago
Great article, but I'm saddened by their view that C is too hard to work with, so the 2-year-old extension must be rewritten in Rust. C certainly has its faults, and while I have no real experience with Rust, I'm willing to believe that it's significantly better as a language. But pgvector, at least from a quick scan, looks like a well-written, easily comprehensible C codebase with decent tests. There are undoubtedly lots of hard problems that the developers have solved in implementing it. Putting time and effort into reimplementing all of that in another language because of an aversion to C feels like a waste of effort that could be put into enhancing the existing extension. Maybe there's something I'm missing? Is the C implementation not as solid as it looks at first glance?
- maximilianburke 3y ago
  Rust makes concurrency really easy, at least in comparison to C or C++. It has great cross-platform frameworks available, like Tokio which pgvecto.rs uses, and makes using them safe and straightforward.
  - ink_13 3y ago
    C also makes concurrency very easy. So easy the uncareful regularly shoot off both feet. Too easy.
    - packetlost 3y ago
      It's hard to get concurrency right in C
  - throwawaymaths 3y ago
    I wonder if having that sort of "safe concurrency" causes developers to overuse concurrency and introduce coordination costs. Do we know that tokio's concurrency strategy is optimal for database access?
    - fsociety 3y ago
      Let’s assume that this is true. Then is the solution to go back to C where it is hard to get concurrency running correctly? Seems backwards thinking.
dmezzetti 3y ago
There isn't a best universal choice for all situations. If you're already using Postgres and all you want is to add vector search, pgvector might be good enough. txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) sets out to be an all-in-one embeddings database. This is more than just being a vector database with semantic search. It can embed text into vectors, run LLM workflows, has components for sparse/keyword indexing and graph based search. It also has a relational layer built-in for metadata filtering. txtai currently supports SQLite/DuckDB for relational data but can be extended. For example, relational data could be stored in Postgres, sparse/dense vectors in Elasticsearch/Opensearch and graph data in Neo4j. I believe modular solutions like this where internal components can be swapped in and out are the best option but given I'm the author of txtai, I'm a bit biased. This setup enables the scaling and reliability of existing solutions balanced with someone being able get started quickly with a POC to evaluate the use case.
- drawnwren 3y ago
  Am I reading this right? You’ve written a vector database in python? Are there any performance benchmark comparisons for me to look at?
  - dmezzetti 3y ago
    txtai is in Python but much of the underlying code isn't. For example, PyTorch/Transformers uses C/C++/CUDA for most operations. The approximate nearest neighbor (ANN) libraries for vector indexes are Faiss, Hnswlib and Annoy. They are all C/C++. Much of the sparse vectors code is NumPy and uses Python's array module for storing term vectors. There are no 3rd party benchmarks for txtai as of now. You would have to compare how it does on your own data to judge.
utopcell 3y ago
The article makes the argument that it is easier to query vector spaces when your database already supports them: why use an external vector db when you can use pgvector in postgreSQL ? That is a fine argument if you don't mind that pgvector is second-to-worst amongst all open-source vector search implementations, and two orders of magnitude slower than the state of the art [1]. The author also makes the argument that traditional DBs are better because they are battle-tested, and then goes and rewrites the pgvector plugin from C to rust. [1] http://ann-benchmarks.com http://ann-benchmarks.com
nologic01 3y ago
"there is a postgres plugin for that"
keskival 3y ago
It's also possible to use HNSW indices with Postgres using pg_embedding instead of pgvector: https://neon.tech/blog/pg-embedding-extension-for-vector-search https://neon.tech/blog/pg-embedding-extension-for-vector-sea... Edit: Ah I see pgvector will soon support HNSW as well (from 0.5.0): https://github.com/pgvector/pgvector/issues/181#issuecomment-1633037078 https://github.com/pgvector/pgvector/issues/181#issuecomment...
tbalsam 3y ago
If you can do it in GPU memory, do it in GPU memory. If it takes quantization + buying a 1 TB RAM server ($4k of RAM + parts), do that in memory with the raw tensors and shed a small tear -- both for cost and the joy of the pain that you are saving yourself, your team, and everyone around you. If you need more, then tread lightly and extremely carefully. Very few mega LLM pretraining datasets are even on this order of magnitude, though some are a few terabytes IIRC. If you are exceeding this, then your business usecase is likely specialized indeed. This message brought to you by the "cost reduction by not adding dumb complexity" group. I try to maintain a reputation for aggressively fighting unnecessary complexity, which is the true cost measure IMO of any system.
gwern 3y ago
> For instance, the 7B model requires approximately 10 seconds for inference on 300 Chinese characters on an Nvidia A100 (40GB). Is that right? That sounds like, 2 OOMs higher than I would've expected. Is he doing something wrong like loading the model from a cold start?
- mergejoin 3y ago
  i was _very_ surprised by that number as well. 10s for inference on 300 chars? using an entire A100 GB GPU? that does not make any sense. OpenAI and other companies would NEVER be able to scale to 100s of millions of users if that was the real number, right?
adam_oxla 3y ago
Great article. I was recently wondering why are vector databases useful at all. There are cases (LLMs) where they are useful. But apart from that, most of the cases I have encountered, requires indexing on different data types than vector. I might be biased due to my experience but in a very typical use cases e.g. analyzing event streams or building recommendation engine you want to filter rows by timestamp and client id or product category and availability. Vector index is not that useful in those cases. That's why I think that outside some very specific use cases vector databases are not very useful.

147 points by gaocegege 3y ago

What if your main data is in mysql? Lack of consistency is not a big deal and you often don't have it at scale anyways.

rvba 3y ago

> Search the prompt vector to find the most similar chunk vector. Does this mean that a question is changed to a similar question? Doesnt it decrease the quality of answers significantly? If someone asks "who is the best student in California?" will the question be changed to "what is the best school in California?". This would explain the terrible drops in quality of amswers that we saw. The underlying technology is changed for easier scaling, but is much worse. It's like the current google (what has multiple problems), where it also sometimes doesnt search your keywords - even when they are in quotation marks - because it knows "better"..

_pdp_ 3y ago
Without exception, LLM frontends out there that use vector search simply do a query and hope for the best. The reality is that the answer may not be in the result. It is a hard problem to solve. We are at least 1-2 stepping stones away from having a solution that actually works!
- danielbln 3y ago
  You can do a lot with hybrid approaches, where you mix and match semantic search results with classic text based search. In addition, depending on what kind of knowledge you pull in, the LLM pretraining can often fill the gaps pretty well even if the retrieval augmentation isn't ideal. But yes, it's still pretty fuzzy, though it works more often than not.
  - kordlessagain 3y ago
    Like this: https://GitHub.com/featurebasedb/DoctorGPT https://GitHub.com/featurebasedb/DoctorGPT
charcircuit 3y ago
>Does this mean that a question is changed to a similar question? No. It is retrieving the most semantically similar document(s) to work with. For a question about students information about students will be more semantically similar than information about schools. >It's like the current google Google has used vector search for years. It's why you can just type questions in and Google will understand what you are talking about. >doesnt search your keywords - even when they are in quotation marks Quotation marks work. If there are no results it will show results without the quotes. The mobile site doesn't make this clear when it happens though.

danielbln 3y ago

We've gone with Postgres and pg_vector which works really well thus far, especially since we know and like the existing Postgres tooling and vector functionality there is just a (widely supported) pg_vector extension away. Since it now also supports HNSW indices it should perform well on large amounts of vectorized data as well (though we haven't tested this yet). From a consultant perspective, Postgres as a requirement is a much easier sell then some new, hot but unknown dedicated vector DB.

kiwicopple 3y ago
> HNSW indices it should perform well on large amounts of vectorized data as well (though we haven't tested this yet) here are a few benchmarks from this week's commit: https://jkatz05.com/post/postgres/pgvector-hnsw-performance/ https://jkatz05.com/post/postgres/pgvector-hnsw-performance/

weinzierl 3y ago

It feels like Postgres is eating the world. In a thread a couple of days ago many argued that with the capability of efficiently storing and processing JSON we can do without MongoDB et al. Now we also replace vector databases. There are extensions that provide columnar storage and to be honest I never completely understood the need for specialized time series databases. It is a bit as if Postgres has become what Oracle wanted to be - a one-stop solution for all your data storage needs.

rewmie 3y ago
> (...) and to be honest I never completely understood the need for specialized time series databases. The need for specialized time series databases is explained by the basis of any software development process: reuse ready-made high-level abstractions instead of having to reinvent the wheel, poorly. There are a few key features of time series databases that you either go without (i.e., data compression) or have to reinvent the wheel (querying, resolution-based retention periods, statistical summary of time periods,etc). You might live without those features,but in some applications there's a hefty price tag tied to not having them. If you adopt a time series database, you benefit from all those features without having to implement a single one of them. Perhaps that has no value to you, but it does have for the world.
- starfallg 3y ago
  I think what OP was getting at is why it has to be specialized. Time-series features can be implemented in a general purpose DB engine like postgres or mongo.
  - nonameiguess 3y ago
    How? I just tried to look for time series in Postgres and it looks like TimescaleDB is the plugin you'd want? At least web search seems to love it. Their landing page is just marketing fluff and says next to nothing about their feature set. In a database that isn't column-oriented, how can they do automatic downsampling? Or do they just not?

Roark66 3y ago

Is there a better article explaining how vector DBs are used with LLMs exactly? This 4 point description raises more questions than it answers for me. For example what is the difference between a "chunk vector" and a "prompt vector"? Aren't they essentially the same thing (a vector representation of text)? How do we "search the prompt vector for the most similar chunk vector"? A short code snippet is shown that queries the DB, but it's not shown what is done with what comes back. What format is the output in? I suspect this works by essentially replacing chunks of input text by shorter chunks of "roughly equivalent" text found in the vector DB and sending that as the prompt instead, but based on this description I can't be sure.

dcastm 3y ago
The article is referring to the problem of having a limited context length in LLMs. That is you can only pass X tokens in the prompt. For example, let’s say you have a prompt that lets you answer questions about a book. If the book is long enough, you won’t be able to include it as is in the prompt, so you have to figure out what are the most relevant passages you must include to answer a given question. What you usually do is find the passages that are the most semantically similar to your question. Chunk vectors are the vectorized passages of the book (i.e., a numerical vector that represents a passage), and the prompt vector is usually the vectorized question. To find the most similar vectors you need a distance measure, cosine similarity being the most popular. The output of finding the most similar vectors is the vectors + it’s metadata (chunk, page, chapter, etc)
- Roark66 3y ago
  Thank you for explaining it. I was aware of the problem of too short context length. I'd love to be able to pass an entire programming project with my prompt for example (or a book in your example). I think I understand how it works now for many kinds of prompts where the information to be extracted is contained in one(or more) of the chunks of much bigger whole. I'm not sure about prompts where it is required to "understand" entire input to answer properly. For example summarising a book. Although even with this vector search could perhaps help by looking for things not near "please provide a summary", but certain hand crafted values such as "important to the plot" etc. I guess I need to do some experimenting with It. I found some open source alternatives to (not at all)OpenAI in form of "Sentence Transformers" to create embeddings. However, what would be really neat is to have a large open source dataset of embeddings already created on some general purpose collection of texts, to try searches etc.

presentation 3y ago

Neon also made a new pg_embedding extension to add to the Postgres mix https://neon.tech/blog/pg-embedding-extension-for-vector-search https://neon.tech/blog/pg-embedding-extension-for-vector-sea...

ssabev 3y ago
Any first hand experience? We are using pg_vector and it works really well
- presentation 3y ago
  Nope, just saw their blog post about it recently.

blackcat201 3y ago

I have been following the vector database trend back in 2020 and I ended up with the conclusion: vector search features are a nice to have features which adds more value on existing database (postgres) or text search services (elasticsearch) than using an entirely new framework full of hidden bugs. You could get way higher speedup when you are using the right embedding models and encoding way than just using the vector database with the best underlying optimization. And the bonus side is that you are using a stack which was battle tested (postgres, elasticsearch) vs new kids (pinecone, milvus ... )

ThinkBeat 3y ago

The work with deciding what words closely related and and which ones are not seems like a hugly difficult problem where there is no single "right" or "wrong". I would think it fits within linguistics.

mark_l_watson 3y ago

Author makes a good argument. I wanted to experiment with local do-it-myself in-memory vector embeddings to use with OpenAI APIs in Common Lisp and Swift. Simple to do, and I added short examples to both my Common Lisp and Swift books. I was also experimenting with simply persisting the data stores to SQLite. Anyway, really simple stuff, and fun to play with.

pmcf 3y ago

The Cassandra project recently[1] added vector search. Relative to a lot of other features, it was fairly simple. A new 'vector' type and an extension of the existing indexing system using the Lucene HNSW library. Now we'll be finding ways to optimize and improve performance with better algorithms and query schemes. What we won't be doing is figuring out how to scale to petabytes of data distributed across multiple data centers in a massive active-active cluster. We've spent the last 14 years perfecting that, and still have work to do. With the benefit of hindsight, if you have a database that is less than 10 years old, all I have to say is good luck. You have some challenging days ahead. 1. https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30...

lampington 3y ago

Great article, but I'm saddened by their view that C is too hard to work with, so the 2-year-old extension must be rewritten in Rust. C certainly has its faults, and while I have no real experience with Rust, I'm willing to believe that it's significantly better as a language. But pgvector, at least from a quick scan, looks like a well-written, easily comprehensible C codebase with decent tests. There are undoubtedly lots of hard problems that the developers have solved in implementing it. Putting time and effort into reimplementing all of that in another language because of an aversion to C feels like a waste of effort that could be put into enhancing the existing extension. Maybe there's something I'm missing? Is the C implementation not as solid as it looks at first glance?

maximilianburke 3y ago
Rust makes concurrency really easy, at least in comparison to C or C++. It has great cross-platform frameworks available, like Tokio which pgvecto.rs uses, and makes using them safe and straightforward.
- ink_13 3y ago
  C also makes concurrency very easy. So easy the uncareful regularly shoot off both feet. Too easy.
  - packetlost 3y ago
    It's hard to get concurrency right in C
- throwawaymaths 3y ago
  I wonder if having that sort of "safe concurrency" causes developers to overuse concurrency and introduce coordination costs. Do we know that tokio's concurrency strategy is optimal for database access?
  - fsociety 3y ago
    Let’s assume that this is true. Then is the solution to go back to C where it is hard to get concurrency running correctly? Seems backwards thinking.

dmezzetti 3y ago

There isn't a best universal choice for all situations. If you're already using Postgres and all you want is to add vector search, pgvector might be good enough. txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai) sets out to be an all-in-one embeddings database. This is more than just being a vector database with semantic search. It can embed text into vectors, run LLM workflows, has components for sparse/keyword indexing and graph based search. It also has a relational layer built-in for metadata filtering. txtai currently supports SQLite/DuckDB for relational data but can be extended. For example, relational data could be stored in Postgres, sparse/dense vectors in Elasticsearch/Opensearch and graph data in Neo4j. I believe modular solutions like this where internal components can be swapped in and out are the best option but given I'm the author of txtai, I'm a bit biased. This setup enables the scaling and reliability of existing solutions balanced with someone being able get started quickly with a POC to evaluate the use case.

drawnwren 3y ago
Am I reading this right? You’ve written a vector database in python? Are there any performance benchmark comparisons for me to look at?
- dmezzetti 3y ago
  txtai is in Python but much of the underlying code isn't. For example, PyTorch/Transformers uses C/C++/CUDA for most operations. The approximate nearest neighbor (ANN) libraries for vector indexes are Faiss, Hnswlib and Annoy. They are all C/C++. Much of the sparse vectors code is NumPy and uses Python's array module for storing term vectors. There are no 3rd party benchmarks for txtai as of now. You would have to compare how it does on your own data to judge.

utopcell 3y ago

The article makes the argument that it is easier to query vector spaces when your database already supports them: why use an external vector db when you can use pgvector in postgreSQL ? That is a fine argument if you don't mind that pgvector is second-to-worst amongst all open-source vector search implementations, and two orders of magnitude slower than the state of the art [1]. The author also makes the argument that traditional DBs are better because they are battle-tested, and then goes and rewrites the pgvector plugin from C to rust. [1] http://ann-benchmarks.com http://ann-benchmarks.com

nologic01 3y ago

"there is a postgres plugin for that"

keskival 3y ago

It's also possible to use HNSW indices with Postgres using pg_embedding instead of pgvector: https://neon.tech/blog/pg-embedding-extension-for-vector-search https://neon.tech/blog/pg-embedding-extension-for-vector-sea... Edit: Ah I see pgvector will soon support HNSW as well (from 0.5.0): https://github.com/pgvector/pgvector/issues/181#issuecomment-1633037078 https://github.com/pgvector/pgvector/issues/181#issuecomment...

tbalsam 3y ago

If you can do it in GPU memory, do it in GPU memory. If it takes quantization + buying a 1 TB RAM server ($4k of RAM + parts), do that in memory with the raw tensors and shed a small tear -- both for cost and the joy of the pain that you are saving yourself, your team, and everyone around you. If you need more, then tread lightly and extremely carefully. Very few mega LLM pretraining datasets are even on this order of magnitude, though some are a few terabytes IIRC. If you are exceeding this, then your business usecase is likely specialized indeed. This message brought to you by the "cost reduction by not adding dumb complexity" group. I try to maintain a reputation for aggressively fighting unnecessary complexity, which is the true cost measure IMO of any system.

gwern 3y ago

> For instance, the 7B model requires approximately 10 seconds for inference on 300 Chinese characters on an Nvidia A100 (40GB). Is that right? That sounds like, 2 OOMs higher than I would've expected. Is he doing something wrong like loading the model from a cold start?

mergejoin 3y ago
i was _very_ surprised by that number as well. 10s for inference on 300 chars? using an entire A100 GB GPU? that does not make any sense. OpenAI and other companies would NEVER be able to scale to 100s of millions of users if that was the real number, right?

adam_oxla 3y ago

Great article. I was recently wondering why are vector databases useful at all. There are cases (LLMs) where they are useful. But apart from that, most of the cases I have encountered, requires indexing on different data types than vector. I might be biased due to my experience but in a very typical use cases e.g. analyzing event streams or building recommendation engine you want to filter rows by timestamp and client id or product category and availability. Vector index is not that useful in those cases. That's why I think that outside some very specific use cases vector databases are not very useful.