3 ms·

What is a vector database? https://www.pinecone.io/learn/vector-database/ https://www.pinecone.io/learn/vector-database/ ...was less than informative.

jiggawatts 4y ago
Standard row-oriented databases store columns on disk like so: ABCABCABCABC Vector databases store them like this: AAAABBBBCCCC This allows faster queries if you just need one (or a few) columns, because unrelated columns don’t have to be processed at all. Caches are more efficient, vector CPU instructions can be used, etc… The downside is that random single row access is more expensive because a row has to be reassembled from many locations.
- code_biologist 4y ago
  You're describing a column store there, not a vector store.
  - smitty1e 4y ago
    I always thought a columnar store was a series of key:value dictionaries that could produce an RDBMS table result on demand.
    - jiggawatts 4y ago
      That’s a specific physical implantation. If using a key-value storage system then the keys are: Row store: “table id; row id; column id” Column store: “table id; column id; row id” The difference is the order used to sort the data in storage. In reality, most column store systems use a more complex partitioning scheme with row groups and the like…
      - smitty1e 4y ago
        I mean, sure. But even DNS is a key:value store, or would you say that's over-simplifying?
code_biologist 4y ago
Try this one maybe? https://milvus.io/docs/v2.1.x/overview.md https://milvus.io/docs/v2.1.x/overview.md
julienmarie 4y ago
It's a database storing machine learning embeddings. Example: Let's suppose I've downloaded all the Gutemberg library books. I can feed a transformer like Bert or GPT-3 to calculate the embeddings of these books. These embeddings will represent in vector form (an array of numbers of fixed size) the meaning of these books. I can save these vectors in this database and this database can then calculate the distance between these vectors, so basically how closely related they are in terms of topic. If I query this database with the embedding of a sentence like "Love story between teenagers from 2 enemy families in Italy, they die at the end", hopefully the best result will be Romeo & Juliette. I'm no ML person so take my comment with a grain of salt in terms of how well it works. But in theory, that's the goal.

jiggawatts 4y ago

Standard row-oriented databases store columns on disk like so: ABCABCABCABC Vector databases store them like this: AAAABBBBCCCC This allows faster queries if you just need one (or a few) columns, because unrelated columns don’t have to be processed at all. Caches are more efficient, vector CPU instructions can be used, etc… The downside is that random single row access is more expensive because a row has to be reassembled from many locations.

code_biologist 4y ago
You're describing a column store there, not a vector store.
- smitty1e 4y ago
  I always thought a columnar store was a series of key:value dictionaries that could produce an RDBMS table result on demand.
  - jiggawatts 4y ago
    That’s a specific physical implantation. If using a key-value storage system then the keys are: Row store: “table id; row id; column id” Column store: “table id; column id; row id” The difference is the order used to sort the data in storage. In reality, most column store systems use a more complex partitioning scheme with row groups and the like…
    - smitty1e 4y ago
      I mean, sure. But even DNS is a key:value store, or would you say that's over-simplifying?

code_biologist 4y ago

Try this one maybe? https://milvus.io/docs/v2.1.x/overview.md https://milvus.io/docs/v2.1.x/overview.md

julienmarie 4y ago

It's a database storing machine learning embeddings. Example: Let's suppose I've downloaded all the Gutemberg library books. I can feed a transformer like Bert or GPT-3 to calculate the embeddings of these books. These embeddings will represent in vector form (an array of numbers of fixed size) the meaning of these books. I can save these vectors in this database and this database can then calculate the distance between these vectors, so basically how closely related they are in terms of topic. If I query this database with the embedding of a sentence like "Love story between teenagers from 2 enemy families in Italy, they die at the end", hopefully the best result will be Romeo & Juliette. I'm no ML person so take my comment with a grain of salt in terms of how well it works. But in theory, that's the goal.