8 ms·
Vector Databases
- nitrogen 5y agoThis article won't load -- just a headline and picture is visible. Is there another source?
- dmitrykan 5y agoHi! It is a metered post atm, will consider making it more public over time. If you want to get practical with vector search in Solr/Elasticsearch/Opensearch you can try out this repo: https://github.com/DmitryKey/bert-solr-search https://github.com/DmitryKey/bert-solr-search
- password4321 5y agohttps://archive.is/R2vUg https://archive.is/R2vUg
- gk1 5y agoThis sort of assumes you’re familiar with vector search and vector databases. If you’re not, this might be a good place to start: https://www.pinecone.io/learn/vector-database/ https://www.pinecone.io/learn/vector-database/ TL;DR - They store vector embeddings and index them for efficient nearest-neighbor retrieval. Some also allow CRUD operations, metadata filtering, fault-tolerance, and all that jazz. Useful for things like semantic search, similarity search (of unstructured data), and recommender systems.
- Zababa 5y agoThank you for that, I don't know anything about vector databases so this was useful. I'm not sure if it's normal but the math formulas aren't rendered, they look like what I assume is latex text? But the diagrams are great to get an intuition.
- thirdtrigger 5y agoThis is also a great start (video) to learn about vector databases / vector search engines https://youtu.be/YkK5IKgxp-c https://youtu.be/YkK5IKgxp-c
- dmitrykan 5y agoHi all, the blog's author here. I'm really curious to learn what is the perception of vector search in general, and if you had any exposure to vector databases of any sort / considering to try one out.
- ryohkyo 5y agoHaving written some genetic algorithms and stored the chromosomes, which can be a representation of vectors in any type of database, I find this interesting and will some of them a try. Please correct me if I am wrong, with genetic algorithm, DL, etc. The vector embedding happen on the software layer (inside the network). I imagine it would be somewhat tricky to separate the vector embedding and put it inside a vector database. It could also be a performance hit during the training process.
- dmitrykan 5y agoSuch a great question! I think vector search is designed to solve a specific task, like finding K nearest neighbors for your given vector embedding. If you have a few hundred thousand, a million, it might work fine with a linear scan. Again, this depends on your use case. But if you have tens of millions, hundreds of millions or even a billion of vectors, you will need something more robust to store, index and search over this data. An example in NLP world is BERT-like NNs, that allow you to embed your text into a dense vector representation.
- svcrunch 5y ago> An example in NLP world is BERT-like NNs, that allow you to embed your text into a dense vector representation. I might say transformer-based NNs instead. The problem with cross-attentional models like BERT is that they won't scale to large datasets. They are more often used in reranking results within an IR pipeline. However, even for that use they require distillation.
- dmitrykan 5y agobtw, if would be cool to see some example of a genetic algorithm input and resulting vector? And how do you use the vectors in down-stream tasks?