3 ms·

I am super ignorant about vector databases, but don't they store n-dimensional vectors, and aren't they used to answer geometric(ish) problems, getting the k cl

I am super ignorant about vector databases, but don't they store n-dimensional vectors, and aren't they used to answer geometric(ish) problems, getting the k closest entries to a particular point? This sounds a problem more similar to what 3D physics engines can do, but generalized to higher dimensions, as opposed to traditional text and key-based database stuff. The algorithms and data structures in physics engines (bounding volume hierarchies, kd-trees, etc.) are quite different from how a traditional database index (B-trees and skip lists are popular there, if I remember correctly) is searched and stored.

mhink 3y ago
Yeah, they do store n-dimensional vectors, but they're part of the LLM zeitgest. The basic idea is that given a particular corpus of text, an LLM can produce a very high-dimensional vector that encodes the features of that text, according to its training parameters. Vectors that are near each other in the high-dimensional space represent pieces of text which the LLM "thinks" are similar to each other. So, by using databases which can efficiently answer geometric nearest-neighbor questions, you can quickly search for chunks of text that are similar to each other.
morgango 3y ago
You are right about this, but in this case a vector would represent a collection of words. Getting the k closes entries would show the k closest text entries to a given query. This is super interesting for "semantic" search, where you are looking for meaning as opposed to just a textual match. For example, the text "chocolate milk" is all the same characters as "milk chocolate", but likely have very different usage within the context of retailers or cooks. So, their vectors should be very different. Word2Vec is NLP that uses a neural net to build these vectors: https://en.wikipedia.org/wiki/Word2vec https://en.wikipedia.org/wiki/Word2vec

mhink 3y ago

Yeah, they do store n-dimensional vectors, but they're part of the LLM zeitgest. The basic idea is that given a particular corpus of text, an LLM can produce a very high-dimensional vector that encodes the features of that text, according to its training parameters. Vectors that are near each other in the high-dimensional space represent pieces of text which the LLM "thinks" are similar to each other. So, by using databases which can efficiently answer geometric nearest-neighbor questions, you can quickly search for chunks of text that are similar to each other.

morgango 3y ago

You are right about this, but in this case a vector would represent a collection of words. Getting the k closes entries would show the k closest text entries to a given query. This is super interesting for "semantic" search, where you are looking for meaning as opposed to just a textual match. For example, the text "chocolate milk" is all the same characters as "milk chocolate", but likely have very different usage within the context of retailers or cooks. So, their vectors should be very different. Word2Vec is NLP that uses a neural net to build these vectors: https://en.wikipedia.org/wiki/Word2vec https://en.wikipedia.org/wiki/Word2vec