4 ms·

Respectfully if you don’t know what a vector is, you probably don’t need a vector DB. When they say “vector-search” they mean semantic search. I.e. “which docu

Respectfully if you don’t know what a vector is, you probably don’t need a vector DB. When they say “vector-search” they mean semantic search. I.e. “which document is the most semantically similar to the query text”. So how do we establish semantic similarity? In a database like Elasticsearch, you store text and the DB indexes the text so you can search. In a vector DB you don’t just store the raw text, you store a vectorized version of the text. A vector can be thought of as an array of numbers. To get a vector representation we need some way to take a string and map it to an array while also capturing the notion of semantics. This is hard, but machine learning models save the day! The first popular model used for this was called “word2vec” while a more modern model is BERT. These take an input like “fish” and output a vector like [ 3.12 … 4.092 ] (with over a thousand more elements). So let’s say we have a sentence that we vectorized and we want to compare some user input to see how similar it is to that sentence. How? If we call our sentence A and the input vector B, we can compute a number between zero and one that tells us how similar they are. This is called cosine similarity and is computed by taking the dot product of the two vectors and dividing by both of their magnitudes. When you load a bunch of vectors in a vector DB, the principal operation you will perform is “give me the top K documents that are similar to the input”. The databases indexing process computes k nearest neighbors algorithm on all vectors in the DB and stores this for use at query time. Without the indexing process there is no real difference between a vector db and key value store.

mikeryan 3y ago
Respectfully if you don’t know what a vector is, you probably don’t need a vector DB. I wasn't looking for one ;-) I was looking for a recommendation engine, similarly most often I'm looking for various ways to use ML and AI to improve various features and workflows. Which I guess is my point, I don't know who Pinecone's target market is but from following this thread it seems like all the folks who know how to do what they do have alternatives that suit them better. If they are targeting folks like me they're not doing it well. Pinecone's examples[1] (hat tip to Jurrasic in this thread - I've seen these) all show potential use cases that I might want to leverage, but when you dive into them (for example the Movie Recommender[2] - my use case) I end up with this: The user_model and movie_model are trained using Tensorflow Keras. The user_model transforms a given user_id into a 32-dimensional embedding in the same vector space as the movies, representing the user’s movie preference. The movie recommendations are then fetched based on proximity to the user’s location in the multi-dimensional space. It took me another 5 minutes of googling stuff to parse that sentence. And while I could easily get the examples to run I was still running back and forth to Google to figure out what it was doing in the examples - again the documentation is poor here. I'm not a Python dev but I could follow it but I still had to google tqdm to figure out it was a progress bar library? Also, and this is not unique to Pinecone, I've found generally that while some things are fairly well documented on "Here's how to build a Movie Recommender based on these datasets) frequently in this space there's very little data on how to build a model using your own datasets ie how to take this example and do it with your own data. [1] https://docs.pinecone.io/docs/examples https://docs.pinecone.io/docs/examples [2] https://docs.pinecone.io/docs/movie-recommender https://docs.pinecone.io/docs/movie-recommender
- visarga 3y ago
  Don't worry, you're just catching up in one hour on 10 years of NLP research. There has to be some conceptual gap to cross. After you clarify the "vector" and "computing similarity" concepts, it's pretty nifty. You have a text emb = model(text) Now you got the embedding. What can you do with it? you can calculate how similar it is to other texts. emb1 = model(text1) emb2 = model(text2) similarity = sum([a * b for a, b in zip(emb1, emb2)]) Just a multiply and add, this is trivial! So if you do that for a million texts, you got a search engine. Vector DBs are automating this for you. There are free libraries just as good. And free models to embed text with, OpenAI also have some great embeddings. You can use np.dot to compute similarities fast, up to 100,000 vectors it's the best way and get exact, not approximate results. The great thing about embedding text is the simplicity of the API and the similarity operation. It's dead simple to use. You can do clustering, classification, near neighbour search / ranking, recommendation, or any kind of semantic operations between two texts that can be described as a score. If you cache your vectors you can search very very quickly with np.dot or other methods, in a few ms. Today you can also embed images to the same vector space and do image classification by taking the text label with max dot product. You can also train a very small model on top of embeddings to classify the input into your desired classes, if you can collect a dataset. Embeddings are the best features for text classification. You can think of this embedding method as a way to slice and dice in the semantic space like you do with strings in character space. All fast and local, without GPUs.

mikeryan 3y ago

Respectfully if you don’t know what a vector is, you probably don’t need a vector DB. I wasn't looking for one ;-) I was looking for a recommendation engine, similarly most often I'm looking for various ways to use ML and AI to improve various features and workflows. Which I guess is my point, I don't know who Pinecone's target market is but from following this thread it seems like all the folks who know how to do what they do have alternatives that suit them better. If they are targeting folks like me they're not doing it well. Pinecone's examples[1] (hat tip to Jurrasic in this thread - I've seen these) all show potential use cases that I might want to leverage, but when you dive into them (for example the Movie Recommender[2] - my use case) I end up with this: The user_model and movie_model are trained using Tensorflow Keras. The user_model transforms a given user_id into a 32-dimensional embedding in the same vector space as the movies, representing the user’s movie preference. The movie recommendations are then fetched based on proximity to the user’s location in the multi-dimensional space. It took me another 5 minutes of googling stuff to parse that sentence. And while I could easily get the examples to run I was still running back and forth to Google to figure out what it was doing in the examples - again the documentation is poor here. I'm not a Python dev but I could follow it but I still had to google tqdm to figure out it was a progress bar library? Also, and this is not unique to Pinecone, I've found generally that while some things are fairly well documented on "Here's how to build a Movie Recommender based on these datasets) frequently in this space there's very little data on how to build a model using your own datasets ie how to take this example and do it with your own data. [1] https://docs.pinecone.io/docs/examples https://docs.pinecone.io/docs/examples [2] https://docs.pinecone.io/docs/movie-recommender https://docs.pinecone.io/docs/movie-recommender

visarga 3y ago
Don't worry, you're just catching up in one hour on 10 years of NLP research. There has to be some conceptual gap to cross. After you clarify the "vector" and "computing similarity" concepts, it's pretty nifty. You have a text emb = model(text) Now you got the embedding. What can you do with it? you can calculate how similar it is to other texts. emb1 = model(text1) emb2 = model(text2) similarity = sum([a * b for a, b in zip(emb1, emb2)]) Just a multiply and add, this is trivial! So if you do that for a million texts, you got a search engine. Vector DBs are automating this for you. There are free libraries just as good. And free models to embed text with, OpenAI also have some great embeddings. You can use np.dot to compute similarities fast, up to 100,000 vectors it's the best way and get exact, not approximate results. The great thing about embedding text is the simplicity of the API and the similarity operation. It's dead simple to use. You can do clustering, classification, near neighbour search / ranking, recommendation, or any kind of semantic operations between two texts that can be described as a score. If you cache your vectors you can search very very quickly with np.dot or other methods, in a few ms. Today you can also embed images to the same vector space and do image classification by taking the text label with max dot product. You can also train a very small model on top of embeddings to classify the input into your desired classes, if you can collect a dataset. Embeddings are the best features for text classification. You can think of this embedding method as a way to slice and dice in the semantic space like you do with strings in character space. All fast and local, without GPUs.