5 ms·

Has anyone actually used embeddings for anything other than Approximate Nearest Neighbor and clustering? Some speculative possibilities that come to mind: - p

by ozb 3y ago

Has anyone actually used embeddings for anything other than Approximate Nearest Neighbor and clustering? Some speculative possibilities that come to mind: - projection, indexing and sorting on arbitrary axes (eg "hot minus cold", "happy minus sad", "scifi minus realism", "literary minus commercial") - SVM-style classification in Embeddings space - word2vec-style reasoning (woman-man+king=queen) - directly training embeddings (ie, not just taking a layer off an LLM); I know people use contrastive training methods, but I'd expect that other methods might be worth exploring, eg you could train embeddings together with neural nets representing functions, generate functional equations, and calculate MSE loss But really, I'm just surprised that it seems to be so focused on semantic search, to the exclusion of anything else... Surely there are other interesting applications?

mcswell 3y ago
Cross-language embeddings, where you create an embedding space in each of two languages and then use a seed dictionary to align the spaces, has potential (actual?) applications in cross-lingual search and probably MT.
mcswell 3y ago
I forgot, we also used word2vec to build an embedding space based on PubMed abstracts. We found lots of variant spellings (including hyphenated, unhyphenated, and space-separated), acronyms and abbreviations for chemical and biochem names. We could probably have built a lexicon of technical terminology from that. Not sure how far we would have gotten with definitions (vectors don't quite work...), but it would be a start. Pretty sure others have done dictionary building using that mechanism too.
godelski 3y ago
I'm a bit confused by the comment because it would seem that these are all relatively common tasks. The first and third being identical. Good example is in vision you might want to semantically change the photo like adding a pair of glasses or doing one of those things you see on the Google commercials. That's done in a latent space I think this is clearest in the Normalizing Flow cases because you're just turning your space into a Gaussian (diffusion does this too, but through approximation methods and isn't invertible, though it is reversible). You project the image/sentence/data you want to manipulate, manipulate within gaussian space, then return back to target space. Or maybe my confusion is shared confusion because embedding is an overloaded word that means a lot of things? Maybe you're just thinking of the first block that converts discrete integer tokens into continuous floats? But we learn those embeddings so even though it becomes like a lookup table it's still a neural process. People do things like SVMs on this space alone. But I think it is like latent space, which is only a bit more abstract. At least embeddings need to be injective, well... mathematically...
- ozb 3y ago
  > relatively common I'd love to see some links, all I see used in practice (including in the OP blog post) is semantic search and a bit of clustering > adding a pair of glasses Actually that (and generally all the SD/VAE stuff) is a great example of the kind of thing I was thinking of, though I have yet to see that concept being used together with a vector database; generally all the user-facing stuff I've seen fits it into the standard "train a model, then do inference" workflow, in contrast to something like semantic search which more obviously focuses on the embeddings themselves > First and third being identical Definitely related, but I make the distinction between projection/sorting along an axis vs constructing a new vector by addition/subtraction > Manipulate within gaussian space, then return to target space This is definitely along the lines of what I had in mind, any example of this being used in practice? > Embedding is an overloaded word Yeah I'm using the term somewhat loosely and broadly here, as basically "a vector in a real vector space where distance represents some notion of semantic similarity" > People do things like SVMs Who?
  - godelski 3y ago
    Here's an example from a Normalizing Flow. Good at density, not great at sampling. https://openai.com/research/glow https://openai.com/research/glow Here's a video of moving around in the latent space of a diffusion model https://www.youtube.com/watch?v=vEnetcj_728 https://www.youtube.com/watch?v=vEnetcj_728 Here's a stylegan one https://www.youtube.com/watch?v=bRrS74RXsSM https://www.youtube.com/watch?v=bRrS74RXsSM Or a VAE on mnist https://www.youtube.com/shorts/pgmnCU_DxzM https://www.youtube.com/shorts/pgmnCU_DxzM I mean it is a bit hard to answer your other questions because like I was pointing out, embeddings and latent spaces are pretty vague terms. For the mathy side, normalizing flows are a great choice since you can parameterize whatever data you want into whatever distribution you want. You then work in that distribution you created, which is approximately isomorphic to the data. But other models do similar things, just more lossy but better at things like sample generation. That's the tradeoff, interpretability/density vs expressitivity/sample quality. But diffusion and NODEs/Score models are closing that gap. But you're going to need to look at applied papers to view more people using them in ways like operational vector spaces. For example, there's VITs TTS uses a NF to parameterize parts of the model or controller networks tend to use similar things. It's more about thinking how your network works and communicates. I think a lot of people are just not thinking to hack away and operate on networks as if they're mathematical models instead of a locked box.
Beefin 3y ago
Data deduplication!
nerdponx 3y ago
> - SVM-style classification in Embeddings space This is a bread-and-butter technique in NLP and machine learning in industry. > directly training embeddings This is literally the original embedding model, Word2Vec.

by ozb 3y ago

mcswell 3y ago

Cross-language embeddings, where you create an embedding space in each of two languages and then use a seed dictionary to align the spaces, has potential (actual?) applications in cross-lingual search and probably MT.

mcswell 3y ago

I forgot, we also used word2vec to build an embedding space based on PubMed abstracts. We found lots of variant spellings (including hyphenated, unhyphenated, and space-separated), acronyms and abbreviations for chemical and biochem names. We could probably have built a lexicon of technical terminology from that. Not sure how far we would have gotten with definitions (vectors don't quite work...), but it would be a start. Pretty sure others have done dictionary building using that mechanism too.

godelski 3y ago

I'm a bit confused by the comment because it would seem that these are all relatively common tasks. The first and third being identical. Good example is in vision you might want to semantically change the photo like adding a pair of glasses or doing one of those things you see on the Google commercials. That's done in a latent space I think this is clearest in the Normalizing Flow cases because you're just turning your space into a Gaussian (diffusion does this too, but through approximation methods and isn't invertible, though it is reversible). You project the image/sentence/data you want to manipulate, manipulate within gaussian space, then return back to target space. Or maybe my confusion is shared confusion because embedding is an overloaded word that means a lot of things? Maybe you're just thinking of the first block that converts discrete integer tokens into continuous floats? But we learn those embeddings so even though it becomes like a lookup table it's still a neural process. People do things like SVMs on this space alone. But I think it is like latent space, which is only a bit more abstract. At least embeddings need to be injective, well... mathematically...

ozb 3y ago
> relatively common I'd love to see some links, all I see used in practice (including in the OP blog post) is semantic search and a bit of clustering > adding a pair of glasses Actually that (and generally all the SD/VAE stuff) is a great example of the kind of thing I was thinking of, though I have yet to see that concept being used together with a vector database; generally all the user-facing stuff I've seen fits it into the standard "train a model, then do inference" workflow, in contrast to something like semantic search which more obviously focuses on the embeddings themselves > First and third being identical Definitely related, but I make the distinction between projection/sorting along an axis vs constructing a new vector by addition/subtraction > Manipulate within gaussian space, then return to target space This is definitely along the lines of what I had in mind, any example of this being used in practice? > Embedding is an overloaded word Yeah I'm using the term somewhat loosely and broadly here, as basically "a vector in a real vector space where distance represents some notion of semantic similarity" > People do things like SVMs Who?
- godelski 3y ago
  Here's an example from a Normalizing Flow. Good at density, not great at sampling. https://openai.com/research/glow https://openai.com/research/glow Here's a video of moving around in the latent space of a diffusion model https://www.youtube.com/watch?v=vEnetcj_728 https://www.youtube.com/watch?v=vEnetcj_728 Here's a stylegan one https://www.youtube.com/watch?v=bRrS74RXsSM https://www.youtube.com/watch?v=bRrS74RXsSM Or a VAE on mnist https://www.youtube.com/shorts/pgmnCU_DxzM https://www.youtube.com/shorts/pgmnCU_DxzM I mean it is a bit hard to answer your other questions because like I was pointing out, embeddings and latent spaces are pretty vague terms. For the mathy side, normalizing flows are a great choice since you can parameterize whatever data you want into whatever distribution you want. You then work in that distribution you created, which is approximately isomorphic to the data. But other models do similar things, just more lossy but better at things like sample generation. That's the tradeoff, interpretability/density vs expressitivity/sample quality. But diffusion and NODEs/Score models are closing that gap. But you're going to need to look at applied papers to view more people using them in ways like operational vector spaces. For example, there's VITs TTS uses a NF to parameterize parts of the model or controller networks tend to use similar things. It's more about thinking how your network works and communicates. I think a lot of people are just not thinking to hack away and operate on networks as if they're mathematical models instead of a locked box.

Beefin 3y ago

Data deduplication!

nerdponx 3y ago

> - SVM-style classification in Embeddings space This is a bread-and-butter technique in NLP and machine learning in industry. > directly training embeddings This is literally the original embedding model, Word2Vec.