3 ms·
An array of floats (an n-dimensional vector) which represents some piece of data like a text or an image. Different embeddings can be more or less close to each
by cubefox 3y ago
An array of floats (an n-dimensional vector) which represents some piece of data like a text or an image. Different embeddings can be more or less close to each other, and this closeness indicates similarity.
- teruakohatu 3y ago^^^ this is a really succinct explanation. I would just add that an embedding is an array/vector of numbers. Does not have to be floats although they are in practice. The magic is creating the embedding, embeddings themselves are nothing special.
- specproc 3y agoYeah, I'd been working with them for ages before I really wrapped my head around the sausage machine. There was a Wolfram blog a while back that really helped me grok it [0]. The last state of a neural network prior to prediction, if I understand correctly. [0] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
- chenghuzi 3y agoNot just similarity. It also includes all other possible relationships if you want, such as order, connectivity, etc.
- pedrosorio 3y agoWhat is order/connectivity in R^n?
- chenghuzi 3y agoIt's pretty arbitrary. For example, you can define order as the product of all order relationships along each single dimension (binary product). As for connectivity, they can also be defined as boolean functions under certain radius, like the persistent homology spectrum.
- TeMPOraL 3y agoI'd say similarity, but any and all kinds of similarity you could ever think of - as long as you have enough dimensions and training data to feed it. My current belief is that a hundred thousand+ dimensional latent space, as in SOTA LLMs, seems to have enough dimensions to reduce a large array of cognitive skills into a proximity search.
- LudwigNagasena 3y agoGenerally, it is any transformation that preserves relevant structure.
- nborwankar 3y agoImportantly it’s a vector space with a metric. Not just an array of vectors. The original Word2Vec paper had the jaw dropping “king - man + woman -> queen” which needs the ability to do addition/subtraction on vectors. Also words “close” in meaning result in vectors “close” in some sense e.g. it needs a formal measure of distance (eg cosine similarity) which you hint at.