6 ms·
What are embeddings?
- neonate 3y agoPaper: https://github.com/veekaybee/what_are_embeddings/blob/main/embeddings.pdf https://github.com/veekaybee/what_are_embeddings/blob/main/e...
- deleted 3y ago[deleted]
- charcircuit 3y agoA mapping whose codomain is an N dimensional space.
- moralestapia 3y agoDon't know why you're downvoted since you're absolutely correct. They're just locality-sensitive hash functions.
- layer8 3y agoThose seem like two very different definitions.
- pizza 3y agolsh: similarity(x, y) < thresh => E[P(d(lsh(x), lsh(y)) < 1)] > 1-eps for some eps For a model that learns a good representation, and some suitable distance function to measure distances between embeddings (e.g. Euclidean) model: similarity(x, y) < thresh => E[P(d(emb(x), emb(y)) < r] > 1-eps for some cutoff distance r in the vector space
- charcircuit 3y agoThe difference in definitions is that my definition does not imply that the mapping maps similar objects to similar locations in space. Embeddings that do can be useful which is why you see people using machine learning to create them, but it is not a necessary property of an embedding.
- pizza 3y agoTrue, it’s not a necessity just based on the types of things, but in practice it’s the intended use
- corobo 3y agoI'd imagine it's because it's the title of the thing being linked to, not an actual question to be answered. Farting out a quick one line answer is a boring comment
- deleted 3y ago[deleted]
- quickthrower2 3y agoThere is "wikipedia article summary" kind of correct, but that doesn't mean resources that are aimed at teaching are not valuable too.
- rewmie 3y ago> Don't know why you're downvoted since you're absolutely correct. OP might be down voted because, even though the answer might be technically correct, it still completely fails to explain the concept (i.e., it's a useless explanation) and does so with a hint of arrogance and demeaning to readers. Yours too, by the way. It's like someone asks what is temperature and some smartass answers it's a scalar quantity that's mapped from a physical state into a semi-infinite continuous set. Great to know.
- quickthrower2 3y agoA.k.a. a matrix.
- esperent 3y agoThis is the kind of explanation where, by the time you can understand it, you don't need it. Except as a reminder I guess. But it's useless to beginner and intermediate learners.
- accoil 3y agoI think there's value in that style of definition. I found "a monad is just a monoid in the category of endofunctors" the most useful explanation of monads. I didn't know what an endofunctor—or a moniod—was when I read it, but it gave me some hooks for research. Other explanations using analogies just left me confused, and nothing further I could look into. Though that said, I revised that explanation after the others did not help me, so maybe I had absorbed enough context.
- sp332 3y agoI'd like to find a way to start with an embedding and have the computer generate some text that corresponds, at least approximately. There are tools that do that for images, right? Like Stable Diffusion, you can put an image in, get an embedding, then do gradient descent in latent space to find a new embedding, then generate a new image from that.
- kreeben 3y agoThe most basic of embedding (that I can think of) is one where the number of dimensions corresponds to the number of unique characters in your lexicon. If there is a number in one of the components of your embedding that is greater than "0", then you know what characters they are. This embedding does not encode the order of the characters, though. They are just a "bag of characters". If you were to then also encode the order of the characters in, say, yet another embedding, you could use those two embeddings to recreate the original word. Combine the two embeddings into a new vector space and BAM, you've invented "embedding2word".
- jorlow 3y agoGpt (and many others) just add these embeddings together in the model, so you could do that and have one vector that encodes both things together
- TeMPOraL 3y agoThat seems suboptimal, for the same reason LLMs are trained on tokens, and not characters. Tokens seem like a much better "unit of meaning" than characters. Curiously, this also applies to humans: we learn words first, spelling later; we almost always think in terms of words, subwords and phrases, and we're really fast at it - while anything to do with spelling seems to demand much more focus, and some degree of conscious hand-holding. As for encoding sequence, I'm curious how that happens and need to find relevant papers, but I imagine there might be some "n-gram dimensions" in the vector, as in one value for "I'm a first token in a sequence", one value for "I'm a second token in a sequence", etc., which would encode the "occurs before"/"occurs after" relationships using few dimensions, leaving the rest for more interesting relationships.
- KRAKRISMOTT 3y agoOP you need to make it clear that it's a book. The website is confusing.
- gtirloni 3y agoThere is a big green button that says "Get PDF" when I visit it.
- plewd 3y agoiirc HN has a rule/guideline to add [pdf] to the post title in that case, although it might not always be enforced
- esperent 3y agoMy mind automatically dismissed that in the same way it dismisses site headers, ads, mailing list sign up links etc. But even if I did consider it, a link called Get PDF wouldn't make me think it's a book. PDFs can contain books but they usually don't.
- cubefox 3y agoAn array of floats (an n-dimensional vector) which represents some piece of data like a text or an image. Different embeddings can be more or less close to each other, and this closeness indicates similarity.
- teruakohatu 3y ago^^^ this is a really succinct explanation. I would just add that an embedding is an array/vector of numbers. Does not have to be floats although they are in practice. The magic is creating the embedding, embeddings themselves are nothing special.
- specproc 3y agoYeah, I'd been working with them for ages before I really wrapped my head around the sausage machine. There was a Wolfram blog a while back that really helped me grok it [0]. The last state of a neural network prior to prediction, if I understand correctly. [0] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
- chenghuzi 3y agoNot just similarity. It also includes all other possible relationships if you want, such as order, connectivity, etc.
- pedrosorio 3y agoWhat is order/connectivity in R^n?
- chenghuzi 3y agoIt's pretty arbitrary. For example, you can define order as the product of all order relationships along each single dimension (binary product). As for connectivity, they can also be defined as boolean functions under certain radius, like the persistent homology spectrum.
- quickthrower2 3y agoThis looks really good and will add to my reading list. Bear with me on this question as I am still learning... From a quick skim I couldn't find it. It might be there? But with a transformer, at which layer do you consider the vector to be the embedding? Or are there multiple choices? For example you have a positional and token embedding - so they are of course embeddings, but simple ones. King - Man + Woman = Queen type thing. But I imagine you might want to consider the output of the 1st ? 2nd? Nth? transformer on the decoder side as an embedding of sorts. For example the sentence "attention is all you" has gone through a couple of layers you now have a vector that understands those words pretty well. But if you go too far through I imagine you get closer to the "prediction" and so lose the meaning and just get something closer to "need". But if possible to get, that sentence embedding (by which I mean N tokens up to context size) might be useful for search indexing of long text.
- farhanhubble 3y agoThere are multiple choices. For example BERT authors use the pooled output from the last four layers after experimenting with a few other choices like concatenation and only using the last or penultimate layers.
- colordrops 3y agoSomething that is confusing to me is how to use embeddings outside of trivial use cases. Say you want to build something that takes a natural language query, then uses a heterogeneous data set with, say, three different types of data, such as text documents, database schema, and single phrase tags, and you want to create an embedding that maps that native language request to a particular table in a particular database. I have no idea where to begin.
- funerr 3y agoShameless plug, I just wrote about Vector Databases and Embeddings 2 weeks ago in my newsletter (5 minute dev trends): https://unzip.dev/0x014-vector-databases/ https://unzip.dev/0x014-vector-databases/ It also explains how to use them IRL and why.
- alexalx666 3y agothank you!
- funerr 3y agoI'm glad you like it.
- dmitrykan 3y agoNice post, thanks for sharing! Learnt about new dbs / solutions for vector search from it. Vicki mentions this survey I wrote some time ago: https://towardsdatascience.com/milvus-pinecone-vespa-weaviate-vald-gsi-what-unites-these-buzz-words-and-what-makes-each-9c65a3bd0696 https://towardsdatascience.com/milvus-pinecone-vespa-weaviat... Hoping it'll be useful as well!
- platelminto 3y agoHalf way through and though I’m enjoying it, it’s not super clear at times. For example: the Flutter example is introduced, then not mentioned for a while, then mentioned again with a specific problem, then ignored again, then used again with some other different problems. I keep getting lost about what we’re actually trying to achieve with it, and sometimes it feels like it’s entirely unnecessary to refer to it. There are other times where we jump from pretty simple explanations to an intense matrix decomposition. A lot of jargon also gets used that, though previously explained, is used really densely. I feel like keeping the wording simpler would still get the point across. Other than that, I’m finding it quite useful, and I’m only on page 30/75. Thank you for writing it :) (my experience: CS degree, self-studied some AI stuff for the past half a year)
- ossicones 3y agoThe way that this book answered its title question was unsatisfying to me. It felt like it was so focused on the data structure of embeddings that it totally glossed over questions about what the structures mean, why they work so well, and emerging research issues in embeddings. It's like if someone wrote a book called "What is consciousness?" and just provided an annotated list of neurotransmitters. That approach leaves a lot out.